I am attempting to produce an Ubuntu live and Windows version of this utility. I have found suitable ffmpeg and produced sidecar for both environments. I have pulled the hrwconv script from a humax box. It uses "HRW2HDR=/mod/bin/hrw2hdr" but I can not find anything on hrw2hdr using the forum search. Any pointers please.
-
The forum software that supports hummy.tv has been upgraded to XenForo 2.3!
Please bear with us as we continue to tweak things, and feel free to post any questions, issues or suggestions in the upgrade thread.
Request for information re hrwconv
- Thread starter sceedy
- Start date
MymsMan
Ad detector
hw2dr is a compiled C program that lives in the /mod/bin/ directory if he package is installed.
To use it on Ubuntu you would need to compile it for that environment but the source is not included in the package, you would need to contact the package owner (@xyz321) to see if he was willing to make the source available or provide a Ubuntu executable
edit: already answered whilst I was delayed in posting
To use it on Ubuntu you would need to compile it for that environment but the source is not included in the package, you would need to contact the package owner (@xyz321) to see if he was willing to make the source available or provide a Ubuntu executable
edit: already answered whilst I was delayed in posting
prpr
Well-Known Member
I think you'll find sceedy is doing something rather clever somehow, which doesn't involve source code. I haven't had time to look in any detail at what/how.
But if the person with the source is willing to port, then that's got to be an offer not to refuse I'd have thought.
I would like to use hrw2hdr in 2 ports of hrwconv, one that can be run from a live linux distro and the other that can be run from windows. Therefore I would like 2 ports. For a live linux distro it needs to be a static linux executable (my problems with sidecar was all to do with it being linked dynamically). For windows I would prefer it to be a 32bit executable as this will execute on everything from xp to 10. It was at 7 that 95% of windows installs were 64 bits. Thanks for the offer xzy321.
prpr - The "something clever" was to produce a staticly linked version of sidecar i.e. one that can be run when aproximatly 70% of the dynamic library is missing. The windows version was an asperation that I managed to achieve. My final goal has always been a port of hrwconv that can take advantage of a pc with more processing umph than the hummy!
prpr - The "something clever" was to produce a staticly linked version of sidecar i.e. one that can be run when aproximatly 70% of the dynamic library is missing. The windows version was an asperation that I managed to achieve. My final goal has always been a port of hrwconv that can take advantage of a pc with more processing umph than the hummy!
Does anyone know of a wiki that explains H264 headers ?
Background : I picked up windows Vn2.7 of sidecar and it worked in every environment (hardware and OS) I tried it in. I have not (yet ?) found a version of ffmpeg that was OK with subtitles and worked in every environment, so I wondered about “cutting out the middle man”. I put files from various other sources onto the HDR-FOX T2 and they all played so I wondered if they could be indexed without having to process them with ffmpeg (hence migrate - see attachment - .7z renamed to .zip). It turns out that although it will play anything with an arbitrary amount of preamble and postamble around a ts packet (the 188 byte bit) indexing such files makes them unplayable (if you want to see what I mean recompile with nonew defined and index a file of 188 byte packets). The HDR-FOX T2 not only wants a 192 byte packet for indexing to work but constructed as a 4 byte preamble (followed by) the 188 ts packet (as I found out indexing a file with a 2 byte preamble and 2 byte postamble around the ts packet) so migrate now detects if the input file can be indexed and produces a .new file when it produces sidecar files. This is faster than running ffmpeg and then sidecar and copes with files that ffmpeg does not recognise (i.e. non standard preamble and postamble lengths). Everything was going nicely with my testing when I came across some files recorded on my test 9300T of the community channel. Sidecar indexes the .new file renamed to .ts and shows it as being “Format SD H264, 544 x 576i.”. I could just get migrate to give a different exit code for H264 and use sidecar to index, but I am finding working it out from scratch educational.
Background : I picked up windows Vn2.7 of sidecar and it worked in every environment (hardware and OS) I tried it in. I have not (yet ?) found a version of ffmpeg that was OK with subtitles and worked in every environment, so I wondered about “cutting out the middle man”. I put files from various other sources onto the HDR-FOX T2 and they all played so I wondered if they could be indexed without having to process them with ffmpeg (hence migrate - see attachment - .7z renamed to .zip). It turns out that although it will play anything with an arbitrary amount of preamble and postamble around a ts packet (the 188 byte bit) indexing such files makes them unplayable (if you want to see what I mean recompile with nonew defined and index a file of 188 byte packets). The HDR-FOX T2 not only wants a 192 byte packet for indexing to work but constructed as a 4 byte preamble (followed by) the 188 ts packet (as I found out indexing a file with a 2 byte preamble and 2 byte postamble around the ts packet) so migrate now detects if the input file can be indexed and produces a .new file when it produces sidecar files. This is faster than running ffmpeg and then sidecar and copes with files that ffmpeg does not recognise (i.e. non standard preamble and postamble lengths). Everything was going nicely with my testing when I came across some files recorded on my test 9300T of the community channel. Sidecar indexes the .new file renamed to .ts and shows it as being “Format SD H264, 544 x 576i.”. I could just get migrate to give a different exit code for H264 and use sidecar to index, but I am finding working it out from scratch educational.
My last post has been overtaken by events. I have managed to find a copy of ‘H.264 Standard2007.pdf’ entitled ‘ITU-T H.264 (11/2007) Advanced video coding for generic audiovisual services’ hence my latest attachment (again .7z renamed to .zip). It now has MPEG and H264 capabilities. It is Vn0.3 because Vn0.1 and Vn0.2 had an underflow problem.
It is now at the stage where it would be useful to me for people to break it. I know that file closure errors are not yet reported, Vn0.4’s first update will be to put in error messages for all library functions that return errors.
As this utility is intended to process non-native HDR video streams, I would strongly recommend using sidecar for native HDR video streams. If I have not made any codeing errors, this utility will be between 10% to 75% slower as it does not read the disk in multiples of the physical disk block size. This utility does not have any pedigree. It was generated from various wiki websites and downloaded pdfs but the maximum file size does not allow me to include these in the attached archive.
It is now at the stage where it would be useful to me for people to break it. I know that file closure errors are not yet reported, Vn0.4’s first update will be to put in error messages for all library functions that return errors.
As this utility is intended to process non-native HDR video streams, I would strongly recommend using sidecar for native HDR video streams. If I have not made any codeing errors, this utility will be between 10% to 75% slower as it does not read the disk in multiples of the physical disk block size. This utility does not have any pedigree. It was generated from various wiki websites and downloaded pdfs but the maximum file size does not allow me to include these in the attached archive.
I must admit I have been sidetracked by Migrate. Some international relatives have recorded onto USB sticks directly from their TVs so that I can watch series before they air in the UK if I visit before that time. I assumed they were all encrypted because they did not play on other TVs (even the same model) but approximately three quarters of them became playable on my HDR-FOX T2 using Migrate. It now has all the functionality I intended for 9**0T file processing, has passed all the test cases I have thrown at it, and no bugs have been reported hence Vn0.4 (i.e. the first one with a full set of error messages) (again a .7z renamed to .zip).
I am dyslexic, so it may just be me, but having used the NTS File Format wiki from this site “in anger” I believe it could do with some minor clarifications. The latest version contains a text file (view as landscape) with my proposed clarifications. I would also appreciate a peer review of this file to tell me if the original or this version is clearer. Note that dyslexic people think differently so I am aware (from before I retired) that my “clarifications” of technical documents sometimes confuses people more. Such comments do not upset me as my philosophy of life is that if I have learned something, it is a good day!
I am dyslexic, so it may just be me, but having used the NTS File Format wiki from this site “in anger” I believe it could do with some minor clarifications. The latest version contains a text file (view as landscape) with my proposed clarifications. I would also appreciate a peer review of this file to tell me if the original or this version is clearer. Note that dyslexic people think differently so I am aware (from before I retired) that my “clarifications” of technical documents sometimes confuses people more. Such comments do not upset me as my philosophy of life is that if I have learned something, it is a good day!
Try using 7-Zip (note the comment "a .7z renamed to .zip" (.7z is the native format for 7-Zip))
MymsMan
Ad detector
7-zip can create pure zip files which would avoid renaming and make life easier for those wanting to use the package who might not have 7-Zip installed
From your included text file :
MPEG2 - 32 byte entries
Offset Purpose Value
--------------------------------------------------------------------------------------------------------------
0x03 1 byte Frame Type MPEG frame type from the Picture header 0x01 = I-Frame, 0x02 = P-Frame, 0x03 = B-Frame, 0x04 = D-Frame.
Can you clarify your clarification ?
I'll take your lack of response as a "no comment" then.
D-frames were part of the original mpeg1 video spec but were rarely ever used even then. Consequently, they were not included in later video specs, including the mpeg2 standard used in SD broadcast streams. As such, they could never be identified with a 0x04 code in any .nts file produced on the HDR T2, or any other Humax box that I'm aware of.
So there is really no need for the wiki nts format for the HDR T2 to be 'clarified' to include this non-existent identifier which only you seem to be aware of.
D-frames were part of the original mpeg1 video spec but were rarely ever used even then. Consequently, they were not included in later video specs, including the mpeg2 standard used in SD broadcast streams. As such, they could never be identified with a 0x04 code in any .nts file produced on the HDR T2, or any other Humax box that I'm aware of.
So there is really no need for the wiki nts format for the HDR T2 to be 'clarified' to include this non-existent identifier which only you seem to be aware of.
Sorry about the perceived lack of response, “good weather stopped play”.
Thanks for the comment on .zip. In future, if it fits, I will use native zip files.
As far as I am aware, all MPEG2 capable devices can play MPEG1 so I tried it. If you don’t duplicate the the temporal sequence number and frame type from the picture header then you get pixilation on fast forward/reverse (and skip). I was typing to process non-HDR native video so, as you have pointed out, D frames need to be clarified as non-native. Thanks for the informative comment!
I was mainly concerned over Note 1. I was trying to test the hypothesis that the byte at offset 0x04 was only used when the first entry was read. The contents makes no difference (in common with most entries) unless you fast forward/reverse (or skip). I placed a corruption at 30s played time but it’s effects occurred at a random time and included premature playback termination and pixilation. I am not sure how to encapsulate that in a comment.
Thanks for the comment on .zip. In future, if it fits, I will use native zip files.
As far as I am aware, all MPEG2 capable devices can play MPEG1 so I tried it. If you don’t duplicate the the temporal sequence number and frame type from the picture header then you get pixilation on fast forward/reverse (and skip). I was typing to process non-HDR native video so, as you have pointed out, D frames need to be clarified as non-native. Thanks for the informative comment!
I was mainly concerned over Note 1. I was trying to test the hypothesis that the byte at offset 0x04 was only used when the first entry was read. The contents makes no difference (in common with most entries) unless you fast forward/reverse (or skip). I placed a corruption at 30s played time but it’s effects occurred at a random time and included premature playback termination and pixilation. I am not sure how to encapsulate that in a comment.
FINALY I can release Migrate Vn 0.5! I made the mistake of treating the user guide as though it was a technical spec so had lots of dead ends before achieving the audio processing I desire.
When I started this I believed that I would be bulk processing 1T of 9300T .ts files on a regular basis. It turned out what my family wanted was the new PVR to be preloaded with videos of family events that can be skipped, fast forwarded etc. On average 6 files needed to be transferred so everything I wrote to cope with bulk processing has not been used in anger. My goal has changed. I wish to achieve Vn 0.9 functionality before publishing bulk processing.
I am no longer the family "go to" guy for authoring videos of family events as they are now filed in HD but rips of the Blu-rays caused me grief until I found the following "magic incantation" :-
ffmpeg -i filename.ts -vcodec copy -acodec aac -scodec copy filename.m2ts
It is not the same as sidecar. Sidecar is intended to index native files with any usefulness for non-native files as a welcome side effect.
Differences :-
Open source.
Parameters - Sidecar is not my IP so I can’t use the same parameter passing mechanism without permission.
Input file packet length - Sidecar is 192 bytes. Migrate is >= 188 bytes. If packet length <> 192, or other incompatibilities with native files, it creates a HDR T2 compatible file. The original intention was to index files from 9200T/9300T. I was given files from other sources that are basically a transport stream and the HDR T2 compatible file can be indeed to give skip, fast forward etc. hence this design decision.
PTS timestamp - Sidecar is totally reliant on this being correct to index the input file. Default processing ignores the timestamp. I counts frames and use the video frame rate to index the files. My families DVD wedding videos were usually shot on 3 cameras. One of the digital cameras time reset to 00:00 1 Jan 1970 every time the battery was changed. The timestamp of the analogue camera footage indicated when it was digitised. The DVD authoring tool used preserved timestamps. The re-multiplexing method used also preserved timestamps. I know the timestamps on most of the video I wish to process is not worth the file space it is stored in hence this design decision.
Audio stream selection - I have found no author’s statement for sidecar on this. ETSI TS 101 154 V2.4.1 Digital Video Broadcasting (DVB); Specification for the use of Video and Audio Coding in Broadcast and Broadband Applications states that up to 3 audio streams can be present in order primary language audio, secondary language audio, audio description (implied to be in the primary language). I give preference to the stream in the preferred language. The default preferred language is English but can be changed by parameter. This allows me to listen to recordings of Canadian broadcasts in English using the default processing.
hmt files - Sidecar allows these to be updated. I always regenerate from scratch (not there on the roadmap to Vn 0.9 yet).
When I started this I believed that I would be bulk processing 1T of 9300T .ts files on a regular basis. It turned out what my family wanted was the new PVR to be preloaded with videos of family events that can be skipped, fast forwarded etc. On average 6 files needed to be transferred so everything I wrote to cope with bulk processing has not been used in anger. My goal has changed. I wish to achieve Vn 0.9 functionality before publishing bulk processing.
I am no longer the family "go to" guy for authoring videos of family events as they are now filed in HD but rips of the Blu-rays caused me grief until I found the following "magic incantation" :-
ffmpeg -i filename.ts -vcodec copy -acodec aac -scodec copy filename.m2ts
It is not the same as sidecar. Sidecar is intended to index native files with any usefulness for non-native files as a welcome side effect.
Differences :-
Open source.
Parameters - Sidecar is not my IP so I can’t use the same parameter passing mechanism without permission.
Input file packet length - Sidecar is 192 bytes. Migrate is >= 188 bytes. If packet length <> 192, or other incompatibilities with native files, it creates a HDR T2 compatible file. The original intention was to index files from 9200T/9300T. I was given files from other sources that are basically a transport stream and the HDR T2 compatible file can be indeed to give skip, fast forward etc. hence this design decision.
PTS timestamp - Sidecar is totally reliant on this being correct to index the input file. Default processing ignores the timestamp. I counts frames and use the video frame rate to index the files. My families DVD wedding videos were usually shot on 3 cameras. One of the digital cameras time reset to 00:00 1 Jan 1970 every time the battery was changed. The timestamp of the analogue camera footage indicated when it was digitised. The DVD authoring tool used preserved timestamps. The re-multiplexing method used also preserved timestamps. I know the timestamps on most of the video I wish to process is not worth the file space it is stored in hence this design decision.
Audio stream selection - I have found no author’s statement for sidecar on this. ETSI TS 101 154 V2.4.1 Digital Video Broadcasting (DVB); Specification for the use of Video and Audio Coding in Broadcast and Broadband Applications states that up to 3 audio streams can be present in order primary language audio, secondary language audio, audio description (implied to be in the primary language). I give preference to the stream in the preferred language. The default preferred language is English but can be changed by parameter. This allows me to listen to recordings of Canadian broadcasts in English using the default processing.
hmt files - Sidecar allows these to be updated. I always regenerate from scratch (not there on the roadmap to Vn 0.9 yet).
Black Hole
May contain traces of nut
Thanks for sharing, hopefully it will be useful to somebody... but perhaps a clear summary of what it's for will help.
prpr
Well-Known Member
Of course you can.
Black Hole
May contain traces of nut
I thought that was a bit extreme too.
MymsMan
Ad detector
And whoever needs it is highly unlikely to find it hidden in a thread entitled "Request for information re hrwconv"
A properly title and described announcement post in a new thread would help potential users find it.