sidecar package on Ubuntu ?

[prpr]

Here you go johnb, try this.

I've tried this (on Ubuntu 14.04) and it dies after reading all of the input file:

sidecar -i -n test

test.ts
Duration 2254 seconds.
Video ID $0100, Type MPEG.
Audio ID $0101, Type MPEG.
Audio Desc ID $0000, Type not present.
PMT ID $1000.
Prog ID $0001.
Channel Number 0.
Channel Name Freeview.
No event information available.
Create NTS file.
Analyzing source video.
Writing NTS file entries.
*** Error in `sidecar': double free or corruption (fasttop): 0x08b54f20 ***
Aborted

I tried running it under valgrind, which does confirm some not very pretty things about memory allocation.
Is there any chance you could look at what's going on with this and fix it please?

(Repeating on the T2 itself, it just spins apparently in a 100% CPU loop.)

 

[johnb]

Curious - I've been running it on Ubuntu Server 12.04 without any problems at all.

The command I have been using is:

sidecar -n -h -p -i -c 99 -t "Channel Name" "file name"​

Where 99 represents the channel number.

It would be interesting to know if you get the same error when using the same command. If you don't that might indicate sidecar is expecting certain parameters and doesn't error catch when they are missing. (Not that I know anything about it.)

 

[raydon]

Curious - I've been running it on Ubuntu Server 12.04 without any problems at all.

The command I have been using is:

sidecar -n -h -p -i -c 99 -t "Channel Name" "file name"​

Where 99 represents the channel number.

You don't really need all those dashes. i.e. sidecar -nhpi -c 99 -t "Channel Name" "file name"

It would be interesting to know if you get the same error when using the same command. If you don't that might indicate sidecar is expecting certain parameters and doesn't error catch when they are missing. (Not that I know anything about it.)

I very much doubt it, problem persists even when processing source on the T2 itself. Sidecar does catch any missing parameters, just try it and see..... I suspect the problem lies elsewhere, namely with the source video.

 

[raydon]

I've tried this (on Ubuntu 14.04) and it dies after reading all of the input file:

sidecar -i -n test
 
test.ts
Duration 2254 seconds.
Video ID $0100, Type MPEG.
Audio ID $0101, Type MPEG.
Audio Desc ID $0000, Type not present.
PMT ID $1000.
Prog ID $0001.
Channel Number 0.
Channel Name Freeview.
No event information available.
Create NTS file.
Analyzing source video.
Writing NTS file entries.
*** Error in `sidecar': double free or corruption (fasttop): 0x08b54f20 ***
Aborted

I tried running it under valgrind, which does confirm some not very pretty things about memory allocation.
Is there any chance you could look at what's going on with this and fix it please?

(Repeating on the T2 itself, it just spins apparently in a 100% CPU loop.)

From the output from sidecar that you've provided it's apparent that the source video is not a native T2 recording, or it has already been processed using some third party application. In the original Sidecar thread I did respond to BH on the subject of native T2 recordings stating "Native means native to the HDR/HD Fox T2, which is what it was built for, and is all that I will guarantee." The format of a source video must, in the main, match that created by the T2 for sidecar to work correctly. However, if you can upload your source video somewhere then I will endeavour to determine exactly what it is in your particular file that is causing sidecar to fail.

 

[raydon]

@prpr Can I assume from you not responding that you no longer wish me to pursue the issue you reported ? I'm quite happy to look into it but I do need your cooperation in order to resolve it. If you can't upload the whole file then a short clip cut from the start would do, so long as it exhibits the same properties as the original.

Edit: Sorry, I note you have not posted anywhere on the forum for four days now, so it's possible your away. I'll await your return.

 

[prpr]

@prpr Can I assume from you not responding that you no longer wish me to pursue the issue you reported ? I'm quite happy to look into it but I do need your cooperation in order to resolve it. If you can't upload the whole file then a short clip cut from the start would do, so long as it exhibits the same properties as the original.

Edit: Sorry, I note you have not posted anywhere on the forum for four days now, so it's possible your away. I''ll await your return.

Busy with work and other stuff and no time to play.
To answer the previous questions... I'd forgotten this was T2 recordings only and the videos in question are from an old Panasonic analogue PVR. These were in .VRO format (as the only way I could get them off the disk) but converted to .m2ts (then renamed to .ts) and audio converted to mp2 instead of ac3 (not sure if I needed to do this bit? - Edit: Yes I do!). Video is mostly 704x576 which I guess rules it out, but some are 720x576 (I don't know why/how they are different as they all came from the same machine). They all fail the same way though.
Video files are very big and I don't know how to sub-clip something out of them reliably now. May just have to record some gash video from somewhere to test with if you think there's any point.
However, it is not very nice for a program to just crash if it encounters unexpected data, whatever it may be. Something that tells us what is actually wrong is much more preferable, as we may then be able to pre-process the input to correct whatever it doesn't like.

 

[johnb]

I too once had a Panasonic HD recorder (actually the first consumer HD recorder on the market) - it was then a great leap forward from VHS but, IMO, a flawed implementation of the idea.

VideoReDo Suite can edit the .VRO files.

There was one recording mode on the old Panasonic that is very tricky edit: 'FR'. In this mode the Panasonic automatically chose the best 'quality' to fill a DVD and it did this by alternating between two of the other modes (say XP and SP).

 

[prpr]

Mine's a DMR-E85H which I think was fairly early. I had to do the standard PSU fix a few years ago. The hard disk is OK but the DVD drive won't read/write DVD-R but will do DVD-RAM most of the time (I guess it's dust related but it's hard to get inside the drive). You have to do the transfer to PC one programme at a time, which is tedious in the extreme and the (slightly dodgy) remote doesn't help with the sheer number of button presses needed. I quite liked it 10 years ago but it does seem badly aged now and needlessly convoluted in its operation. Thankfully I never used FR mode - mostly SP and occasionally XP; the others weren't worth using.

 

[raydon]

audio converted to mp2 instead of ac3 (not sure if I needed to do this bit? - Edit: Yes I do!).

Actually, you don't. Sidecar will create an appropriate entry in the hmt file to enable the T2 to recognise, and playback AC3 audio.

Video files are very big and I don't know how to sub-clip something out of them reliably now.

You can use HJ-Split to cut or rejoin any file, with user selectable splitfile size. It's available for a variety of platforms including Windows and Linux. Just set the splitfile size to "video size/duration minutes" to the nearest Mb to give you one minute chunks. Rename the first chunk back to .ts and just ditch the rest. You don't even need to let HJ-Split complete the operation, so long as it's ouput the first chunk. The resulting short clip should still be playable and generate the same errors in sidecar. If you can upload samples (both 704x576 and 720x576) I'll have a look at them.

it is not very nice for a program to just crash if it encounters unexpected data, whatever it may be. Something that tells us what is actually wrong is much more preferable, as we may then be able to pre-process the input to correct whatever it doesn't like.

I agree, which is why I need the sample for analysis to determine exactly what is causing the problem. I still can't guarantee that it will generate an nts file, but a least it should exit gracefully with an appropriate error message. It's very difficult to create a catch-all utility because non-native .ts files may not be what they purport to be. Sidecar needs the same format as the broadcast stream for an .nts to be created successfully, hence the caveat on native recordings only, and also why AV2HDR-T2 needs to remux a non-native source into a format known to work. I do try very hard to cover all eventualities but ultimately I have no control over what people try to feed into sidecar, or AV2HDR-T2.

 

[prpr]

Actually, you don't. Sidecar will create an appropriate entry in the hmt file to enable the T2 to recognise, and playback AC3 audio.

Having tried it again, it does work, so I guess I screwed up somewhere along the line with all the various tests.

You can use HJ-Split to cut or rejoin any file, with user selectable splitfile size.

I just used dd in the end to do this. Seemed a lot simpler.

If you can upload samples (both 704x576 and 720x576) I'll have a look at them.

For everyone else's benefit, this was done and Raydon has fixed the bugs. Sidecar 1.4 (see post #8) now works for me in all tests done so far.

I have no control over what people try to feed into sidecar, or AV2HDR-T2.

Indeed, but meaningful error messages help everybody! Thanks once again for the fix.

 

[raydon]

Indeed, but meaningful error messages help everybody!

I agree, and I can assure you that failure to display a meaningful message was not intentional. Appropriate error messages are already built into the utility. Unfortunately, on this occasion, an undetected bug was causing a premature crash rather than exiting gracefully with one of the canned messages.

 

[sceedy]

I have produced some variants of sidecar (including one for Windows). The attachments are .7z files renamed to .zip. The sidecar2 archive does not have the test results in it but does have the variants. The other archive is the tool that makes variants of sidecar possible.

Now I have overcome the “speedbump” of trying to use sidecar with Ubuntu live I can go back to generating hrwconv for Linux (or Windows, I have options now)!

I have e-mailed the author of STATIFIER (valery_reznic@users.sourceforge.net) a copy of ProofOfConceptUnlink.7z.

** EDIT - Sorry guys. raydon obviously believes that my variants break his copyright despite my efforts not to modify the binary, just provide wrappers. Obviosly this means hrwconv for windows and linux live can not be releseased.

 

[andyk]

I wanted to generate new sidecar files for "Cowboys & Aliens" using the web interface. This didn't work. The window displayed was blank, without the usual buttons etc. After a bit of messing around, I discovered the problem was with the "&" in the file name. I renamed the file to replace "&" with "and" all was well.

The (Windows) version of sidecar provided by sceedy doesn't seem to have a problem with the filename i.e. reads the file and generates an nts file. So maybe the opt+=>sidecar web interface is unhappy when operating on filesets with "&" in the name.

 

[andyk]

I see the problem with ampersand has been reported before, last September (but I can't find out how to add a reference so here's the whole thing)

I've been having a few problems with sidecar initiated from the web-if where it appears to hang.
It appears that it doesn't like an ampersand in the filename.
Not a problem now that I know what te problem is as I can just remove the ampersand before processing the video with sidecar but I thought I'd mention it in case anyone else has the same problem in the future.
Diagnostics shows that the filename is truncated at the ampersand causing a file not found error.

 

[sceedy]

I am human so I can make mistakes. I can also suffer from “garbage in, garbage out”. When I first read the comment on the windows version before I reached the end of the sentence …(windows)… I was trying to work out what could be wrong. I have done enough testing to know that the file and console IO work perfectly (I could be human and thus be mistaken). There are 3 other components that have no direct equivalents in the MinGW32 libraries namely __xpg_basename, __isoc99_sscanf & regexec. __xpg_basename is trivial, you want MinGW32’s basename because it correctly parses windows file paths (I could be human and thus be mistaken). My research found a missing __isoc99_sscanf more than 3 times each of which said that differences with MinGW32’s sscanf are trivial (“garbage in, garbage out”). regexec’s documentation in the MinGW32 compatible source talks about internal private formats not being binary compatible with the Linux versions as the only problem (“garbage in, garbage out”).

Plan B is to take sidecar.o and link it to __isoc99_sscanf & regexec static versions leaving everything else dynamically linked in a Linux executable. Then I can do my “magic” again and thus eliminate the possible “garbage in, garbage out” errors. I am a ones and zeros guy and was the windows guy in a group of linux experts before I retired, so that is beyond my Linux knowledge. If somebody can educate me, or provide me with it, that would give a solution in the proverbial back pocket. I just hope that Plan B is a waste of time but better to be safe than sorry!

 

[andyk]

Just to be clear, I know of know of no problem with the Windows version of sidecar - it was just a convenient quick (and not very good) check to see if the problem was with sidecar (running on the Humax) or the interfacing to it via the web interface, which might be corrected.

Incidentally I intend to start using the Windows sidecar, it's so much faster than the one on the box for generating nts files if you've transferred the ts file to a PC. So thanks for your efforts in creating this.

 

[MontysEvilTwin]

Out of interest, I presume these new sidecar variants still have the non-monotonically increasing timestamps bug? This is most obvious with reverse search at x2 and x4 speeds with mpeg video, and causes pixellation.

 

[sceedy]

andyk - I am violently agreeing with you. Your post was clear. It was me. I am dyslexic but intelligent enough that I was only diagnosed two weeks before my degree’s final exams. What I was trying to explain comes from the way I read. I get the whole page at once and then randomly get words decoded. Once all the words are decoded I then get them sorted into the correct order and then decode the conditional modifications of adjectives and adverbs. The practical upshot is that there are several seconds between me seeing a page and “reading it” in the traditional sense. In this time I process lots of word combinations (a bit like a processor’s speculative execution) which are usually discarded. However I believed the combination error, sidecar and (windows) was worthy of comment.

The reason I believed the comment was worth airing is that I has suffered from raydon’s “under bus” and did not wish the same to be true of me. No criticism of your post was intended. I can suffer from being verbose when I attempt to explain things that are a result of me thinking differently. If so I apologise.

Off topic : When you have worked in the computing industry for any length of time you pick up a set of puns on, or misuses of, technical terms that are of questionable taste but convey meaning perfectly e.g. applying “head crash” to a person to convey too intoxicated to function correctly. “over bus” is the technical term, the “under bus” pun indicates something that the person could do in a few minutes but is either unable or unwilling to do being as though they were suffering from having been in an accident where they went under a bus.

MontysEvilTwin – You are correct. ProofOfConceptUnlink performs “archioligy”, it just tries to unearth the .o from the dynamically linked executable so the core functionality is correct. I made a design decision NOT to produce any relocation records for any internal function calls that need not be touched. That has the effect that p<8 digit hex number> can not be hooked (I was respecting the (c) in the announcement). I have had feature requests from other forums so am considering an update. Sidecar having bugs that should be fixed would be a reason for such a release. Please give a score out of 10 for this bugs effect on functionality or make it clear that you are really requesting an update where hooks of p… is possible (including the usage instructions for the new option flag).

 

[Black Hole]

I have had feature requests from other forums so am considering an update.

I think we would appreciate full disclosure, particularly as this is the one and only official support forum for the HDR-FOX Custom Firmware.

 

[sceedy]

Black Hole - I was refereing to ProofOfConceptUnlink which is a tool to reverse the linking process of dynamicly linked linux executables. Without help from others this tool would not be possible. People on stackoverflow have realised who I am and e-mailed me directly. What else do you want to know under full disclosure ?

The feature I was refering to was to create relocation records for the p...'s i.e. the procedures/functions that have been discovered because they are called from somewhere and then extend the hook mechanism to allow this to happen i.e. create 2 symbols in the symbol table p... and hookp... and change the linker records so it calls hookp... . If you look at hook.c in the sidecar archive you will see how the hook mechanism was used to produce a pseudo Posix environment that makes the windows variant posible. If somebody wanted to update sidecar you would need to either do lots of hex editing or do the hook mechanism update so parts of the executable can be replaced or funtion modified. I used hook not wrapper because the mechanism is not a full wrapper.