Auto-processing incorrectly flagging encrypted files as "Dec", if a decrypted duplicate exists on the disk

R

Robobunny

Member
[Edit - this has been diagnosed, jump to post 9 to skip the debug and see the resolution]

I had a bunch of programmes that are still encrypted (couldn't play them in Windows), despite showing "Dec" in the WebIf. While working out what's going on, I may have found a bug in the way both Auto-processing and the "fixencflags" script applies flags to files (or maybe in the way the files' flags themselves are impemented?). I better stop speculating and let those more qualified decide.

Due to my fairly unique scenario, it will likely bother hardly anyone (not even me, I can easily work around it), so I'm happy if the forum admins or clever people authoring the WebIf scripts want to close this as a "won't fix". But for the moment I'll keep my setup the same in case the bug might have further implications and/or anyone wants me to try something.

Scenario:
  • I have a bunch of BBC HD programmes, duplicated in a different directory on the disk (I only today found this duplication out, while trying to diagnose my undecrypted files, the duplicate probably happened by accident years ago when I put a new HDD in and copied stuff back).
  • One copy is genuinely still encrypted, the other has been decrypted.
  • The parent directory of all of the above used to have recursive-decrypt and recursive-shrink set.
  • Both copies of the files show both the "Dec" and "shrunk" icon.
  • Diagnostic "encheck" correctly finds the incorrectly marked "Dec" files, saying "HMT marked encrypted: 0 Stripts thinks encrypted: 1" for each.
  • Diagnostic "fixencflags" corrects this, showing: "FIXED DEC /media/My Video/.../xxxx", just for each "still encrypted" file. (There's also one rogue "FIXED ENC" in there too, in the same set of directories.)
  • After "fixencflags": The WebIf does not show "Dec" for BOTH copies (even though one set is).
  • [EDIT - inserting this bullet] Another diagnostic "encheck" following the "fixencflags" now correctly finds the incorrectly NOT marked "Dec" files, saying "HMT marked encrypted: 1 Stripts thinks encrypted: 0" for each.
  • Next time Auto-processing runs, BOTH copies get the "Dec" flag put back on. "auto.log" shows (only for the decrypted copy) "decrypt: /media/My Video/path-to-decrypted-version/yyyy.ts - already decrypted but the HMT flag is wrong."
I'm running the latest CFW and WebIf (and did a reinstall). Packages "auto-unprotect" and "auto-update" are installed, a 'long check' fixdisk (2TB disk) shows no issues, and nothing recursive is set on these directories or any of their parents any more. I do use ad-detect, but these are BBC.
 
Last edited:
Are you quite sure these really are duplicated files, and not the same files referenced from two different directory locations (eg Flatview)?
 
Yes, quite sure, it's a good suggestion but I don't use Flatview, it is not installed. If I enable the Samba share, I can browse the video folders, and can play the files from the directory that's decrypted, and not from the other one. Just for belt & braces I'll FTP the same .ts file from both directories, compare, and edit this post with the result. [Edit - one can be played, the other not, files are binary different, mostly 192 byte blocks - m2ts (8 bytes the same, 184 bytes different), so one encrypted, one not]
 
Last edited:
Although my original issue with the directories in question remains, I've been unable to manually recreate it with a new recording (so the post title isn't entirely accurate as written). I tried making a short BBC HD recording and copying to another directory, manually decrypting one, processing queue... and also using off-the-box copy/paste (FTP) of another recording's files (ts, nts, hmt), and a variety of recursive decrypt, and everything behaves exactly as one would expect.

So even though these "problem child" files are definitely both physically there and are binary different, it does seem somehow along @Black Hole 's idea that somehow they are being referenced to "the same" by the way that Auto and fixencflags are accessing the recordings. It's in such a way that:
  • Auto process sets the "Dec" flag on both sets of files (reminder, one has really been decrypted, the other still encrypted).
  • "fixencflags" removes the "Dec" flag on both sets of files.
  • Running "encheck" correctly spots the erroneous "Dec" flag (or lack thereof) after both of the above.
 
Last edited:
Sounds very mysterious, and if there is no way to replicate it on demand there will be no way to get to the bottom of it.
 
Do any of the problem files play (on the Humax) at all under the various flag conditions?
I'm wondering if there's a double decryption gone on which is confusing things.
It's all rather hard to follow though, without having the files to analyse.
Would you be prepared to ship me one pair of recordings, each with the .hmt and .ts (well the first 20MB or so anyway, not the whole thing) and tell me what the key on your unit is?
 
Interestingly the problem files don't play on the Humax, but in a counter-intuitive way (so much so I've double checked the following really is true before posting...):

Recording in "directory 1": Plays on Humax, Plays via Humax DLNA, does NOT play if copied to a PC via Samba share (so assuming encrypted even though WebIF says "Dec"). Running fixencflags says "FIXED DEC".

Same recording in "directory 2": Does NOT play on Humax ("The channel is scrambled or not available"), does NOT play via Humax DLNA, does play if copied to a PC via Samba share (so must be decrypted). Running fixencflags says "FIXED ENC".

@prpr I'll get you those files...
 
Last edited:
I should add, the previous post (and files sent to prpr) show the situation immediately after running "fixencflags".
Once the Auto-processing has run again, files in both directories get the "Dec" flag put back on them (auto.log only mentions "already decrypted but the HMT flag is wrong" for files in "directory 2" though), and the "Plays on Humax, Plays via Humax DLNA" situation is then reversed (i.e. "directory 1" files do not play on the Humax or DLNA, "directory 2" files do play on the Humax and DLNA).
 
Problem solved. At least the cause is known and not a Humax bug. Although the .ts video files in the two directories are unique, from the CLI we could see all the sidecar files for each video were pointing to the same inode (so were hard links to the same file, as @Black Hole initially alluded to).

Because only one set of .ts were actually decrypted, the "fixencflags" script (which checks if the "Dec" flag is correct or missing) was turning off the "Dec" flag for those that weren't, and the "auto" script (which looks for files to put on the decrypt queue) was spotting the missing "Dec" flag on those that were and putting it back on. Hence explaining the above wacky behaviour.

No idea how I managed to get a duplicated (hard linked) directory - one of which must've got successfully decrypted at some point - but using WebIF to delete the encrypted versions (and emptying the bin for good measure) has put everything back to working as expected. Thanks to @prpr for some pointers.
 
You must log in or register to reply here.
Back
Top