[unencrypt] Decrypt-in-place

[Sam Widges]

I've just finished writing the first cut of a simple script that has rescued the filesystem of a failing HDR-T2 onto an external hard drive (http://hummy.tv/forum/threads/backing-everything-up.697/) and while that was being discussed, it looks like it would be very easy to tweak the script to do decrypt-in-place. In much the same way that auto-unprotect works its way through the filesystem, this script would decrypt the files on the hard drive and update the sicedar files so that there was no apparent difference but FTPing or sharing the files would become much simpler.

I'm thinking of 2 ways that the script would run:
1) Big Bang - set it running and wait a day or so until it completes, and
2) Ongoing updates - this would be run once an hour and would work through batches of files, maybe 5 at a time.

The idea would be that you run the script in "Big Bang" mode once and, after that, you just rely on the background version to keep everything up-to-date.

You would need to be using the modified firmware to run this, but the idea is to make it as simple as possible.

Are there any requests or suggestions that would be worth considering before I start tweaking? Please don't ask for webpage integration - af123 might do that if he's kind enough, but it's outside of my control.

 

[af123]

webif 0.8.0 has experimental decrypt-in-place support for single files through the media browser - click on the OPT+ button next to a recording but don't pick something you care about at this stage. Any comments to the webif thread though please, I don't want to hijack this one.

A script that runs in the background and watches for new recordings to appear would be a very useful addition to the custom firmware for people who want their recordings to be decrypted automatically. It would certainly make backing up the disk a lot quicker if/when it becomes necessary. Maybe hooking it into inotify is the way to go but you'll have to wait for the DLNA server to index new recordings and I don't know exactly when that happens.

The latest version of the hmt utility indicates whether a file is encrypted on disk via the ODEncrypted flag - that might come in useful for you when deciding what still needs processing.

humax# hmt 9_22._Mr_Saturday_Knight.hmt  | grep '^Flags'
Flags: SD,New,Unlimited Copies,Guidance,ODEncrypted,
humax# hmt 10_22._Fish_out_of_Water.hmt | grep '^Flags'
Flags: SD,Unlimited Copies,Guidance,

 

[Sam Widges]

webif 0.8.0 has experimental decrypt-in-place support for single files through the media browser - click on the OPT+ button next to a recording but don't pick something you care about at this stage. Any comments to the webif thread though please, I don't want to hijack this one.

A script that runs in the background and watches for new recordings to appear would be a very useful addition to the custom firmware for people who want their recordings to be decrypted automatically. It would certainly make backing up the disk a lot quicker if/when it becomes necessary. Maybe hooking it into inotify is the way to go but you'll have to wait for the DLNA server to index new recordings and I don't know exactly when that happens.

The latest version of the hmt utility indicates whether a file is encrypted on disk via the ODEncrypted flag - that might come in useful for you when deciding what still needs processing.

humax# hmt 9_22._Mr_Saturday_Knight.hmt  | grep '^Flags'
Flags: SD,New,Unlimited Copies,Guidance,ODEncrypted,
humax# hmt 10_22._Fish_out_of_Water.hmt | grep '^Flags'
Flags: SD,Unlimited Copies,Guidance,

That's really useful - thanks!

 

[SiHa]

That would be useful, it would certainly make copying files off much quicker as the decrypt-on-the-fly really slows down transfer speeds.

That said, would I really want to deccrypt everything (2 x HDD wear) when I'm only likely to want to back up a very small proportion of the recordings? Probably not. And if a single-file decrypt-in-place is available via web-if, then I'd probably stick with that.

 

[xyz321]

From the other thread...

I've had a look at auto-unprotect, but it'll probably be easier to just tweak the existing code to work alongside it because it only seems to run the unprotect script on HD files. If that could be extended to SD content as well, then it would be possible to merge both scripts.

At the moment the unprotect script is run on both HD & SD files. I am hoping to release a new version of auto-unprotect in the next few days (this should fix a bug where the flag is not cleared in chase play mode where the user doesn't press stop). The plan was to only run the unprotect script only on HD files when it was really necessary but it could be enabled for SD as well.

Both versions call up the unprotect more than once for the same file. The new version will minimise this as much as possible but it could still be called up to about six times. The important call would be the last so it would have to abort any transfers that were kicked off by previous calls. The main problem is that the setting of the encryption flag as seen in the hmt file on disk doesn't necessarily reflect the internal state of the humaxtv application.

It is not easy to monitor the state of the DLNA server since the db file is never closed. Perhaps the best way to go would be to trigger a transfer from the unprotect script but with a suitable delay. If new events occur during the delay period then the original invocation is aborted. Ths should also help where the box powers down immediately after the recording has ended.

You would also need something to restart/resume aborted transfers on power up. A queuing system might also be needed since two recordings could potentially finish around the same time.

Perhaps this is all too complicated and a cron job would be best for the copy process.

 

[oijonesey]

Only just seen this so quick suggestions for now:

- Maybe only do this on nominated folders (to the point about not needing to do everything)
- Set it to run for a time span i.e. for a few hours through the night (assuming auto power offs not in the way)
- Also saw the new stuff about editing on the box so I hate to say it I now may not need it at all!! I wanted to decrypt on the box so that whenever I wanted to edit I could just drag stuff off the box en mass rather than copy to USB first or hit download one file at a time. However I might still do editing off the box because even though I think the new function is brilliant, off the box editing doesn't get in the way of watching TV. To be honest it's all so new I haven't exactly worked out how I'm going to work with it all yet!!!!!

 

[Sam Widges]

@oijonesey

So as to not risk destroying the files on my box, I'm implementing 2 features, just for testing
1) It will only process a folder that you specify on the command line
2) It will only process the first unencrypted file it finds

This will then run very neatly from cron, so you could set it up to run once every hour (or possibly half hour) overnight. It will then gradually chew its way through all the files on the box, decrypting them.

Like you, I love the on-box editing stuff - drutt rulez!

 

[Sam Widges]

I'm just doing some testing and it's looking like a beta version can be released soon. Before I do, is anyone who is interested in this script worried about auto-decrypting SD content, or is it just the HD content that is important?

 

[keytouch]

I'd like all content to auto decrypt so that I can map the drive to my HD fox and watch all the files so that files can be resumed etc.

 

[Sam Widges]

Thanks for getting back - that's one of the ways that I hoped the script would be used. Once everything is packaged up, would you be happy editing a file to choose between HD or Both?

 

[Black Hole]

I'd like all content to auto decrypt so that I can map the drive to my HD fox and watch all the files so that files can be resumed etc.

That's an interesting idea, and would suit my set-up. I will be looking to implement it when it has come off the bleeding edge.

Thanks for getting back - that's one of the ways that I hoped the script would be used. Once everything is packaged up, would you be happy editing a file to choose between HD or Both?

I don't understand the benefit of only doing one or the other. In my case, I mostly record in StDef but have a few highlights in HiDef. Naturally I want to manipulate them all the same. Somebody with a keener interest in HiDef will have the opposite balance, and will also want to manipulate them all the same. Am I wrong?

 

[Trevor Jasper]

Decrypted a SD file and have the nice "DEC" icon to indicate it's been decrypted - nice touch. However, just tried to use the webif to right-click -> download on the decrypted file and it's only offering the "xxxx.ts" numeric file name from the database i.e. it'll download with decypting on the fly, not the new file. As it is now identified as decrypted, can the download option just offer to copy the new file as is, please.

 

[Sam Widges]

I'm afraid that's a question about the webif and I can't answer for af123.

 

[Trevor Jasper]

Sam,

I meant it as a question for af123, hoping he was reading this thread. Will post above text in webif thread. Posted here to record success on decryption in place. Thanks for your sterling efforts on this Sam, and would love to have your back everything up script in a package accessible from the webif to back up marked files. Is that possible?

I don't know.... give people an inch and they'll take a mile, or at least give them an idea and they'll ask for miles more ideas!!!!!

 

[Black Hole]

just tried to use the webif to right-click -> download on the decrypted file and it's only offering the "xxxx.ts" numeric file name from the database i.e. it'll download with decypting on the fly, not the new file.

I think this stems from the database taking a while to catch up. The download option uses the DLNA service, and that takes its folder and file information from the SQL database and not the OS file system. The database gets corrected to reflect disk changes in slow time.

 

[Trevor Jasper]

Just tried FTPing versus the download via the webif click download. The decrypted file and an encrypted file FTP'd off at 2.58MB/s and 2.68MB/s respectively. Using the webif download, the decrypted xxx.ts file downloaded at 2.6MB/s versus the encrypted file at 2.5MB/sec with on-the-fly decryption. So on my network, it seems that the decryption doesn't really slow down the transfer. That isn't what I was expecting. Do others have faster network transfer speeds? I Have homeplugs to the router and wifi to the laptop.

I have confirmed that of the ftp'd files, the decrypted one played and the encrypted one didn't, whereas both files downloaded from the webif played.

So maybe the webif download does truly reflect the updated db file of the in-place decrypted file, but it makes no difference to the download rate whether the file is decrypted - not what I had expected. What would matter is whether I remote shared the files to a laptop and streamed them without copying off first.

Can the webif save with the recorded filename rather than the xxx.ts name?

 

[Black Hole]

The encryption/decryption is performed in hardware (as far as I know), so it shouldn't make too much impact on data rate. It must at least be capable of real-time HiDef, which is about 1.5MB/s, x2.

 

[Sam Widges]

Sam,

I meant it as a question for af123, hoping he was reading this thread. Will post above text in webif thread. Posted here to record success on decryption in place. Thanks for your sterling efforts on this Sam, and would love to have your back everything up script in a package accessible from the webif to back up marked files. Is that possible?

I don't know.... give people an inch and they'll take a mile, or at least give them an idea and they'll ask for miles more ideas!!!!!

It's probably better to post this sort of request on a thread about webif - I try to not assume that everyone reads every thread otherwise we'd not get on with any of the actual development . If af123 wants to convert the backup script to jimsh (the webif programming language), I'll happily contribute.

One of the things that I love about this sort of forum is that people who initially 'scratch an itch' chuck out their code to all and sundry and features get added that people actually want, not just what the manufacturers decide that we should want.

 

[Trevor Jasper]

And I'm glad you have thrown out some of your stuff. Scratched a few people's itches, I bet. Have posted request to the other thread, so we'll see what happens. The decrypt-in-place is SUCH a cool feature and now people need not fear losing the box (warranty replacement or whatever) and thus losing their recordings.

Thanks to Humax for not absolutely crippling the device so that this is all possible. The power of the GPL, I hope.

 

[Trevor Jasper]

The encryption/decryption is performed in hardware (as far as I know), so it shouldn't make too much impact on data rate. It must at least be capable of real-time HiDef, which is about 1.5MB/s, x2.

I does seem to be quick enough. I thought I'd read that the decryption seemed to slow down people's wget transfers? Maybe I'm mis-remembering. Whatever, it fine for me.