[unencrypt] Decrypt-in-place

[woodbar]

Thanks for the quick reply. Strange, but after I sent the previous it suddenly started working, sort of! I was just about to send an update when I saw your reply.

It suddenly said it was "Watching Masterchef......" which is odd because this was one of the files I had already decrypted using "opt+" - and no, I had already deleted the "originals" sub folder?????

I had a look at the log file and I was getting, what I assume is the correct output, saying it had processed quite a few files then found one to operate on and showed the progress (this was the first of a batch of other "originals" which I got rid of before it scanned again!)

Anyway, I have done the opkg update and upgrade as you suggested and it has started to process a file again, so I am going to leave it alone for a while, have something to eat (might be difficult with fingers crossed!) and see what has happened a bit later.

Thanks again for your efforts.

 

[Ezra Pound]

Have loaded unencrypt 0.1.2b-1 and it now processes (Via cron) both Hi-Def and SD files correctly, However when it got to a 6.2Gig Hi-Def file it appeared to finish (it took 23 Mins) but the 'FLAG' said it was still encrypted, 15 Mins later it tried the same file again and the process hung, The log repeated 'already running' and the files were called 1822.TS and 1822.hmi so it got a bit stuck somewhere.
EDIT:- I think there is a 4Gig limit at the moment 3.7G Hi-Def was O.K. but 6.2G failed

 

[woodbar]

OK, I have not got any files that big yet - I presume you can stop it processing if you toggle the "ENC" flag back on before unencrypt gets to it? Need to be pretty quick I would think.

Also with 0.1.2b-1 and it has processed everything now including a couple of test recordings done earlier today.

Only one anomaly - I was chase playing an ongoing recording of HIGN4U which was in its own directory as I had requested a series. When I stopped the playback there was the recording I had just watched as well as another file, something along the lines of "81.TS" I think - obviously the temporary decrypt file?

I just let it be and exited from "media" to "off air" - I have just come back to the PC and I note that the web-if is just showing the original, unencrypted HIGN4U file?

Had a look at the directory with Filezilla and it also contains "81.TS" and "81.hmi" as well as the 4 normal files + the .series entry - unencrypt does not appear to be "having another go"?

Sorry, FORGET all the above - IT HAS JUST PROCESSED IT - obviously I am too impatient! The file is now showing "DEC" so that is absolutely brilliant!

The "81.hmi" file is still in the directory - will unencrypt "housekeep" this on the next run I wonder?

Anyway, I have ftp'd a load of files over to my 2Tb NAS disk/server and they all appear to play 100% back to the HDR-Fox media player - watched 2 HD programs earlier and no faults, stutters or anything - it seems the Gigabyte network NAS/Switch etc. is doing its stuff even though the link to the Humax etc. is only via 200AV Homeplugs.

Sorry, getting carried away there - but this is now a brilliant method of increasing the storage space or even archiving some recordings - just a few button presses in either web-if or Filezilla and your recordings are on the NAS server, all unencrypted, and ready to go (anywhere!)

Just checked again - "81.htm" is still sitting in the directory - should I be worried about this?

 

[tonycollinet]

Sam - Didn't give you feedback after your help last weekend. Everything now seems to be running fine, and the humax is just getting on and decrypting stuff as it gets recorded. The whole system now works just as I have always wanted*

Huge thanks to you, and the other guys working on this firmware for perfecting the Hummy.

Or at least it was - currently getting DLNA Crashed - will check later when family have stopped watching stuff.

 

[Ezra Pound]

OK, I have not got any files that big yet - I presume you can stop it processing if you toggle the "ENC" flag back on before unencrypt gets to it? Need to be pretty quick I would think.

Just checked again - "81.htm" is still sitting in the directory - should I be worried about this?

You cant stop processing files by toggling the ENC flag because unencrypt needs auto-unprotect to be running and that will just flip it back. If you have setup unencrypt to work only on a sub folder like My Video/archive then move it out of archive to stop it being processed. Unencrypt works by streaming the original file, your file is given a streaming number, in your case '81.TS' plus the smaller 'sidecar' files. If you end up with 81.TS type file the process did not complete and won't be fixed by the next cron. There are some notes on unencypt HERE

 

[4291]

With the latest version of auto-unprotect (1.0.4) any recordings made before the last reboot will be ignored by auto-unprotect so you can then set the ENC flag in the webif if you wish.

Is this not the case anymore?

 

[tonycollinet]

OK, I've rebooted the box, and cron initiated decrypts don't seem to be taking place.

Manually starting them from the command line works though. Any ideas?

humax# ls -l
-rw-rw-rw- 1 root root 566842 Dec 9 15:15 rs.log
-rw-rw-rw- 1 root root 122 Dec 10 00:30 unencrypt.log
humax# tail unencrt.log
tail: can't open 'unencrt.log': No such file or directory
tail: no files
humax# tail unencrypt.log
pscount = 2
/mod/sbin/unencrypt: line 73: netstat: not found
The DLNA server seems to have crashed
Please restart the box

 

[woodbar]

You cant stop processing files by toggling the ENC flag because unencrypt needs auto-undelete to be running and that will just flip it back.

Yes, of course it will - I'm not thinking straight (I know you meant auto-unprotect)

Yes I have read the notes, thanks - including the very helpful (but too late for me - found out the hard way) subnote about unticking the ^M in putty! Oops, just noticed the typo in there as well - auto-undelete should read auto-unprotect I think?

The "81.TS" file has gone now and I am pretty certain that unencrypt did have a second "bite" at it - the "81.TS" was showing in the media player on the TV/HDR-Fox when I stopped play - when I came up to the PC web-if was not "watching" anything and the unencrypt log was not showing any processes - I ftp'd the directory and both the "TS" and "hmi" files were present - it was only whilst I was typing the previous post that I checked again and everything was done correctly except for the remaining "hmi" file - which is still there. Does that seem logical - or should I just pull the plug and go and lie down?

I notice I called the "hmi" a "hmt" file in the previous post - definitely getting a bit of brain-fade here I think! Just found a Telnet client for Android - can use the tablet now if I need to!

 

[Ezra Pound]

Yes, of course it will - I'm not thinking straight (I know you meant auto-unprotect)

Yes Sorry it should have read auto-unprotect, I have corrected it above and on the WiKi. Just a quick note about files bigger than 4Gig. I was not using the seperate WGET 1.12 package, and now that I an I think it fixes big files. A 4.8Gig HD file worked and I will try the 6.1Gig file again soon

 

[Sam Widges]

OK, I've rebooted the box, and cron initiated decrypts don't seem to be taking place.

Manually starting them from the command line works though. Any ideas?

That's a bug that has been fixed in the latest version - 'opkg update' followed by 'opkg install unencrypt' should sort you out.

 

[Sam Widges]

Yes Sorry it should have read auto-unprotect, I have corrected it above and on the WiKi. Just a quick note about files bigger than 4Gig. I was not using the seperate WGET 1.12 package, and now that I an I think it fixes big files. A 4.8Gig HD file worked and I will try the 6.1Gig file again soon

That's a bit strange - the version of wget should be hardcoded and it shouldn't find another one, and that's because the old version of wget in busybox couldn't cope with files over 4G. Let me know how you get on with that big file again: af123 says the wget version in busybox should be fixed now so we may need to revert to that one again.

 

[Sam Widges]

OK, I have not got any files that big yet - I presume you can stop it processing if you toggle the "ENC" flag back on before unencrypt gets to it? Need to be pretty quick I would think.

Also with 0.1.2b-1 and it has processed everything now including a couple of test recordings done earlier today.

Only one anomaly - I was chase playing an ongoing recording of HIGN4U which was in its own directory as I had requested a series. When I stopped the playback there was the recording I had just watched as well as another file, something along the lines of "81.TS" I think - obviously the temporary decrypt file?

I just let it be and exited from "media" to "off air" - I have just come back to the PC and I note that the web-if is just showing the original, unencrypted HIGN4U file?

Had a look at the directory with Filezilla and it also contains "81.TS" and "81.hmi" as well as the 4 normal files + the .series entry - unencrypt does not appear to be "having another go"?

Sorry, FORGET all the above - IT HAS JUST PROCESSED IT - obviously I am too impatient! The file is now showing "DEC" so that is absolutely brilliant!

The "81.hmi" file is still in the directory - will unencrypt "housekeep" this on the next run I wonder?

Anyway, I have ftp'd a load of files over to my 2Tb NAS disk/server and they all appear to play 100% back to the HDR-Fox media player - watched 2 HD programs earlier and no faults, stutters or anything - it seems the Gigabyte network NAS/Switch etc. is doing its stuff even though the link to the Humax etc. is only via 200AV Homeplugs.

Sorry, getting carried away there - but this is now a brilliant method of increasing the storage space or even archiving some recordings - just a few button presses in either web-if or Filezilla and your recordings are on the NAS server, all unencrypted, and ready to go (anywhere!)

Just checked again - "81.htm" is still sitting in the directory - should I be worried about this?

I'm glad it's working for you, at last

81 is the media ID of the file that you needed to request from the DLNA server to get HIGN4U. Unencrypt downloaded the programme as 81.TS and when it had done that successfully it checked that the programme wasn't in use and renamed it to that episode's filename, updating the hmt file at the same time.

From what I can tell, the .hmi file is created when you watch a file that isn't in the media database (there is no 81.TS in the database, the database just uses the number 81 as the index number for the real filename). The .hmi file saves where you were so that it can resume play later, so I suspect that you played the file as it was downloading and the .hmi file can safely be deleted.

 

[xyz321]

Just a quick note about files bigger than 4Gig. I was not using the seperate WGET 1.12 package, and now that I an I think it fixes big files. A 4.8Gig HD file worked and I will try the 6.1Gig file again soon

The latest busybox version of wget seems to be fine with the DLNA server. However, the Humax DLNA client appears to be broken for large files so it is probably best to use CIFS or NFS when playing back from a NAS server - see this thread.

 

[Sam Widges]

Yes Sorry it should have read auto-unprotect, I have corrected it above and on the WiKi. Just a quick note about files bigger than 4Gig. I was not using the seperate WGET 1.12 package, and now that I an I think it fixes big files. A 4.8Gig HD file worked and I will try the 6.1Gig file again soon

Let us know how you get on because Black Hole seems to have shown that there's another 4G bug in the DLNA server and if that's true, it will almost certainly break decrypt-in-place and the webif decrypt.

 

[xyz321]

Let us know how you get on because Black Hole seems to have shown that there's another 4G bug in the DLNA server and if that's true, it will almost certainly break decrypt-in-place and the webif decrypt.

Fortunately the bug is in the Humax client, not the server.

 

[Ezra Pound]

Let us know how you get on because Black Hole seems to have shown that there's another 4G bug in the DLNA server and if that's true, it will almost certainly break decrypt-in-place and the webif decrypt.

I still think there is problem with big files, I take your point about wget not being the problem, I now think it may be a time problem, a 4.8Gig file went through in less that 15Mins. but the 6.2Gig one gets to 100% done in 20Mins or so, Obviously another cron tried to start before it got to 100%. If you examin the Flag it is still at encrypted and I think the first run never completes, so on the next cron it has another attempt which also doesn't finish properly, so the 6.2Gig file gets worked on multiple times. here is a ps showing two processes for the same file :-

21132 root      1240 S <  /bin/sh -c /mod/sbin/unencrypt "/mnt/hd2/My Video/ar
21133 root      1292 S <  {unencrypt} /bin/sh /mod/sbin/unencrypt /mnt/hd2/My
24518 root      1248 S <  /bin/wget http://127.0.0.1:9000/web/media/1852.TS
24689 root      1240 S <  /bin/sh -c /mod/sbin/unencrypt "/mnt/hd2/My Video/ar
24691 root      1292 S <  {unencrypt} /bin/sh /mod/sbin/unencrypt /mnt/hd2/My
28103 root      1248 S <  /bin/wget http://127.0.0.1:9000/web/media/1852.TS
28162 root      1332 R    ps

 

[Sam Widges]

Fortunately the bug is in the Humax client, not the server.

Confirmed - I've just successfully decrypted a 6.8G HD file and it worked fine.

 

[Sam Widges]

I still there is problem with big files, I take your point about wget not being the problem, I now think it may be a time problem, a 4.8Gig file went through in less that 15Mins. but the 6.2Gig one gets to 100% done in 20Mins or so, Obviously another cron tried to start before it got to 100%. If you examin the Flag it is still at encrypted and I think the first run never completes, so on the next cron it has another attempt which also doesn't finish properly, so the 6.2Gig file gets worked on multiple times. here is a ps showing two processes for the same file :-

21132 root      1240 S <  /bin/sh -c /mod/sbin/unencrypt "/mnt/hd2/My Video/ar
21133 root      1292 S <  {unencrypt} /bin/sh /mod/sbin/unencrypt /mnt/hd2/My
24518 root      1248 S <  /bin/wget http://127.0.0.1:9000/web/media/1852.TS
24689 root      1240 S <  /bin/sh -c /mod/sbin/unencrypt "/mnt/hd2/My Video/ar
24691 root      1292 S <  {unencrypt} /bin/sh /mod/sbin/unencrypt /mnt/hd2/My
28103 root      1248 S <  /bin/wget http://127.0.0.1:9000/web/media/1852.TS
28162 root      1332 R    ps

The code is designed to not run multiple parallel copies but cron does strange things with the process count and it is possible that a second copy of unencrypt could run in certain circumstances.

I'm beginning to wonder if there is a fault in this file because I have just decrypted a 6.8G HD file on my own system.

Could you view the unencrypt.log file via the diagnostics page on webif and paste the top and bottom sections?

 

[Ezra Pound]

Could you view the unencrypt.log file via the diagnostics page on webif and paste the top and bottom sections?

Unfortunately I have moved the remains of the 6.2Gig files out of my working directory "My Video/archive" (it had an encrypted TS file with no sidecar files )
and as time has moved on all I get in the log now is this (Telnet view and webif are the same) :-

humax# cd /mod/tmp
humax# cat unencrypt.log
pscount = 4
Unencrypt process already running - exiting
humax#

 

[xyz321]

From the ps listing above it is not possible to determine which files are being processed by the shells ('ps -w' would have shown this). However, it appears that the script is checking for duplicate sub-shells but only at startup. Would it be better to check for duplicate parents to ensure only one can run at a time?

A simpler alternative might be to create a lock file in /tmp and just exit the parent shell if it exists.