Windows version of offline decryption (HFODU)

[Black Hole]

Makes no sense to use Linux line ends in Windows output.

Unless, like me you, have a text editor which can cope with both forms, and save in either as required

That's a work-around, not joined-up thinking.

Until the program defect is corrected that causes it, users could run a Unix-to-DOS utility on the file before opening it. The discussion HERE (click) offers a variety of options, I would probably code something in AWK*.

* AWK is designed for text processing. I can't remember the syntax off the top of my head, but an AWK program to do this would go something like:

Set input record delimiter = "LF"​

Set output record delimiter = "CR LF"​

Copy input records to output until end of file​

The AWK call can then be wrapped in a .BAT to hide the details away, which (if suitably tweaked) can even be drag-and-drop (I have done this with another conversion I do regularly). So, in Explorer, with "output.log" and "unix2dos.bat" files in the folder, you would just drag output.log onto unix2dos.bat and bingo, output.log.txt would appear and be openable in normal Windows utilities.

Is it worth me doing this? Probably not, anyone wanting to use it would need not only the unix2dos.bat file and the .awk program but the gawk.exe AWK interpreter as well. A compiled unix2dos.exe would be much better (and is almost certainly out there somewhere).

 

[dandnsmith]

That's a work-around, not joined-up thinking.

Using a tool which requires that much effort is a really clumsy work-around - and may require much training to get working.
Any such approach is, perforce, a work-around, anyway.

 

[Black Hole]

I never said a Unix-to-DOS utility isn't also a work-around (but I don't see how drag&drop requires any extra training beyond normal Windows UI operations). The problem needs fixing in the originating program. Work-around are only appropriate (a) as a temporary fix, or (b) when there is no likelihood of fixing the originating program.

 

[af123]

The problem needs fixing in the originating program.

It /is/ fixed - https://hummy.tv/forum/threads/windows-version-of-offline-decryption-hfodu.8770/post-124855

 

[EEPhil]

Alright everybody - especially Black Hole - I hold my hands up. I was a little lax in the testing. I'm used to using Notepad++ not the editors supplied with Windows. I tested the log with Notepad++ and not Notepad or Wordpad. Notepad++ is a bit more forgiving and treats "\n" in C/C++/Java like CR/LF so I didn't spot that it hadn't worked. I've become so used to that behaviour that I forgot to use "\r\n" for Windows.
For the record - I was made aware of the mistake at 2.45pm on 1/9/19 and put the correction out there at 3.11pm on the same day. So BH's post at 7.47am 2/9/18 (#61) whilst valid misses the point.

Until the program defect is corrected that causes it,

It /is/ fixed

The only way I could have corrected it any faster would have been not to make it in the first place.

 

[Black Hole]

Please don't take what I say as criticism or complaint - I think at the overall philosophical level.

 

[prpr]

I've become so used to that behaviour that I forgot to use "\r\n" for Windows

I don't know anything about Java, but it seems a bit crap for it to foist this task on to the programmer.
If you did printf("\n"); in C then you would get \n on Linux and \r\n on Windows as the run-time library does the necessary translations so that you can write cross-platform source code. Likewise for all the other functions that do anything with line endings.
Java is supposed to be cross-platform from the outset, so it's kinda embarrassing if you need to use different code on different platforms.
No wonder one of my ex-colleagues called it "a donkey" if this is how it works.

 

[EEPhil]

I don't know anything about Java,

Yes, we know your opinion of Java.
I would have thought the language might have sorted the CR/LF properly. It's possible I screwed up. I need to check whether this is just a result of me overloading the log formatter or some general problem (I used a logger to produce the output log ( ), but by default it includes various bits of like method names. So I messed with the formatter to remove them. I'll even go as far to point out some more of my lazyness here. I'd already used this formatter for my 2000T remote access program. It was tested and worked there - using Notepad++ to read the output. I never thought... !)

Edit:
It is my fault. The preferred way to indicate end of line in Java is %n not \n. %n is, it is claimed, a platform independent way of providing the correct end of line sequence. News to me. But I'll file that one away for future use.

 

[EEPhil]

Please don't take what I say as criticism or complaint - I think at the overall philosophical level.

None taken.

 

[sceptic]

@EEPhil Many thanks for HFODU2. Its nearly perfect for my requirements now, the only thing I'd like to seen included was subdirectory scanning support which presumably would also require the inclusion of .hmt checking (if present) to skip any files already decrypted, If not then I could always code my own scanner to locate any encrypted files and shift them into a single directory for HFODU2 decryption I guess...

 

[EEPhil]

@sceptic Subdirectories of subdirectories? How deep do you want to go? (A better question might be, how many files are you trying to decrypt?)
BTW it doesn't scan .hmt files, but does change them if they exist. To be honest, the current version doesn't actually check that the file has already been decrypted. It tries to decrypt it, finds that decrypting a decrypted file causes the CRC comparison to fail, and gives up.

Edit:
You should be able to go into the top directory containing the subdirectories and select multiple directories (hold down control key and select each directory). Does that do what you want?

 

[sceptic]

Edit:
You should be able to go into the top directory containing the subdirectories and select multiple directories (hold down control key and select each directory). Does that do what you want?

Yes, but I do wonder how long it might be running for then if it does not check the .hmt flag as I think there are probably in excess of 20TB of recordings in my archive! I suspect only a couple of recordings out of that lot are actually still encrypted.

 

[EEPhil]

The .ts files are checked until it finds the so-called PAT packet(s). They occur quite frequently within the file. I'd be very surprised if it took as long as the first minute of a file of any length to determine this. Not sure how long this takes to process - that is dependent on your computer. On my slow laptop it took less than one second to determine that files were/were not encrypted.

 

[EEPhil]

I think there are probably in excess of 20TB of recordings in my archive!

I'll pre-empt some comments by BH . (Consider any questions rhetorical). Why so many? You are never going to watch all those again. It's only television. Have a good clear out. Why didn't you decrypt the files as you archived them? etc. etc.

 

[EEPhil]

Following on (sort of) from ...

If you did printf("\n"); in C then you would get \n on Linux and \r\n on Windows as the run-time library does the necessary translations so that you can write cross-platform source code. Likewise for all the other functions that do anything with line endings.

Something I found on the web at https://www.horstmann.com/ ...
---
The March of Progress
1980: C printf("%10.2f", x);
1988: C++ cout << setw(10) << setprecision(2) << fixed << x;
1996: Java

java.text.NumberFormat formatter = java.text.NumberFormat.getNumberInstance(); 
formatter.setMinimumFractionDigits(2); 
formatter.setMaximumFractionDigits(2); 
String s = formatter.format(x); 
for (int i = s.length(); i < 10; i++) System.out.print(' '); 
System.out.print(s);

2004: Java System.out.printf("%10.2f", x);
---
Think I might even go "ee-aw" at the 1996 code.

 

[prpr]

Oh, so that's how you output formatted stuff in C++.
I don't bother because it was just too damned complicated to work out and instead use your 1980 C example. I now see I'm still right to do so.
And I know extra care has to be taken making sure the format string matches, but it's a price worth paying to save me from the other misery.

 

[sceptic]

I'll pre-empt some comments by BH . (Consider any questions rhetorical). Why so many? You are never going to watch all those again. It's only television. Have a good clear out. Why didn't you decrypt the files as you archived them? etc. etc.

It would not be the first time I have heard comments along those lines! For the record and BH I never watch anything live and everything I do record gets dumped onto my NAS archive of TV and movies. Whenever I do get around to watching something I just have a look in there and pick whatever takes my fancy. Some stuff I wipe after watching but others I keep as I may (and often do) end up watching it again at some point. My thinking is if you have the space why not keep it! All the files are decrypted I think apart from just a few that were copied off before auto-decrypt had run successfully...

 

[sceptic]

The .ts files are checked until it finds the so-called PAT packet(s). They occur quite frequently within the file. I'd be very surprised if it took as long as the first minute of a file of any length to determine this. Not sure how long this takes to process - that is dependent on your computer. On my slow laptop it took less than one second to determine that files were/were not encrypted.

Ah OK, sounds like performance should not be an issue then I'll do some tests and once I'm happy I'll run it across my NAS to see what it finds and report back with the results. Thanks again for making these enhancements.

 

[lashers]

Thanks for this great utility, which I've been trying on some undecrypted offline backups. Unfortunately, following a recent Java update (from 1.8.0.181 to 191) the file selecter no longer works properly; you can go up the directory tree but it takes the first directory you select thereafter as your final selection and fails if there are no .ts files in it.

Let me take this opportunity to give you some performance data; running it under 64-bit Windows 10 on a Skylake i7-6700K processor with 16 GB of RAM gives a throughput of about 36GB per hour. As a comparison, standard auto-decrypt on the Humax does about 30GB per hour. I hope this is useful for you.

 

[EEPhil]

Unfortunately, following a recent Java update (from 1.8.0.181 to 191) the file selecter no longer works properly; you can go up the directory tree but it takes the first directory you select thereafter as your final selection and fails if there are no .ts files in it.

This is going to require some thought. I'm working on an old PC and don't always update Java - I keep getting warnings that it isn't meant for my version of Windows. I'll see if I can load 191 or 192. Your description of the problem suggests it's a programming fault . But it could be a change in some library within Java. If it is I'm going to start agreeing with prpr's opinion of Java.

Did you ever select a directory with no .ts files in before the update? When you say "fail", what happens? Nothing? An error message on screen or an error message in the log file?

Now, where did I put my testing filestore ...?