Scripting HD backups in linux

bash_backup

New Member
These are just some notes detailing some recent experimentation - hopefully some of you may find them useful. Probably all of this functionality is available with the custom firmware - but while the box is still in warranty not everybody may be ready to take the plunge...

While it is possible to remove the "ENC" flag from *.hmt files in linux using foxy running under mono it is quite a manual process - something like:

1. ftp into your Humax using FileZilla, or similar.
2. copy the hmts onto your local machine.
3. run foxy and then drag and drop each hmt in turn for conversion.
4. copy the hmts back using Filezilla again.

If you have a lot a files this can take a while, and so it would be nice to accomplish all of this with a single command. Here's how.

Step one:

Build yourself a command line foxy equivalent. It seems that it is only the bit at 0x3DC that needs to be updated to get your Humux to decrypt HD recordings for you, so:

touch prog.cpp

then open the file in a text editor and paste in:

---
Code:
#include <iostream>
#include <fstream>
#include <cstdlib>
#include <string>
 
using namespace std;
 
void start_sm();
void usage();
 
int main(int argc, char** argv)
{
start_sm();
if(argc != 2)
{
usage();
return 1;
}
 
string input_file = &argv[1][0];
fstream in;
in.open(&input_file[0], fstream::in | fstream::out | ios::binary);
 
if(!in)
{
cout<<"Unable to open: "<<input_file<<endl;
return 1;
}
unsigned int val;
unsigned int DC = 0x04;
 
cout<<"Input file: "<<input_file<<" opened OK."<<endl;
in.seekg(0x3dc);
val = in.get();
cout<<"Encryption bit:\t"<<val<<endl;
 
if(val != 0x04)
{
cout<<"File has encryption..."<<endl;
cout<<"Overwrite encryption bit.."<<endl;
in.seekp(0x3dc);
in.put(DC);
}
else
{
cout<<"No file encryption..."<<endl;
}
 
 
in.close();
cout<<"\n\tDone!"<<endl<<endl<<endl;
return 0;
}
void start_sm()
{
cout<<"Foxy_DC called"<<endl;
return;
}
 
void usage()
{
cout<<"***********"<<endl;
cout<<"usage: please call Foxy_DC with a single argument"<<endl;
cout<<"that argument should be the name of the file to process"<<endl;
cout<<"***********"<<endl;
 
}
---

save it, and compile with:

g++ prog.cpp -o Foxy_DC

If that works, you should be able to run it with:

./Foxy_DC <your_filename>

Step two:

It turns out you'll need another command line tool later too, so you may as well build that now:

touch split.cpp

and paste in:

---
Code:
#include <iostream>
#include <fstream>
#include <cstdlib>
#include <string>
 
using namespace std;
 
void usage();
 
int main(int argc, char** argv)
{
if(argc != 3)
{
usage();
return 1;
}
 
string input_file = &argv[1][0];
fstream in;
in.open(&input_file[0], fstream::in | fstream::out);
 
if(!in)
{
cout<<"Unable to open: "<<input_file<<endl;
return 1;
}
 
string output_file = &argv[2][0];
ofstream out;
out.open(&output_file[0]);
 
if(!out)
{
cout<<"Unable to open: "<<output_file<<endl;
return 1;
}
 
string line_in, line_out;
while(!in.eof())
{
getline(in,line_in);
if(line_in.length())
{
line_out = line_in.substr(44);
cout<<line_out<<endl;
out<<line_out<<endl;
}
}
 
in.close();
out.close();
return 0;
}
 
void usage()
{
cout<<"***********"<<endl;
cout<<"usage: please call Split with <inflie> <outfile>"<<endl;
cout<<"***********"<<endl;
 
}
---

And complie with:

g++ split.cpp -o splitter

This is a nasty hack to get the file names of your recordings into a more useful format.

Step three:

Automate pulling your hmts off of your box. You'll need a bash script for this, so:

touch ftp.sh

Then open the file with a text editor and paste in this lot:

---
Code:
#! /bin/bash
# Humax FTP details
readonly IP='xxx.xxx.xxx.xxx'
readonly user='humaxftp'
readonly pass='0000'
readonly DIR='My\ Video'
#
clear
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo HUMAX FTP ENC REMOVER
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo
echo Connecting to FTP server:
echo IP- address $IP
echo User- $user
echo Password- $pass
echo Directory- $DIR
echo
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo Get File list from FTP server
# Heredoc to get file list and write to local file
ftp -n -v $IP << End-of-session
user $user $pass
cd $DIR
dir > Dir.txt
bye
End-of-session
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo Process File list
# ignore anything that isn't a hmt_file
grep hmt Dir.txt > hmt_files_long.txt
# clean up
rm Dir.txt
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo parse list
# Get only the filenames
./splitter hmt_files_long.txt hmt_files.txt
# clean up
rm hmt_files_long.txt
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo start downloads
# Build a todo list to download the hmt files from the box
echo >todo.lst user $user $pass
echo >>todo.lst 'cd' $DIR
while read N
do
echo >>todo.lst get \"$N\"
done <hmt_files.txt
echo >>todo.lst bye
# Download hmt files...
ftp -n -v $IP <"todo.lst"
#files downloaded,
#cleanup
rm todo.lst
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo Archive hmt files
for file in *.hmt
do
cp "$file" "Archive/$file"
done
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo process downloads
#now process the downloaded files..
for file in *.hmt
do
./Foxy_DC "$file"
done
echo Files processed, put them back on the Humax...
# make the new todo
echo >todo.lst user $user $pass
echo >>todo.lst 'cd' $DIR
while read N
do
echo >>todo.lst put \"$N\"
done <hmt_files.txt
echo >>todo.lst bye
#upload them again
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo upload Files
ftp -n -v $IP <"todo.lst"
#final cleanup
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo Clean up files
rm todo.lst
rm hmt_files.txt
rm *.hmt
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo DONE
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
---

You'll need to make this script executable on your system. You'll also need to modify the IP address of your box. The DIR can be up dated to 'My\ Video/series_name' if you want to run the script on a different directory.

Step 4

In whatever directory you have these files:

mkdir Archive

so you have somewhere to backup the unmodified hmts (just in case).

Step 5

./ftp.sh

Step 6

Check all your HD recordings are now ready to backup....

Hopefully this will be a quicker process. Like I said - just some notes from playing around yesterday evening.
 
That's great... but you could just install the custom firmware and remove the ENC flag automatically. I suggested some time ago that Foxy could have FTP capability built in; this appears to be an answer and it would be nice to have it as a compiled program for Windows as a one-click "unprotect all" utility. Note that this method of unprotecting will not render HiDef recordings available for playback by streaming (except to another HD-FOX or HDR-FOX, which do not require any form of unprotection), while the custom firmware auto-unprotect package will.

I appreciate some people are nervous about modifying their HDR-FOX within warranty, but first we have not heard of any case where the unit was actually bricked as a result (there are a couple of instances where the unit was bricked by downgrading from standard firmware 1.03.06 to 1.02.32 prior to installing custom firmware - see Things Every... (click) section 1), and second the custom firmware can be removed again in the event of a warranty return.

With skills such as these, I am sure you will find much in the custom firmware (and software) to amuse you (and perhaps contribute to).

It should be noted that in this section:
Code:
readonly IP='xxx.xxx.xxx.xxx'
readonly user='humaxftp'
readonly pass='0000'
...the "xxx.xxx.xxx.xxx" should be replaced with the specific IP address for the user's Humax, as displayed at Menu >> Settings >> System >> Internet Setting >> Configure LAN >> IP Address, and "0000" is the system PIN (which is "0000" by default but may have been changed at Menu >> Settings >> Preferences >> Parental Control >> Change STB Password).

PS: if you edit your post to put the code sections inside [code]...[/code] tags, you will prevent specific character combinations being interpreted as emoticons. Test:-

Without code tags:

:eek:ut

With code tags:

Code:
:out
 
PS: if you edit your post to put the code sections inside [code]...[/code] tags, you will prevent specific character combinations being interpreted as emoticons. Test:-

Without code tags:

:eek:ut

With code tags:

Code:
:out
I was wondering about the relevance of the "Eek!" :eek: emoticons.:D
 
Thanks for the tips on the markup.

I'm certainly not disputing that the custom firmware would do this all for me (hence the second sentence in the post). And I'm sure I'll move over to it in the future. It is just that my current preference is not to make changes to the hardware. And being able to schedule this script gives me the key functionality I'd want from the custom firmware (the automatic removal of the ENC flag) on a standard box.

It may be that there are others who'd like to stick with the standard hardware for a while, and if so, then this may be of use to them.

I'm not making any quality assessments of either approach.
 
Since the code sections have been added to post #1 the :eek: emoticons have gone, but have been replaced by their text equivalent ;eek; (semi-colons used here instead of colons)
Step one includes
Code:
in.open(&input_file[0], fstream::in | fstream::eek:ut | ios::binary);
Step two includes
Code:
in.open(&input_file[0], fstream::in | fstream::eek:ut);
Should they be replaced by other text?
 
That's very interesting! It reveals that the forum software does not simply interpret the text that is present and display an emoticon, but actually substitutes the "primary" text code for any synonyms. In this case, ":o" has been switched to ":eek:", so all occurrences of the string ":eek:" need replacing with ":o" in the original post.

Curiously, when I go to edit my post that includes the :eek:ut string, it still displays as ":out" in my edit window (I use BBCode mode editing, enforced on iOS but available to others as the button at the right of the edit toolbar). This suggests a slightly more complicated scenario: the forum software does just interpret what's there, but the WYSIWYG editor performs some kind of substitution.
 
Hi bash_backup, good stuff, I would have to say that most users capable of compiling / running these routines are the kind that are more willing to install the Custom Firmware. As far as the warranty goes, it is possible to remove the Custom Firmware prior to sending the unit back in all cases except where it is impossible to write to the hard disk, no problems in this area have been encountered to date.

Please don't take this the wrong way, I think you could contribute greatly to this product, keep up the good work :)
 
Since the code sections have been added to post #1 the :eek: emoticons have gone, but have been replaced by their text equivalent ;eek; (semi-colons used here instead of colons)
Step one includes
Code:
in.open(&input_file[0], fstream::in | fstream::eek:ut | ios::binary);
Step two includes
Code:
in.open(&input_file[0], fstream::in | fstream::eek:ut);
Should they be replaced by other text?
Yes - they have been updated.

Thanks
 
Back
Top