[progbackup] Backing everything up

Sam Widges · Nov 1, 2011

OK, this is hot off the presses and is very much work in progress...

My HDR-T2 has been on the fritz for a while now and several posts on the forum make me think that the hard drive is on the way out. Apart from the fact that we only bought the dratted thing in February, how do we go about rescuing the contents before it goes back for repair?

Well, the answer seems to be that we need:
1) Custom firmware
2) Auto-unprotect installed and running
3) An external hard drive with enough space that you can write to
4) A cunning script to save you lots of time

Assuming that you can sort steps 1-3, here's my first cut at step 4:

progbackup.sh

Code:

#!/bin/sh

echo "Humax HDR-T2 backup utility"
echo
echo "Decrypts and backs up your videos to a local drive of your choosing."
echo "Other content will not be encrypted and can therefore be transferred off separately."
echo "HD content will not be decrypted unless you have cleared the encryption flag,"
echo "so, the script will exit if 'auto-unprotect' is not running"
echo

IFS=$'\n' #You need this or the for loop gets confused by spaces

host='127.0.0.1' # Yay localhost works!

used=`df -m | sed 's/ */ /g' | grep /mnt/hd2 | cut -f 4 -d\ `

echo "Examining your local drive, you need up to $used MB for the backup"
echo "Attached drives have the following free space (MB):"

for freespace in `df -m | sed 's/ */ /g' | grep /media/drive | cut -f 5,7 -d\ `
do
        echo "$freespace"
done

target=""
answer="N"

while [ "$answer" != "y" ]
do
        result=""

        while [ "$result" = "" ]
        do
                echo "Which drive do you want to back up to? (CTRL-C to abort)"

                read target

                result=`df -m | grep $target`

                echo "You have selected $target"
                if [ "$result" = "" ]; then
                        echo "That is not a valid drive"
                fi

        done

        targetspace=`df -m | sed 's/ */ /g' | grep $target | cut -f 5 -d\ `
        echo "$target has $targetspace MB free and you need up to $used MB"
        echo "Do you want to use this drive? [y/N]"
        read answer
done

echo "Backing up to $target"

# What if someone has plugged in an NTFS drive and it's mounted read only?
# Check for Fat32/ntfs or check for R/W, or both?

#You really don't want to run this if you have HD contect and you aren't running auto-unprotect
AP=`opkg list-installed | grep auto-unprotect`

if [ "$AP" = "" ]; then
        echo "Auto-unprotect isn't running, you don't want to download encrypted HD files"
        echo "Install Auto-unprotect and wait 2 hours before rerunning"
        echo "If you do not wait 2 hours, you could lose access to some HD files"
        exit
fi

#How many programmes are there to backup?
lines=`sqlite3 /mnt/hd2/dms_cds.db "select mediaID,folder from tblMedia" | grep -v '|(NULL)' | wc -l`

echo "There are $lines programmes to backup"

#Should we have a pause here to confirm the target destination and whether or not to proceed?

#For each of the recorded programmes...
for line in `sqlite3 /mnt/hd2/dms_cds.db "select mediaID,folder from tblMedia"`
do
        sep=`expr index "$line" \|` #There is a '|' separator between the fields
        mediaID="${line:0:$(($sep-1))}"
        filename="${line:$sep}"
        filename=${filename%.ts}
        dir=`dirname $filename`
        file=`basename $filename`

        if [ "$filename" != "(NULL)" ]; then
                echo "$lines files left"
                echo "Processing \"$filename\", Media ID is $mediaID"

                newdir="$target/$dir"
                mkdir -p "$newdir"
                cd "$newdir"
                if [ ! -e "$file.mpg" ]; then
                        url="http://$host:9000/web/media/$mediaID.TS"
                        echo "Getting $url"
                        wget $url
                        mv "$mediaID.TS" "$file.mpg"
                fi
                lines=$(($lines-1))
        fi

done

It's a half-baked script but it seems to be working fairly well at the moment. The key points are:
1) It'll help you to not shoot yourself in the foot when choosing a target backup drive, but it's still possible if you try hard enough (nothing is foolproof to a sufficiently talented fool)
2) It doesn't yet stop you from suggesting a read-only file system, but that's not really a problem
3) If you stop and restart, or if more programs are recorded during the first run, subsequent runs will only sweep up the files that haven't been saved - that's great when you're trying to run a backup and the other half is still trying to record.
4) You might end up with some partial files lying around if you break out of the program, something to worry about later, but (3) will sort you out as a failsafe.
5) Because the files are being decrypted, fancy features like thumbnails or programme info will not be stored - this is a script to rescue your recordings so that you can store them elsewhere or to put them back onto a replacement machine.
6) The script doesn't copy sidecar files (no point) or any other content (why bother, this script is aimed at decrypting stored content and that's only the recorded programmes, you have ftp for the rest)

As I type, my backup has 899 files left so I expect that it will be busy for the next day or so - any suggestions?

If it helps, here's what the most recent run looks like

Code:

humax# ./progbackup.sh
Humax HDR-T2 backup utility

Decrypts and backs up your videos to a local drive of your choosing.
Other content will not be encrypted and can therefore be transferred off separately.
HD content will not be decrypted unless you have cleared the encryption flag,
so, the script will exit if 'auto-unprotect' is not running

Examining your local drive, you need up to 738614 MB for the backup
Attached drives have the following free space (MB):
19 /media/drive2
9 /media/drive3
12 /media/drive4
0 /media/drive5
0 /media/drive6
0 /media/drive7
3 /media/drive8
1318102 /media/drive1
Which drive do you want to back up to? (CTRL-C to abort)
/media/drive1
You have selected /media/drive1
/media/drive1 has 1318102 MB free and you need up to 738614 MB
Do you want to use this drive? [y/N]
y
Backing up to /media/drive1
There are 920 programmes to backup
920 files left
Processing "My Video/BOB'S/Scrapheap Challenge/Scrapheap Challenge_20110626_1145", Media ID is 5
919 files left
Processing "My Video/BOB'S/Scrapheap Challenge/Scrapheap Challenge_20110626_0935", Media ID is 6
918 files left
Processing "My Video/BOB'S/Scrapheap Challenge/Scrapheap Challenge_20110626_1040", Media ID is 7
917 files left
Processing "My Video/BOB'S/Scrapheap Challenge/Scrapheap Challenge_20110626_1250", Media ID is 8
916 files left
Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110602_2000", Media ID is 10
915 files left
Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110519_1959", Media ID is 11
914 files left
Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110609_2000", Media ID is 12
913 files left
Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110526_2000", Media ID is 13
912 files left
Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110616_2000", Media ID is 14
911 files left
Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110509_2000", Media ID is 16
910 files left
Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110530_2000", Media ID is 17
909 files left
Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110425_2000", Media ID is 18
908 files left
Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110516_2000", Media ID is 19
907 files left
Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110606_2000", Media ID is 20
Getting http://127.0.0.1:9000/web/media/20.TS
Connecting to 127.0.0.1:9000 (127.0.0.1:9000)
wget: can't open '20.TS': File exists
906 files left
Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110611_1030", Media ID is 21
Getting http://127.0.0.1:9000/web/media/21.TS
Connecting to 127.0.0.1:9000 (127.0.0.1:9000)
21.TS                100% |*******************************|  1427M 00:00:00 ETA
905 files left
Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110502_2000", Media ID is 22
Getting http://127.0.0.1:9000/web/media/22.TS
Connecting to 127.0.0.1:9000 (127.0.0.1:9000)
22.TS                100% |*******************************|  1345M 00:00:00 ETA
904 files left
Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110523_2000", Media ID is 23
Getting http://127.0.0.1:9000/web/media/23.TS
Connecting to 127.0.0.1:9000 (127.0.0.1:9000)
23.TS                100% |*******************************|  1359M 00:00:00 ETA
903 files left

jsrobo · Nov 1, 2011

The thread is entitled "Backing Everything Up" but you say: quote "6) The script doesn't copy sidecar files (no point) or any other content (why bother, this script is aimed at decrypting stored content and that's only the recorded programmes, you have ftp for the rest)"
If it is to be usefull and easy to use by us oldies, doesn't it need to be one operation to bu everything?
I don't know, but aren't the files in 6) needed when restoring recordings to a new machine?

af123 · Nov 1, 2011

Nice : )
I'd copy the sidecar files across too then patch the .hmt file so it works with the decrypted .ts

Code:

humax# hmt -encrypted <file>.hmt

then you'll keep the associated programme information and FF/RW etc.

I'm likely to have to the same thing with mine soon if the disk keeps on clicking!

Sam Widges · Nov 1, 2011

af123 said:
Nice : )
I'd copy the sidecar files across too then patch the .hmt file so it works with the decrypted .ts

Code:

humax# hmt -encrypted <file>.hmt

then you'll keep the associated programme information and FF/RW etc.

I'm likely to have to the same thing with mine soon if the disk keeps on clicking!

OK - time to do some digging about the hmt utility, I think I must have missed something along the way.

jsrobo · Nov 1, 2011

Sorry to butt in on the technical discussion but are you looking to produce something for the non technical amongst us?

Sam Widges · Nov 1, 2011

jsrobo said:
Sorry to butt in on the technical discussion but are you looking to produce something for the non technical amongst us?

Once it's been through the more technical crew, I'll be happy to make it more openly accessible. Any comments are appreciated but no promises can be made in the meantime.

Sam Widges · Nov 1, 2011

jsrobo said:
The thread is entitled "Backing Everything Up" but you say: quote "6) The script doesn't copy sidecar files (no point) or any other content (why bother, this script is aimed at decrypting stored content and that's only the recorded programmes, you have ftp for the rest)"
If it is to be usefull and easy to use by us oldies, doesn't it need to be one operation to bu everything?
I don't know, but aren't the files in 6) needed when restoring recordings to a new machine?

As it stands, the simple intention is to have SOME form of backup of the files that would otherwise be completely lost if you sent your box back for repair. af123's comment about using the hmt utility looks interesting, but right now, I just want to have a copy of the recorded files and anything else is just icing on the cake (don't forget my info-dump script which will also be useful if copying the files elsewhere).

The script is not meant to be a finished item, I have literally just chucked it out on the Internet as work-in-progress to see what other people would be interested in having if they were in the same situation. I'm making no promises but I'm hoping that wish-lists may lead to contributions from others and you never know what may happen as a result

jsrobo · Nov 1, 2011

An option on the modified web interface would be great. I can't comment on the technical side. But all these refinements to the Humax are really much appreciated

Trevor Jasper · Nov 1, 2011

Wow, really cool. Liking this. I see you implemented what I asked for in another thread - namely wget copying the "mediaID".ts and renaming it to "file".mpg - on the localhost. If this is modified to copy the file over the original, then it should be possible to share HD content in it's original location. Something like an auto-decrypt script.

Sam Widges · Nov 2, 2011

af123 said:
Nice : )
I'd copy the sidecar files across too then patch the .hmt file so it works with the decrypted .ts

Code:

humax# hmt -encrypted <file>.hmt

then you'll keep the associated programme information and FF/RW etc.

OK, that looks pleasantly straightforward - once I've rescued the programmes I'll have a go at the sidecar files as well

af123 said:
I'm likely to have to the same thing with mine soon if the disk keeps on clicking!

Oh dear, there goes another one. There seem to be a disturbing number of early life failures on these boxes and I'm starting to be glad of the 2 year warranty.

Sam Widges · Nov 2, 2011

Trevor Jasper said:
Wow, really cool. Liking this. I see you implemented what I asked for in another thread - namely wget copying the "mediaID".ts and renaming it to "file".mpg - on the localhost. If this is modified to copy the file over the original, then it should be possible to share HD content in it's original location. Something like an auto-decrypt script.

Very interesting, not something that I'd considered (or even thought possible) when I started writing this. I'll look at that once I've sorted the backup side of things, but recovering SWMBO's programmes from a failing box has to take priority for the moment

Trevor Jasper · Nov 2, 2011

Well you've proved what is possible, so maybe a modified script based on this work can be linked to the webif interface on right click to auto-decrypt the current file - replacing the original and updated sidecar files - maybe moving the originals to a backup-type directory just in case they need to be restored. The user can then delete the directory when confident that the new files work satisfactorily.

Appreciate the need to get those thousand files backed-up first, though.

I'm so loving all the extras that this community is coming up with - really makes the box better than when bought!!! Thanks and kudos to all who've helped get it this far...

Sam Widges · Nov 2, 2011

The copying has gone well overnight - just under 400 files left to copy. Thankfully a lot of the content is radio programmes, so they don't take long.

Once I've got the backup, I'll copy over the sidecar files and patch them for the new location turning off the decryption flag along the way. If that works, it would be very easy to turn this into a script that would do decrypt-in-place.

Black Hole · Nov 2, 2011

It would be very nice to have this as a web page, with selectable parameters for destination etc, a progress bar and an ETA.

Sam Widges · Nov 2, 2011

It was all going quite well until SWMBO started power cycling the box. OK, she now knows to shout before she does it, but we're now going to have to wait until tomorrow to experiment further :rolleyes:

af123 · Nov 2, 2011

Here's a script I've had lying around for a while that does command line decryption. It's written in Jim but I've never gone back to it to finish testing.. It's probably ok to put into a package and let people use without too much more effort.
It leverages the Jim classes in the web interface so it doesn't have to think too hard about things like SQLite..

Code:

#!/mod/bin/jimsh

source /mod/var/mongoose/lib/setup
require ts.class system.class

if {[llength $argv] != 1} {
        puts "Syntax: decrypt <path to .ts file>"
        exit
}

lassign $argv file

set ts [ts fetch $file]
if {[catch {$ts size}]} {
        puts "Invalid TS file, $file"
        exit
}

set ::env(HTTP_HOST) "127.0.0.1"
lassign [$ts dlnaloc] url mime

puts "DLNA URL: $url"
if {$url eq ""} {
        puts "File is not available from DLNA server."
        exit
}

set fbase [file rootname $file]
set nfbase "${fbase}-decrypted"

puts "Downloading decrypted file."
exec /mod/bin/wget --progress=bar:force -O "$nfbase.ts" $url
foreach ext {nts hmt thm} {
        if {[file exists "$fbase.$ext"]} {
                puts "Copying sidecar $ext"
                file copy "$fbase.$ext" "$nfbase.$ext"
        }
}
if {[file exists "$nfbase.hmt"]} {
        exec /mod/bin/hmt -encrypted "$nfbase.hmt"
        exec /mod/bin/hmt "+settitle=[$ts get title] (decrypted)" "$nfbase.hmt"
}

puts "Done."

and an example of running it:

Code:

humax# decrypt /mnt/hd2/My\ Video/Family\ Guy/Breaking_Out_Is_Hard_to_Do.ts
DLNA URL: http://127.0.0.1:9000/web/media/2141.TS
Downloading decrypted file.
Copying sidecar nts
Copying sidecar hmt
Copying sidecar thm
Done.

humax# ls /mnt/hd2/My\ Video/Family\ Guy/Breaking_Out_Is_Hard_to_Do*
/mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do-decrypted.hmt
/mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do-decrypted.nts
/mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do-decrypted.thm
/mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do-decrypted.ts
/mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do.hmt
/mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do.nts
/mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do.thm
/mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do.ts

Trevor Jasper · Nov 3, 2011

Wow, af123, that's pretty cool. Will give that a go sometime. Would be nice to have that as a right-click option on the webif interface...... Any chances?

Thanks.... T

oijonesey · Nov 3, 2011

Would be even better if it just ran in the background (like auto unprotect) and just decrypted anything not already decrypted!!

The first run would be a bugger but after that it would just be on new recordings. Might need to watch out for power offs while it's working though?...

Trevor Jasper · Nov 3, 2011

Ok, extend auto-unprotect to do the decryption as well?

Sam Widges · Nov 3, 2011

oijonesey said:
Would be even better if it just ran in the background (like auto unprotect) and just decrypted anything not already decrypted!! The first run would be a bugger but after that it would just be on new recordings. Might need to watch out for power offs while it's working though?...

I'm hoping to write it so that it can be run from cron and from the command line. As you say, the first run will be a bit of a monster

[progbackup] Backing everything up

Active Member

Member

Administrator

Active Member

Member

Active Member

Active Member

Member

Member

Active Member

Active Member

Member

Active Member

May contain traces of nut

Active Member

Administrator

Member

Hummy.tv SEO Guru

Member

Active Member