1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[progbackup] Backing everything up

Discussion in 'HD/HDR-FOX T2 Customised Firmware' started by Sam Widges, Nov 1, 2011.

  1. Sam Widges

    Sam Widges Active Member

    OK, this is hot off the presses and is very much work in progress...

    My HDR-T2 has been on the fritz for a while now and several posts on the forum make me think that the hard drive is on the way out. Apart from the fact that we only bought the dratted thing in February, how do we go about rescuing the contents before it goes back for repair?

    Well, the answer seems to be that we need:
    1) Custom firmware
    2) Auto-unprotect installed and running
    3) An external hard drive with enough space that you can write to
    4) A cunning script to save you lots of time

    Assuming that you can sort steps 1-3, here's my first cut at step 4:

    progbackup.sh

    Code:
    #!/bin/sh
    
    echo "Humax HDR-T2 backup utility"
    echo
    echo "Decrypts and backs up your videos to a local drive of your choosing."
    echo "Other content will not be encrypted and can therefore be transferred off separately."
    echo "HD content will not be decrypted unless you have cleared the encryption flag,"
    echo "so, the script will exit if 'auto-unprotect' is not running"
    echo
    
    IFS=$'\n' #You need this or the for loop gets confused by spaces
    
    host='127.0.0.1' # Yay localhost works!
    
    used=`df -m | sed 's/ */ /g' | grep /mnt/hd2 | cut -f 4 -d\ `
    
    echo "Examining your local drive, you need up to $used MB for the backup"
    echo "Attached drives have the following free space (MB):"
    
    for freespace in `df -m | sed 's/ */ /g' | grep /media/drive | cut -f 5,7 -d\ `
    do
            echo "$freespace"
    done
    
    target=""
    answer="N"
    
    while [ "$answer" != "y" ]
    do
            result=""
    
            while [ "$result" = "" ]
            do
                    echo "Which drive do you want to back up to? (CTRL-C to abort)"
    
                    read target
    
                    result=`df -m | grep $target`
    
                    echo "You have selected $target"
                    if [ "$result" = "" ]; then
                            echo "That is not a valid drive"
                    fi
    
            done
    
            targetspace=`df -m | sed 's/ */ /g' | grep $target | cut -f 5 -d\ `
            echo "$target has $targetspace MB free and you need up to $used MB"
            echo "Do you want to use this drive? [y/N]"
            read answer
    done
    
    echo "Backing up to $target"
    
    # What if someone has plugged in an NTFS drive and it's mounted read only?
    # Check for Fat32/ntfs or check for R/W, or both?
    
    #You really don't want to run this if you have HD contect and you aren't running auto-unprotect
    AP=`opkg list-installed | grep auto-unprotect`
    
    if [ "$AP" = "" ]; then
            echo "Auto-unprotect isn't running, you don't want to download encrypted HD files"
            echo "Install Auto-unprotect and wait 2 hours before rerunning"
            echo "If you do not wait 2 hours, you could lose access to some HD files"
            exit
    fi
    
    #How many programmes are there to backup?
    lines=`sqlite3 /mnt/hd2/dms_cds.db "select mediaID,folder from tblMedia" | grep -v '|(NULL)' | wc -l`
    
    echo "There are $lines programmes to backup"
    
    #Should we have a pause here to confirm the target destination and whether or not to proceed?
    
    #For each of the recorded programmes...
    for line in `sqlite3 /mnt/hd2/dms_cds.db "select mediaID,folder from tblMedia"`
    do
            sep=`expr index "$line" \|` #There is a '|' separator between the fields
            mediaID="${line:0:$(($sep-1))}"
            filename="${line:$sep}"
            filename=${filename%.ts}
            dir=`dirname $filename`
            file=`basename $filename`
    
            if [ "$filename" != "(NULL)" ]; then
                    echo "$lines files left"
                    echo "Processing \"$filename\", Media ID is $mediaID"
    
                    newdir="$target/$dir"
                    mkdir -p "$newdir"
                    cd "$newdir"
                    if [ ! -e "$file.mpg" ]; then
                            url="http://$host:9000/web/media/$mediaID.TS"
                            echo "Getting $url"
                            wget $url
                            mv "$mediaID.TS" "$file.mpg"
                    fi
                    lines=$(($lines-1))
            fi
    
    done
    
    It's a half-baked script but it seems to be working fairly well at the moment. The key points are:
    1) It'll help you to not shoot yourself in the foot when choosing a target backup drive, but it's still possible if you try hard enough (nothing is foolproof to a sufficiently talented fool)
    2) It doesn't yet stop you from suggesting a read-only file system, but that's not really a problem
    3) If you stop and restart, or if more programs are recorded during the first run, subsequent runs will only sweep up the files that haven't been saved - that's great when you're trying to run a backup and the other half is still trying to record.
    4) You might end up with some partial files lying around if you break out of the program, something to worry about later, but (3) will sort you out as a failsafe.
    5) Because the files are being decrypted, fancy features like thumbnails or programme info will not be stored - this is a script to rescue your recordings so that you can store them elsewhere or to put them back onto a replacement machine.
    6) The script doesn't copy sidecar files (no point) or any other content (why bother, this script is aimed at decrypting stored content and that's only the recorded programmes, you have ftp for the rest)

    As I type, my backup has 899 files left so I expect that it will be busy for the next day or so - any suggestions?

    If it helps, here's what the most recent run looks like

    Code:
    humax# ./progbackup.sh
    Humax HDR-T2 backup utility
    
    Decrypts and backs up your videos to a local drive of your choosing.
    Other content will not be encrypted and can therefore be transferred off separately.
    HD content will not be decrypted unless you have cleared the encryption flag,
    so, the script will exit if 'auto-unprotect' is not running
    
    Examining your local drive, you need up to 738614 MB for the backup
    Attached drives have the following free space (MB):
    19 /media/drive2
    9 /media/drive3
    12 /media/drive4
    0 /media/drive5
    0 /media/drive6
    0 /media/drive7
    3 /media/drive8
    1318102 /media/drive1
    Which drive do you want to back up to? (CTRL-C to abort)
    /media/drive1
    You have selected /media/drive1
    /media/drive1 has 1318102 MB free and you need up to 738614 MB
    Do you want to use this drive? [y/N]
    y
    Backing up to /media/drive1
    There are 920 programmes to backup
    920 files left
    Processing "My Video/BOB'S/Scrapheap Challenge/Scrapheap Challenge_20110626_1145", Media ID is 5
    919 files left
    Processing "My Video/BOB'S/Scrapheap Challenge/Scrapheap Challenge_20110626_0935", Media ID is 6
    918 files left
    Processing "My Video/BOB'S/Scrapheap Challenge/Scrapheap Challenge_20110626_1040", Media ID is 7
    917 files left
    Processing "My Video/BOB'S/Scrapheap Challenge/Scrapheap Challenge_20110626_1250", Media ID is 8
    916 files left
    Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110602_2000", Media ID is 10
    915 files left
    Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110519_1959", Media ID is 11
    914 files left
    Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110609_2000", Media ID is 12
    913 files left
    Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110526_2000", Media ID is 13
    912 files left
    Processing "My Video/BOB'S/Emergency Bikers/Emergency Bikers_20110616_2000", Media ID is 14
    911 files left
    Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110509_2000", Media ID is 16
    910 files left
    Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110530_2000", Media ID is 17
    909 files left
    Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110425_2000", Media ID is 18
    908 files left
    Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110516_2000", Media ID is 19
    907 files left
    Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110606_2000", Media ID is 20
    Getting http://127.0.0.1:9000/web/media/20.TS
    Connecting to 127.0.0.1:9000 (127.0.0.1:9000)
    wget: can't open '20.TS': File exists
    906 files left
    Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110611_1030", Media ID is 21
    Getting http://127.0.0.1:9000/web/media/21.TS
    Connecting to 127.0.0.1:9000 (127.0.0.1:9000)
    21.TS                100% |*******************************|  1427M 00:00:00 ETA
    905 files left
    Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110502_2000", Media ID is 22
    Getting http://127.0.0.1:9000/web/media/22.TS
    Connecting to 127.0.0.1:9000 (127.0.0.1:9000)
    22.TS                100% |*******************************|  1345M 00:00:00 ETA
    904 files left
    Processing "My Video/BOB'S/The Gadget Show/The Gadget Show_20110523_2000", Media ID is 23
    Getting http://127.0.0.1:9000/web/media/23.TS
    Connecting to 127.0.0.1:9000 (127.0.0.1:9000)
    23.TS                100% |*******************************|  1359M 00:00:00 ETA
    903 files left
    
     
  2. jsrobo

    jsrobo Member

    The thread is entitled "Backing Everything Up" but you say: quote "6) The script doesn't copy sidecar files (no point) or any other content (why bother, this script is aimed at decrypting stored content and that's only the recorded programmes, you have ftp for the rest)"
    If it is to be usefull and easy to use by us oldies, doesn't it need to be one operation to bu everything?
    I don't know, but aren't the files in 6) needed when restoring recordings to a new machine?
     
  3. af123

    af123 Administrator Staff Member

    Nice : )
    I'd copy the sidecar files across too then patch the .hmt file so it works with the decrypted .ts
    Code:
    humax# hmt -encrypted <file>.hmt
    then you'll keep the associated programme information and FF/RW etc.

    I'm likely to have to the same thing with mine soon if the disk keeps on clicking!
     
    Sam Widges likes this.
  4. Sam Widges

    Sam Widges Active Member

    OK - time to do some digging about the hmt utility, I think I must have missed something along the way.
     
  5. jsrobo

    jsrobo Member

    Sorry to butt in on the technical discussion but are you looking to produce something for the non technical amongst us?
     
  6. Sam Widges

    Sam Widges Active Member

    Once it's been through the more technical crew, I'll be happy to make it more openly accessible. Any comments are appreciated but no promises can be made in the meantime.
     
  7. Sam Widges

    Sam Widges Active Member

    As it stands, the simple intention is to have SOME form of backup of the files that would otherwise be completely lost if you sent your box back for repair. af123's comment about using the hmt utility looks interesting, but right now, I just want to have a copy of the recorded files and anything else is just icing on the cake (don't forget my info-dump script which will also be useful if copying the files elsewhere).

    The script is not meant to be a finished item, I have literally just chucked it out on the Internet as work-in-progress to see what other people would be interested in having if they were in the same situation. I'm making no promises but I'm hoping that wish-lists may lead to contributions from others and you never know what may happen as a result :)
     
  8. jsrobo

    jsrobo Member

    An option on the modified web interface would be great. I can't comment on the technical side. But all these refinements to the Humax are really much appreciated:D
     
  9. Trevor Jasper

    Trevor Jasper Member

    Wow, really cool. Liking this. I see you implemented what I asked for in another thread - namely wget copying the "mediaID".ts and renaming it to "file".mpg - on the localhost. If this is modified to copy the file over the original, then it should be possible to share HD content in it's original location. Something like an auto-decrypt script.
     
  10. Sam Widges

    Sam Widges Active Member

    OK, that looks pleasantly straightforward - once I've rescued the programmes I'll have a go at the sidecar files as well

    Oh dear, there goes another one. There seem to be a disturbing number of early life failures on these boxes and I'm starting to be glad of the 2 year warranty.
     
  11. Sam Widges

    Sam Widges Active Member

    Very interesting, not something that I'd considered (or even thought possible) when I started writing this. I'll look at that once I've sorted the backup side of things, but recovering SWMBO's programmes from a failing box has to take priority for the moment ;)
     
  12. Trevor Jasper

    Trevor Jasper Member

    Well you've proved what is possible, so maybe a modified script based on this work can be linked to the webif interface on right click to auto-decrypt the current file - replacing the original and updated sidecar files - maybe moving the originals to a backup-type directory just in case they need to be restored. The user can then delete the directory when confident that the new files work satisfactorily.

    Appreciate the need to get those thousand files backed-up first, though.

    I'm so loving all the extras that this community is coming up with - really makes the box better than when bought!!! Thanks and kudos to all who've helped get it this far...
     
  13. Sam Widges

    Sam Widges Active Member

    The copying has gone well overnight - just under 400 files left to copy. Thankfully a lot of the content is radio programmes, so they don't take long.

    Once I've got the backup, I'll copy over the sidecar files and patch them for the new location turning off the decryption flag along the way. If that works, it would be very easy to turn this into a script that would do decrypt-in-place.
     
    oijonesey likes this.
  14. Black Hole

    Black Hole Felinos Guru

    It would be very nice to have this as a web page, with selectable parameters for destination etc, a progress bar and an ETA.
     
  15. Sam Widges

    Sam Widges Active Member

    It was all going quite well until SWMBO started power cycling the box. OK, she now knows to shout before she does it, but we're now going to have to wait until tomorrow to experiment further :rolleyes:
     
  16. af123

    af123 Administrator Staff Member

    Here's a script I've had lying around for a while that does command line decryption. It's written in Jim but I've never gone back to it to finish testing.. It's probably ok to put into a package and let people use without too much more effort.
    It leverages the Jim classes in the web interface so it doesn't have to think too hard about things like SQLite..

    Code:
    #!/mod/bin/jimsh
    
    source /mod/var/mongoose/lib/setup
    require ts.class system.class
    
    if {[llength $argv] != 1} {
            puts "Syntax: decrypt <path to .ts file>"
            exit
    }
    
    lassign $argv file
    
    set ts [ts fetch $file]
    if {[catch {$ts size}]} {
            puts "Invalid TS file, $file"
            exit
    }
    
    set ::env(HTTP_HOST) "127.0.0.1"
    lassign [$ts dlnaloc] url mime
    
    puts "DLNA URL: $url"
    if {$url eq ""} {
            puts "File is not available from DLNA server."
            exit
    }
    
    set fbase [file rootname $file]
    set nfbase "${fbase}-decrypted"
    
    puts "Downloading decrypted file."
    exec /mod/bin/wget --progress=bar:force -O "$nfbase.ts" $url
    foreach ext {nts hmt thm} {
            if {[file exists "$fbase.$ext"]} {
                    puts "Copying sidecar $ext"
                    file copy "$fbase.$ext" "$nfbase.$ext"
            }
    }
    if {[file exists "$nfbase.hmt"]} {
            exec /mod/bin/hmt -encrypted "$nfbase.hmt"
            exec /mod/bin/hmt "+settitle=[$ts get title] (decrypted)" "$nfbase.hmt"
    }
    
    puts "Done."
    
    and an example of running it:

    Code:
    humax# decrypt /mnt/hd2/My\ Video/Family\ Guy/Breaking_Out_Is_Hard_to_Do.ts
    DLNA URL: http://127.0.0.1:9000/web/media/2141.TS
    Downloading decrypted file.
    Copying sidecar nts
    Copying sidecar hmt
    Copying sidecar thm
    Done.
    
    humax# ls /mnt/hd2/My\ Video/Family\ Guy/Breaking_Out_Is_Hard_to_Do*
    /mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do-decrypted.hmt
    /mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do-decrypted.nts
    /mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do-decrypted.thm
    /mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do-decrypted.ts
    /mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do.hmt
    /mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do.nts
    /mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do.thm
    /mnt/hd2/My Video/Family Guy/Breaking_Out_Is_Hard_to_Do.ts
    
     
  17. Trevor Jasper

    Trevor Jasper Member

    Wow, af123, that's pretty cool. Will give that a go sometime. Would be nice to have that as a right-click option on the webif interface...... Any chances?

    Thanks.... T
     
  18. oijonesey

    oijonesey Hummy.tv SEO Guru

    Would be even better if it just ran in the background (like auto unprotect) and just decrypted anything not already decrypted!! :) The first run would be a bugger but after that it would just be on new recordings. Might need to watch out for power offs while it's working though?...
     
  19. Trevor Jasper

    Trevor Jasper Member

    Ok, extend auto-unprotect to do the decryption as well?
     
  20. Sam Widges

    Sam Widges Active Member

    I'm hoping to write it so that it can be run from cron and from the command line. As you say, the first run will be a bit of a monster :D