[progbackup] Backing everything up

OP
S

Sam Widges

Active Member
OK, here's the latest version of progbackup.sh in all its gory glory

Code:
#!/bin/sh

echo "Humax HDR-T2 backup utility"
echo
echo "Decrypts and backs up your videos to a local drive of your choosing."
echo "Other content will not be encrypted and can therefore be transferred off separately."
echo "HD content will not be decrypted unless you have cleared the encryption flag,"
echo "so, the script will exit if 'auto-unprotect' is not running"
echo

IFS=$'\n' #You need this or the for loop gets confused by spaces

host='127.0.0.1'

#What about standby? Can this be tweaked on the fly?

used=`df -m | sed 's/ */ /g' | grep /mnt/hd2 | cut -f 4 -d\ `

echo "Examining your local drive, you need up to $used MB for the backup"
echo "Attached drives have the following free space (MB):"

for freespace in `df -m | sed 's/ */ /g' | grep /media/drive | cut -f 5,7 -d\ `
do
        echo "$freespace"
done

target=""
answer="N"

while [ "$answer" != "y" ]
do
        result=""

        while [ "$result" = "" ]
        do
                echo "Which drive do you want to back up to? (CTRL-C to abort)"

                read target

                result=`df -m | grep $target`

                echo "You have selected $target"
                if [ "$result" = "" ]; then
                        echo "That is not a valid drive"
                fi

        done

        targetspace=`df -m | sed 's/ */ /g' | grep $target | cut -f 5 -d\ `
        echo "$target has $targetspace MB free and you need up to $used MB"
        echo "Do you want to use this drive? [y/N]"
        read answer
done

echo "Backing up to $target"

# What if someone has plugged in an NTFS drive and it's mounted read only?
# Check for Fat32/ntfs or check for R/W, or both?

# You really don't want to run this if you have HD content and you aren't running auto-unprotect
AP=`opkg list-installed | grep auto-unprotect`

if [ "$AP" = "" ]; then
        echo "Auto-unprotect isn't running, you don't want to download encrypted HD files"
        echo "Install Auto-unprotect and wait 2 hours before rerunning"
        echo "If you do not wait 2 hours, you could lose access to some HD files"
        exit
fi

pwrdwn=`sqlite3 /var/lib/humaxtv/setup.db "Select itemvalue from tbl_menuconfig where itemName like 'AUTOMATIC_POWER_DOWN'"`

echo "Auto Power Off is set to: $pwrdwn"

pwrstby=`sqlite3 /var/lib/humaxtv/setup.db "Select itemvalue from tbl_menuconfig where itemName like 'PWR_SAVING_ON_STANDBY'" `

echo "Power Saving In Standby is set to: $pwrstby"

if [ "$pwrstby" -gt 0 ] || [ "$pwrdwn" -gt 0 ];then
        echo "Script will not run with the powersaving options set"
        exit
fi

#How many programmes are there to backup?
lines=`sqlite3 /mnt/hd2/dms_cds.db "select mediaID,folder from tblMedia" | grep -v '|(NULL)' | wc -l`

echo "There are $lines programmes to backup"

#Should we have a pause here to confirm the target destination and whether or not to proceed?

#For each of the recorded programmes...
for line in `sqlite3 /mnt/hd2/dms_cds.db "select mediaID,folder from tblMedia"`
do
        sep=`expr index "$line" \|` #There is a '|' separator between the fields
        mediaID="${line:0:$(($sep-1))}"
        filename="${line:$sep}"
        filename=${filename%.ts}
        dir=`dirname $filename`
        file=`basename $filename`

        if [ "$filename" != "(NULL)" ]; then
                echo -n "$lines files left: "

                newdir="$target/$dir"
                mkdir -p "$newdir"
                cd "$newdir"
                if [ ! -e "$file.ts" ]; then
                        echo "Processing \"$filename\", Media ID is $mediaID"
                        url="http://$host:9000/web/media/$mediaID.TS"
                        echo "Getting $url"

                        # Check that port 9000 is open and exit if not
                        # Needs to be implemented

                        if [ -f "$mediaID.TS" ]; then
                                rm "$mediaID.TS"
                        fi

                        wget $url

                        mv "$mediaID.TS" "$file.ts"
                else
                        echo "Skipping $file.ts"
                fi

                if [ -e "/media/$filename.nts" ] && [ ! -e "$newdir/$file.nts" ]; then
                        cp "/media/$filename.nts" "$newdir/$file.nts"
                fi

                if [ -e "/media/$filename.thm" ] && [ ! -e "$newdir/$file.thm" ]; then
                        cp "/media/$filename.thm" "$newdir/$file.thm"
                fi

                if [ -e "/media/$filename.hmt" ] && [ ! -e "$newdir/$file.hmt" ]; then
                        cp "/media/$filename.hmt" "$newdir/$file.hmt"
                        hmt -encrypted "$newdir/$file.hmt"
                        hmt +setfolder="$newdir" "$newdir/$file.hmt"
                fi

                # Is it worth copying the .series file?

                lines=$(($lines-1))

        fi

done
Changes since the previous version...

1) A bit more error checking, for instance warning that you really don't want to think about running the script if the box is going into standby in a while
2) Some error checking still missing - I've had the webserver go away when testing but that isn't checked for
3) The sidecar files are now copied across and are amended to reflect the new location and the lack of encryption
4) Despite (3), the series file is not being copied over - that might be useful if the files are restored to a Humax hard drive, but they're not copied for the moment.

It should be totally safe to run as it doesn't suggest overwriting the original files, but if you run this and your gerbil explodes, blame the gods of shell scripting.

Any thoughts/comments and suggestions are appreciated as always.
 
OP
S

Sam Widges

Active Member
Would be even better if it just ran in the background (like auto unprotect) and just decrypted anything not already decrypted!! :) The first run would be a bugger but after that it would just be on new recordings. Might need to watch out for power offs while it's working though?...
It's time to knock off for the night, but the backup script seems to have worked successfully, so it's time to move onto the "decrypt in place" utility - time for a new thread.
 
OP
S

Sam Widges

Active Member
Ok, extend auto-unprotect to do the decryption as well?
I've had a look at auto-unprotect, but it'll probably be easier to just tweak the existing code to work alongside it because it only seems to run the unprotect script on HD files. If that could be extended to SD content as well, then it would be possible to merge both scripts.
 
OP
S

Sam Widges

Active Member
I've made some mods to the script - if you run it on its own, it will work the same way and just back up the whole collection, but now if you supply an argument like "/mnt/hd2/My Video/BOB's Films/" then it will only back up recordings under that directory.

I've added some better error handling and trapping - CTRL-C now exits the program cleanly.

Series files are now copied as well since you will want them once you try to put the files back after it has been replaced.

And the good news is that it's now all been packaged up and on its way to the repository.
 
OP
S

Sam Widges

Active Member
I've just fixed a bug - hitting return when selecting the target drive wasn't pretty.

I've also changed the code that tells you how much space you need for your backup to account for a directory that is specified on the command line.

progbackup version 0.1a is on it's way over to the repository now.
 
OP
S

Sam Widges

Active Member
In case anyone is interested, progbackup 0.2 is on its way to the repository.

Changes since the last version:
If you run it twice and delete some programmes in the meantime, those will now be deleted from the backup drive
If you are running unencrypt (decrypt-in-place) it will simply copy decrypted files as you don't need to wget them.
It checks for the DLNA server being enabled or if it has crashed and handles that cleanly.

Edit - I've also done some benchmarking and programs are backed up at around 17GB/hour to an external USB2 drive. If backing up a very full 1TB box, that's going to take about 2 days.
 
OP
S

Sam Widges

Active Member
OK - the replacement box has arrived and the files are being copied back. No apparent problems so far and files are transferring at about 50GB/hour.
 

Chicanery86

New Member
Nice script Sam, Finding it not recognising locally attached USB drives due to it grepping for /media/drive whereas my WD Passports mounted as NTFS3 automatically mount as /media/usb-drive. I appreciate that for your original purpose of emergency recovery that wasn't an issue, would be nice to extend it beyond directly attached to USB drives
 
OP
S

Sam Widges

Active Member
Thanks for the feedback - I've never had any of my usb drives mount like that. When I'm next doing some tweaking, I'll update the script to grep for those disks as well. I guess that, now NTFS has reared its head, I'd better check that drives aren't mounted read only as well.
 

xyz321

Well-Known Member
The drives appear as /media/drive<n> when mounted by the original (read-only) NTFS driver. The ntfs-3g package mounts them under /media/usb-drive<n> because the Humax application still thinks they are read-only if mounted as /media/drive<n>.
 
OP
S

Sam Widges

Active Member
I've just uploaded a new version to the repository, it should be available in an hour or so - look for progbackup version 0.2b.1.

One tip - just because the script tells you info about drives, you don't actually have to use the ones that it lists and just type '/media/usb-drive1' when it asks you where to back up to. The only thing is that you won't know if there's definitely enough room or not. In fact, you could use the same technique to back up to remote drives as well.
 

Chicanery86

New Member
Progbackup is working fine with the ntfs-3G mounted drives
There is a one character tweak for the unencrypt script required to support nested folders under the directory to be backed up. Namely insertion of a wild card * after $targetpath to widen the match for nested subdirectories.
 

Chicanery86

New Member
At present it appears to assume that there is only one level of subdirectories under the directory specified with unencrypt setup in my case that's /mnt/hd2/My Video. My directory structure is as follows -> denotes a subdirectories
My Video
My Video->Archive
My Video->Archive->Drama
My Video->Archive->Drama->Series_Name
My Video->Series_Name
the intention is that files under the Archive hierarchy will be automatically decrypted using unencrypt.
With /mnt/hd2/My Video specified to unencryptsetup the secondary test on media files namely that they are located a single directory under the unencryptsetup specified point fails. I could change the unencryptsetup to My Video to My Video->Archive however once I have located where the mark new flag is stored I want to automatically archive watched recordings which the will be moved off by means of Samba.
 
OP
S

Sam Widges

Active Member
Unencrypt processes programmes in the media server database and simply matches the search directory against the location of each file as returned by a dump of programmes out of the database. If you have My Video->Archive->Prog1, My Video->Archive->ADirectory->Prog2 and My Video->Archive->AnotherDirectory->DifferentDirectory->Prog3, all of them will be decrypted if you tell unencryptsetup to decrypt My Video->Archive.

I'm really not sure, from your description, what you are trying to achieve with your example and why you can't just target unencryptsetup at My Video->Archive.
 

Chicanery86

New Member
The symptom was that each pass through unencrypt simply did nothing, despite the existance of encrypted files within a subdirectory of a subdirectory of the specified location given to unencryptsetup. I manually inspected the dmd_cds.db which appeared to be valid.
I have reverted my change , with no net difference, sorry for that evidentially something went astray.
BTW You can wrapper the call to unencrypt with nice -n 15 <command_line> to prevent it impacting adversely on other tasks such as recording.
 

AlWilson

New Member
OK - the replacement box has arrived and the files are being copied back. No apparent problems so far and files are transferring at about 50GB/hour.
Hi Sam,
Just to add to what has already been mentioned, it would be really great for this to be added to WebIf to make it less 'techie'. Personally, I find it very easy to mess up command line entries!
While I really like this utility (Will probably have to use it soon!) is it really necessary considering that you can backup the complete 'My Video' contents using opt+ and selecting the move/copy option? I did this to an external hard drive and, although it took about 3 days, I can now select and play programs from the external drive after selecting USB as the active media. I expect also that after replacing the hard disk in my Hummy I will be able to use opt+ to copy them all back again.
What is the benefit of using your util (as great as it is!) over built-in functionality? Maybe speed?
Thanks and All the best,
Alan.
 
Top