1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Automating rsync backups

Discussion in 'HD/HDR-FOX T2 Customised Firmware' started by Placemat, Apr 20, 2012.

  1. Placemat

    Placemat New Member

    Hello there chaps,

    I am a first time writer, long time reader on this forum and I would like to start by giving thanks to the geniuses here that have contributed to the customised HDR-FOX T2 firmware. It has turned my bog standard PVR into a a fantastic box of tricks.

    Now to my query which I hope someone can help me with. I am trying to make mirror backups of the content of my HDR-FOX T2 internal hardrive onto a Synology Diskstation. As the HDR-FOX T2 supports rsync and my Synology supports acting as a rsync destination, I thought this would be the best method for backups.

    My understanding of Linux is poor which made the first step difficult enough. After reading some rysnc guides and some trial and error, I finally identified the rsync command that was applicable to my needs


    rsync -avr --progress --delete --exclude 'Tsr/' mnt/hd2/ root@synology ip::NetBackup

    Running this command seemed to work well (any suggestions for improvements to the command would be appreciated.)

    The next step I want to achieve is to automate the rsync process, so that I can ask the Humax to run a backup every evening at a particular time. I already have the Humax set to power on at 2am-6am to run the unencrypt package, so ideally I want to have it run after this step.

    This is where I am completely stuck, and no amount of searching this forum or complicated Linux guides will take me any further. Actually I note that there is little reference made to rsync on this forum, so I wonder if anyone is actually using it at all.

    I would really appreciate any help on the next steps in this process, where do I start?
     
  2. Ezra Pound

    Ezra Pound Well-Known Member

    To auto-run rsync on a regular basis you can setup a cron job, e.g. here is the line that kicks off unencrypt each night :-
    1,31 1-6 * * * /mod/sbin/unencrypt "/mnt/hd2"
    For notes on Cron in the WiKi, CLICK HERE
     
  3. af123

    af123 Administrator Staff Member

    Nicely done. You just need to add a line to the crontab file to kick the job off when you want. The webif built-in file editor is probably the easiest way to do it, just edit the /mod/var/spool/crontabs/root file and a add a line similar to this one:

    Code:
    0 3 * * * /mod/bin/rsync -avr --progress --delete --exclude 'Tsr/' mnt/hd2/ root@synology ip::NetBackup
    
    The first five fields are <minute> <hour> <day of the month> <month of the year> <weekday> with a * meaning any, so the line above will cause the command to be run at 3am each day, providing the box is not in standby at the time.
     
  4. Placemat

    Placemat New Member

    Ah I see, thank you for replying.

    There is an issue that I neglected to put in the original post, my Synology requires a password before the rsync command can be run.

    Now I understand I can just store a default password for rysnc on the Humax (although my attempts to do this failed?) but this is obviously an unsecured method. I've also been reading about using SSH for shared access keys but the mind boggles with such information. Do either of you have any suggestions?
     
  5. af123

    af123 Administrator Staff Member

    The usual way is to use shared SSH keys. It isn't as hard as it sounds.
    You need to create a key pair on the Humax then put the public part of that key on to your Synology somehow.
    If nobody beats me to it I'll post the keypair generation command later when I've looked it up.
     
  6. Placemat

    Placemat New Member

    While I await your instructions on setting up keypairs, I thought I would have a go at running rsync in a SSH environment. I enabled SSH on my Synology Diskstation then opened up a SSH connection in Putty to the the Humax and ran the following command:


    Code:
     humax# rsync -avr --delete --progress --exclude '[Deleted Items]' ssh mnt/hd2/My_Video/ root@192.168.0.11:NetBackup
    I've tried to refine the command a little bit with respect to only copying the "My Video" directory and exclude the "Deleted Items" folder as it's a wasteful area to process.

    All I got in return for my effort was this:

    Code:
    humax# rsync -avr --delete --progress --exclude '[Deleted Items]' ssh mnt/hd2/My_Video/ root@Synology ip:NetBackup
    root@Synology ip password:
    sending incremental file list
    rsync: link_stat "/mnt/hd2/mod/ssh" failed: No such file or directory (2)
    rsync: change_dir "/mnt/hd2/mod/mnt/hd2/My_Video" failed: No such file or directory (2)
     
     
    sent 36 bytes received 12 bytes 3.84 bytes/sec
    total size is 0 speedup is 0.00
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.8]
    Am I correct in thinking that when working in SSH, folder structures appear differently? I don't think I have specified the correct address for transfer.

    I have Dropbear installed and I haven't tweaked it in any way.
     
  7. af123

    af123 Administrator Staff Member

    I'm not sure what you are trying to do here. rsync will already be using SSH as its underlying transport. Your first command was fine, you just need to set up the keys and the password prompt should go away. I don't know how you set up the keys on the Synology - on a UNIX system you put them in a file called authorized_keys (US spelling) under a directory called .ssh within the home directory.
     
  8. Ezra Pound

    Ezra Pound Well-Known Member

    Just a small point (for future reference) there is a space rather than an under-score between My and Video e.g. ssh "mnt/hd2/My Video/"
     
  9. Placemat

    Placemat New Member

    Good spot, I will change that thanks.

    The Synology is capable of using UNIX SSH keys.

    I'm using these 2 guides for setting up keys:

    http://www.cesareriva.com/archives/102
    http://www.cesareriva.com/archives/112

    The first issue I have run in to is the fact that running

    Code:
    ssh-keygen -t rsa -f .ssh/id_rsa -C "localusername@localaddress"
    reports that ssh-keygen is not found.
     
  10. af123

    af123 Administrator Staff Member

    The corresponding command for dropbear is

    Code:
    dropbearkey -t rsa -f /mod/.ssh/id_rsa
    
     
  11. Sam Widges

    Sam Widges Active Member

    Nice work Placemat :)

    I haven't read the full thread, but do bear in mind that cron jobs can fail if they produce error output while running. The magic incantation ' 2>&1' on the end of the crontab line (you can see it in the examples in the Wiki) solves that.

    In Linux, writing to the error stream does not mean that things have gone horribly wrong, it's just a way of making sure that the user sees important issues, even if the rest of the output is being redirected elsewhere, such as to a file.
     
  12. Placemat

    Placemat New Member

    Thanks very much, although much of my success is blind trial and error.

    Thanks for the dropbearkey command af123, I've managed to set up the .ssh/ folder and create the id_rsa key. However doing this only seems to create a public key, and at no point was I prompted to create a password (albeit a blank one for my purposes.) I then went to try and copy the key across using ftp, only to find out that the .ssh folder is hidden. Do I need to change the permissions on the folder?

    Am I doing this in the correct direction? i.e creating the key on Humax and then importing it into the Diskstation?

    Apologise for all the questions but I really do want to conquer this challenge!
     
  13. af123

    af123 Administrator Staff Member

    You're doing it the right way round. You create the key pair which consists of a private and public part on the Humax, then transfer the public part to the NAS.

    The id_rsa file ends up with the private part in it and the public part is shown on the screen and that is what you need to copy over to the NAS. Usually it would be put into a file called authorized_keys on the NAS but that will depend on what system it is running.
     
  14. Placemat

    Placemat New Member

    The NAS is running Busybox with an ash shell profile. So it is simply just a case of copying the key fingerprint information that appears when running the Dropbearkey command, into a new file called authorized_keys on the NAS?

    I have created a ".ssh" folder on my NAS, and using

    Code:
    touch .ssh/authorized_keys
    I have established a blank file for inputting the key information. How do I go about writing the information to the file?
     
  15. af123

    af123 Administrator Staff Member

    The vi editor is probably available on the NAS but it isn't easy to use. Can you create the file elsewhere then transfer it to your NAS?
    If you can't see the .ssh directory then you can always move the file later from the NAS command line - as well as fix the permissions which will be required for it all to work properly.

    Assuming you managed to transfer it to root's home directory on the NAS then:

    Code:
    # mv authorized_keys .ssh/authorized_keys
    # chmod 700 .ssh
    # chmod 644 .ssh/authorized_keys
    
    would move the file into the .ssh directory and set the correct permissions on it and the directory.
     
  16. Placemat

    Placemat New Member

    Well I had a go with VI as scp is disabled on my NAS and I'm running Windows on my PC (so I don't know how I would go about editing the file.) This is what I have achieved so far:

    Cleared any existing .ssh directory and created a new .ssh directory in /mod by:

    Code:
    Connect via SSH to the Humax
    rm -rf .ssh
    mkdir .ssh
    
    Then I generated the SSH key in the new folder:

    Code:
    dropbearkey -t rsa -f /mod/.ssh/id_rsa


    The output provides me with the public key information and the private key file (id_rsa) so I copied:

    Code:
    ssh-rsa "XXXXXXsuperlongpasscodeXXXXX" root@humax
    I ignored the "fingerprint" part of the output as I read somewhere that this was not necessary.

    Then I connected via SSH to my NAS as root

    Code:
    Ensuring I was in root/
    mkdir .ssh
    touch .ssh/authorized_keys
    vi .ssh/authroized keys
    i + right click to paste public key information
    :wq
    Then to configure the NAS to use the key:

    Code:
     vi /etc/ssh/sshd_config 
    and changed

    Code:
    #RSAAuthentication yes
    #PubkeyAuthentication yes
    #AuthorizedKeysFile .ssh/authorized_keys
    to

    Code:
    #RSAAuthentication yes
    PubkeyAuthentication yes
    AuthorizedKeysFile .ssh/authorized_keys
    *lastly I changed the permissions as you advised*
    and of course after the reboot it still prompts for password!:frantic:

    I'm wondering whether the Dropbearkey system is even compatible with Busybox on the NAS. I know from reading in forums that Dropbear and OpenSSH use different formats for the keys, requiring conversion between the two. Everywhere I look it says that the keygen should produce an id_rsa and id_rsa.pub file, which Dropbear does not do. Maybe I could replace Dropbear with OpenSSH?
     
  17. af123

    af123 Administrator Staff Member

    They are definitely compatible and you wouldn't want to run OpenSSH on the Humax (dropbear is a small SSH implementation for embedded systems). While the on-disk key file formats are different, what was shown on your screen should be ok.

    Have you checked permissions on the NAS? The .ssh directory must be mode 700 or less (500 and 100 would work too but 700 makes most sense, before anyone asks!)
     
  18. af123

    af123 Administrator Staff Member

    Just re-read your post and saw the bit about permissions (this isn't easy on a mobile 'phone!)

    The other common problem is that the key in authorized_keys must all be in one line. Open it in vi again and press control-g to see how many lines are in there. If more than one then press J on the first line to join the second one to it and then press x to remove the extra space. Repeat until all on one line then ZZ to save and exit.
     
  19. af123

    af123 Administrator Staff Member

    I have another idea too but will need to try it at home first...
     
  20. Placemat

    Placemat New Member

    Ok cool, I'm pretty much out of ideas. Without any background knowledge in Linux its pretty impossible for me to troubleshoot the problem :S

    Oh and I checked the key, all appears on one line.