• The forum software that supports hummy.tv will be upgraded to XenForo 2.3 on Wednesday the 20th of November 2024 starting at 7pm

    There will be some periods where the forum is unavailable, please bear with us. More details can be found in the upgrade thread.

Automating rsync backups

P

Placemat

New Member
Hello there chaps,

I am a first time writer, long time reader on this forum and I would like to start by giving thanks to the geniuses here that have contributed to the customised HDR-FOX T2 firmware. It has turned my bog standard PVR into a a fantastic box of tricks.

Now to my query which I hope someone can help me with. I am trying to make mirror backups of the content of my HDR-FOX T2 internal hardrive onto a Synology Diskstation. As the HDR-FOX T2 supports rsync and my Synology supports acting as a rsync destination, I thought this would be the best method for backups.

My understanding of Linux is poor which made the first step difficult enough. After reading some rysnc guides and some trial and error, I finally identified the rsync command that was applicable to my needs


rsync -avr --progress --delete --exclude 'Tsr/' mnt/hd2/ root@synology ip::NetBackup

Running this command seemed to work well (any suggestions for improvements to the command would be appreciated.)

The next step I want to achieve is to automate the rsync process, so that I can ask the Humax to run a backup every evening at a particular time. I already have the Humax set to power on at 2am-6am to run the unencrypt package, so ideally I want to have it run after this step.

This is where I am completely stuck, and no amount of searching this forum or complicated Linux guides will take me any further. Actually I note that there is little reference made to rsync on this forum, so I wonder if anyone is actually using it at all.

I would really appreciate any help on the next steps in this process, where do I start?
 
To auto-run rsync on a regular basis you can setup a cron job, e.g. here is the line that kicks off unencrypt each night :-
1,31 1-6 * * * /mod/sbin/unencrypt "/mnt/hd2"
For notes on Cron in the WiKi, CLICK HERE
 
Nicely done. You just need to add a line to the crontab file to kick the job off when you want. The webif built-in file editor is probably the easiest way to do it, just edit the /mod/var/spool/crontabs/root file and a add a line similar to this one:

Code: 
0 3 * * * /mod/bin/rsync -avr --progress --delete --exclude 'Tsr/' mnt/hd2/ root@synology ip::NetBackup

The first five fields are <minute> <hour> <day of the month> <month of the year> <weekday> with a * meaning any, so the line above will cause the command to be run at 3am each day, providing the box is not in standby at the time.
 
Ah I see, thank you for replying.

There is an issue that I neglected to put in the original post, my Synology requires a password before the rsync command can be run.

Now I understand I can just store a default password for rysnc on the Humax (although my attempts to do this failed?) but this is obviously an unsecured method. I've also been reading about using SSH for shared access keys but the mind boggles with such information. Do either of you have any suggestions?
 
The usual way is to use shared SSH keys. It isn't as hard as it sounds.
You need to create a key pair on the Humax then put the public part of that key on to your Synology somehow.
If nobody beats me to it I'll post the keypair generation command later when I've looked it up.
 
While I await your instructions on setting up keypairs, I thought I would have a go at running rsync in a SSH environment. I enabled SSH on my Synology Diskstation then opened up a SSH connection in Putty to the the Humax and ran the following command:


Code: 
 humax# rsync -avr --delete --progress --exclude '[Deleted Items]' ssh mnt/hd2/My_Video/ root@192.168.0.11:NetBackup

I've tried to refine the command a little bit with respect to only copying the "My Video" directory and exclude the "Deleted Items" folder as it's a wasteful area to process.

All I got in return for my effort was this:

Code: 
humax# rsync -avr --delete --progress --exclude '[Deleted Items]' ssh mnt/hd2/My_Video/ root@Synology ip:NetBackup
root@Synology ip password:
sending incremental file list
rsync: link_stat "/mnt/hd2/mod/ssh" failed: No such file or directory (2)
rsync: change_dir "/mnt/hd2/mod/mnt/hd2/My_Video" failed: No such file or directory (2)
 
 
sent 36 bytes received 12 bytes 3.84 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.8]

Am I correct in thinking that when working in SSH, folder structures appear differently? I don't think I have specified the correct address for transfer.

I have Dropbear installed and I haven't tweaked it in any way.
 
I'm not sure what you are trying to do here. rsync will already be using SSH as its underlying transport. Your first command was fine, you just need to set up the keys and the password prompt should go away. I don't know how you set up the keys on the Synology - on a UNIX system you put them in a file called authorized_keys (US spelling) under a directory called .ssh within the home directory.
 
Placemat said:
Code: 
 humax# rsync -avr --delete --progress --exclude '[Deleted Items]' ssh mnt/hd2/My_Video/ root@192.168.0.11:NetBackup
Click to expand...

Just a small point (for future reference) there is a space rather than an under-score between My and Video e.g. ssh "mnt/hd2/My Video/"
 
The corresponding command for dropbear is

Code: 
dropbearkey -t rsa -f /mod/.ssh/id_rsa
 
Nice work Placemat :)

I haven't read the full thread, but do bear in mind that cron jobs can fail if they produce error output while running. The magic incantation ' 2>&1' on the end of the crontab line (you can see it in the examples in the Wiki) solves that.

In Linux, writing to the error stream does not mean that things have gone horribly wrong, it's just a way of making sure that the user sees important issues, even if the rest of the output is being redirected elsewhere, such as to a file.
 
Thanks very much, although much of my success is blind trial and error.

Thanks for the dropbearkey command af123, I've managed to set up the .ssh/ folder and create the id_rsa key. However doing this only seems to create a public key, and at no point was I prompted to create a password (albeit a blank one for my purposes.) I then went to try and copy the key across using ftp, only to find out that the .ssh folder is hidden. Do I need to change the permissions on the folder?

Am I doing this in the correct direction? i.e creating the key on Humax and then importing it into the Diskstation?

Apologise for all the questions but I really do want to conquer this challenge!
 
You're doing it the right way round. You create the key pair which consists of a private and public part on the Humax, then transfer the public part to the NAS.

The id_rsa file ends up with the private part in it and the public part is shown on the screen and that is what you need to copy over to the NAS. Usually it would be put into a file called authorized_keys on the NAS but that will depend on what system it is running.
 
The NAS is running Busybox with an ash shell profile. So it is simply just a case of copying the key fingerprint information that appears when running the Dropbearkey command, into a new file called authorized_keys on the NAS?

I have created a ".ssh" folder on my NAS, and using

Code: 
touch .ssh/authorized_keys

I have established a blank file for inputting the key information. How do I go about writing the information to the file?
 
The vi editor is probably available on the NAS but it isn't easy to use. Can you create the file elsewhere then transfer it to your NAS?
If you can't see the .ssh directory then you can always move the file later from the NAS command line - as well as fix the permissions which will be required for it all to work properly.

Assuming you managed to transfer it to root's home directory on the NAS then:

Code: 
# mv authorized_keys .ssh/authorized_keys
# chmod 700 .ssh
# chmod 644 .ssh/authorized_keys

would move the file into the .ssh directory and set the correct permissions on it and the directory.
 
Well I had a go with VI as scp is disabled on my NAS and I'm running Windows on my PC (so I don't know how I would go about editing the file.) This is what I have achieved so far:

Cleared any existing .ssh directory and created a new .ssh directory in /mod by:

Code: 
Connect via SSH to the Humax
rm -rf .ssh
mkdir .ssh

Then I generated the SSH key in the new folder:

Code: 
dropbearkey -t rsa -f /mod/.ssh/id_rsa
The output provides me with the public key information and the private key file (id_rsa) so I copied:

Code: 
ssh-rsa "XXXXXXsuperlongpasscodeXXXXX" root@humax

I ignored the "fingerprint" part of the output as I read somewhere that this was not necessary.

Then I connected via SSH to my NAS as root

Code: 
Ensuring I was in root/
mkdir .ssh
touch .ssh/authorized_keys
vi .ssh/authroized keys
i + right click to paste public key information
:wq

Then to configure the NAS to use the key:

Code: 
 vi /etc/ssh/sshd_config

and changed

Code: 
#RSAAuthentication yes
#PubkeyAuthentication yes
#AuthorizedKeysFile .ssh/authorized_keys

to

Code: 
#RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

*lastly I changed the permissions as you advised*
and of course after the reboot it still prompts for password!:frantic:

I'm wondering whether the Dropbearkey system is even compatible with Busybox on the NAS. I know from reading in forums that Dropbear and OpenSSH use different formats for the keys, requiring conversion between the two. Everywhere I look it says that the keygen should produce an id_rsa and id_rsa.pub file, which Dropbear does not do. Maybe I could replace Dropbear with OpenSSH?
 
They are definitely compatible and you wouldn't want to run OpenSSH on the Humax (dropbear is a small SSH implementation for embedded systems). While the on-disk key file formats are different, what was shown on your screen should be ok.

Have you checked permissions on the NAS? The .ssh directory must be mode 700 or less (500 and 100 would work too but 700 makes most sense, before anyone asks!)
 
Just re-read your post and saw the bit about permissions (this isn't easy on a mobile 'phone!)

The other common problem is that the key in authorized_keys must all be in one line. Open it in vi again and press control-g to see how many lines are in there. If more than one then press J on the first line to join the second one to it and then press x to remove the extra space. Repeat until all on one line then ZZ to save and exit.
 
Ok cool, I'm pretty much out of ideas. Without any background knowledge in Linux its pretty impossible for me to troubleshoot the problem :S

Oh and I checked the key, all appears on one line.
 
You must log in or register to reply here.
Back
Top