• The forum software that supports hummy.tv will be upgraded to XenForo 2.3 on Wednesday the 20th of November 2024 starting at 7pm

    There will be some periods where the forum is unavailable, please bear with us. More details can be found in the upgrade thread.

Webif vs rsync/scp file copy speed

J

johnb

Member
I've set up two HDDs on my HP Microserver as network shares and I've noticed quite a difference in speed if I copy files using the Webif interface compared with either rsync or scp (both of those "pulling" the data to the Microserver, over ssh).

Webif file copy to network share (HP Microserver): ~ 23 GB/hr
rsync or scp to HP Microserver using ssh: ~7.5 GB/hr (no compression used)

I wonder what the explanation is and whether something is wrong in how I have set up ssh or in how I am using rsync or scp.
 
The difference could easily be due to the overhead of encryption on the CPU.
 
rsync and scp are generally just easier, particularly if you have public key authentication configured.
I do tend to use NFS from my Mac laptop though.
 
af123 said:
The difference could easily be due to the overhead of encryption on the CPU.
Click to expand...
Agreed. The CPU hasn't got much poke, as we well know.
af123 said:
rsync and scp are generally just easier, particularly if you have public key authentication configured.
Click to expand...
Agreed again. I have and it makes things easy.

Aside: I discovered sshfs yesterday and was trying to use it to make remote files appear locally securely. It was a bit too much of a fiddle to get users/groups/permissions right though, but maybe I'll have another go.
 
af123 said:
The difference could easily be due to the overhead of encryption on the CPU.
Click to expand...

Many thanks to everyone. Of course, I should have thought of that (though I am new to the intricacies of rsync and ssh).

I have been backing to my Hummy My Video once a week to an external drive via USB and I thought it might be more convenient to move the backup to my HP Microserver (running Ubuntu Server and mainly used as a glorified NAS to hold backups, video and music files and run squeezebox server).

I remembered that I installed the samba package on the HDR some time ago, so I mounted the Humax My Video share on the Microserver and tested rsync (running on the Microserver), sync'ing between the HDR My Video share and a directory on the Microserver - the result was ~33 GB/hr (vs ~25 GB/hr via the HDR's USB port). Not brilliant but acceptable.

By the way, I realise that most people leave their Windows workgroup unchanged as "WORKGROUP" and it is great that the samba package has that preset. But, for those who have changed the workgroup name from the Windows default - would it be possible to have an option in WebIf to edit the workgroup name used in Samba package?

It would save having to edit the smb.conf file with vi (which is possibly the least user-friendly piece of software I have ever used).
 
johnb said:
By the way, I realise that most people leave their Windows workgroup unchanged as "WORKGROUP" and it is great that the samba package has that preset. But, for those who have changed the workgroup name from the Windows default - would it be possible to have an option in WebIf to edit the workgroup name used in Samba package?

It would save having to edit the smb.conf file with vi (which is possibly the least user-friendly piece of software I have ever used).
Click to expand...
Yup, I agree with all that. You can install nano which is somewhat more friendly (it's all relative though).
 
Further information: WebIF >> Diagnostics >> File Editor >> Open, then browse to the file in question.

It seems like a poor use of effort to provide a graphical facility for something relatively few people would need, particularly when there is another way to skin the same cat.
 
If it isn't already then the file could easily be added to the quick edit links on the file editor page.
 
Thanks for the replies.

Re: Samba - Perhaps the "Custom Firmware Package Notes" on Samba could include a trouble shooting comment to the effect that:

Check whether Samba is enabled on the Humax (WebIf "Services).
Check whether the Workgroup of the Windows PC has been changed from the default of "WORKGROUP". If so, use the WebIf File Editor (available in WebIf "Diagnostics") to amend /mod/etc/smb.conf,​

As I understand it (though I might be wrong) having a different workgroup name doesn't actually stop Windows from seeing the Humax (or any other Samba share) but it becomes erratic whether or not the share is picked up. Having the same workgroup name greatly assists Windows in finding the Samba share.

As far as my exploration of backing up the Humax "My Video" is concerned I've finally come up with a method which is reasonably fast and effective, using the NFS package. I am no expert <cough> so this has been trial and error - there are probably much better ways of achieving the same ends, anyway -

Options tried:

A) Using rsync on the Humax to back up to a USB HDD (WD Green): works fine @ ~25 GiB/hr and is the simplest option to set up. However, I would ideally like to backup the Humax My Video over the network to my HP Microserver (running Ubuntu Server OS and WD Green HDDs), rather than using external HDDs.

B) Using stand-alone rsync (i.e. without rsync daemon at the other end) on the Humax to backup to network connected HP Microserver running Ubuntu Server. Rsync, by default uses ssh for this and the overhead imposed by ssh means the speed is so slow as to make the method impractical.

C) Using stand-alone rsync on a HP Microserver to "pull" data from the Humax: faster than using rsync on the Humax but still very slow (for the same reason) at ~ 7.5GiB/hr.

D) Set up Samba on the Humax, mounted the samba share on the HP Microserver, then ran rsync on the Microserver to sync with the Samba share. This is much faster @ ~ 33 GiB/hr but some files were picked up as different (and needed copying) which rsync had already sync'd when accessing the Humax files directly (probably due to the way Samba 'translates' linux to be Windows compatible). Also Samba converted some of the non-Windows compatible filenames in, say, the "[xxxxx]" network shares folders to DOS 8.3 filenames. None of that is vital, as I am mainly interested in the video files, but I thought I would try the native linux NFS share instead of Samba (NFS this is only useful for sharing between Linux boxes, not between Linux and Windows):

D) Installed the NFS package on the Humax (and the "nfs-common" on the Microserver). Mounted the NFS share on the Microserver and ran rsync on the Microserver to sync with the NFS share. This worked perfectly with a transfer speed of ~36 GiB/hr (equivalent to ~82 Mbit/sec - not bad for the 100 Mbit/sec Humax ethernet connection).


I am very much a novice at all this - so apologies for any crass mistakes.
 
You must log in or register to reply here.
Back
Top