rsync push to humax stalling

Dinosaur · Nov 19, 2019

Having replaced a failing 1GB hard disk with a 2GB one, I did not expect to be able to recover anything from the old disk.

I was pleasantly surprised to be able to copy most of it to some spare space attached to a Raspberry Pi 4B.

So, now, after some re-organisation, I am s l o w l y using rsync to push to the HDR Fox T2.

The problem I am having is that rsync stalls on the pi whenever a scheduled recording occurs on the fox. I have to ^C and restart rsync.

Anybody know why?

I can tell that no files are being copied by connecting to both via Nautilus and also via ssh.

/df · Nov 19, 2019

Are you rsync-ing to a daemon rsync on the HDR, or to a file share on the HDR?

If the former, perhaps the --timeout or --contimeout options might be useful, or switch the master and daemon roles?

I normally do this with rsync running on the destination HDR and a share or USB mount as the source.

Dinosaur · Nov 19, 2019

/df said:
Are you rsync-ing to a daemon rsync on the HDR, or to a file share on the HDR?

If the former, perhaps the --timeout or --contimeout options might be useful, or switch the master and daemon roles?

I normally do this with rsync running on the destination HDR and a share or USB mount as the source.

No, I'm not using a daemon on either end, nor am I using a Windows share. rsync on the pi is connecting using ssh to run rsync om the fox.

I'm using something like ssh -vvzrst <source dir>/ root@192.168.x.x:'/mnt/sd2.My Video'

So, I'm using push rather than pull and no daemon. I realise that it would be quicker to connect the usb3 hdd to the fox, but it is needed on the pi for syncthing.

Obviously, I have the rsync and ssh packages installed on the fox.

/df · Nov 20, 2019

AFAIK the SSH mode of rsync runs the remote rsync server in the remote SSH session using SSH to transport a pipe from the rsync client (as opposed to directly over TCP to a listening rsync in the daemon mode).

No doubt the HDR is busy when recording and the server side of the rsync/ssh link might get blocked, hang or crash through lack of resources. I don't think the rsync timeout options that I mentioned will help. You can enable TCP- and application-level keep-alives for SSH, but I guess that will just make it possible to break a hung remote session earlier. It's possible that the daemon mode would stress the HDR less than the rsync/ssh combo and at least it has fewer failure modes. Or, as previously suggested, share the source or destination folder with NFS or Samba and just use one rsync instance on the other side.

The rsync option --partial-dir=.rsync-partial is recommended, per the man page, if your transfer involves large files that may have to be restarted.

Dinosaur · Nov 21, 2019

Looks like out of memory on the humax. I didn't realise how little memory it has, until I checked. So rsync was probably the victim of oom.

I notce that rsync on the humax runs with a niceness of 10, like most taskd, as against 0 for humaxtv and more important stuff.

Don't think it's worth increasing swap space.

Thanks for the partial-dir tip.

Black Hole · Nov 21, 2019

Dinosaur said:
Don't think it's worth increasing swap space.

Why not? Do you have any swap space installed at all? See swapper: https://hummy.tv/forum/threads/swapper.8844/

prpr · Nov 21, 2019

Swapper is a dependency of WebIf (via Lighttpd) these days, so everyone has it.

Dinosaur · Nov 21, 2019

Black Hole said:
Why not? Do you have any swap space installed at all? See swapper: https://hummy.tv/forum/threads/swapper.8844/

Yes, I do, but only 128M. I'll try 1GiB.

prpr · Nov 21, 2019

Have you tried looking at "free -m" to see what it's actually using?

Dinosaur · Nov 21, 2019

Just killed 2 birds with 1 stone.

I have increased swap space successfully.

Code:

top - 17:52:57 up 25 min,  1 user,  load average: 2.06, 2.12, 1.35
Tasks:  65 total,   2 running,  63 sleeping,   0 stopped,   0 zombie
Cpu(s):  6.5%us,  8.6%sy, 50.5%ni, 20.2%id, 10.9%wa,  0.0%hi,  3.3%si,  0.0%st
Mem:    125016k total,   122180k used,     2836k free,      916k buffers
Swap:  1048568k total,      156k used,  1048412k free,    77828k cached

Code:

humax# ps -ef|grep swap
root        16     9  1 17:27 ?        00:00:01 [kswapd0]
root      2573  2555  0 17:29 pts/0    00:00:00 grep swap
humax# ls -lah /mnt/hd3
total 1.1G
drwxr-xr-x 2 root root 4.0K Nov 21 17:28 .
drwxr-xr-x 7 root root   84 Mar 15  2016 ..
-rw-rw-rw- 1 root root 1.0G Nov 21 17:28 .swap0

I did it the fast and nasty way. Iturned off swapping, deleted the swap file, changed /mod/etc/init.d/S00swapper and rebooted.

The change was to the iteration count.

After I rebooted, I got the benefit of --partial-dir.

btw, why a swap file, not a swap partition?

prpr · Nov 21, 2019

Dinosaur said:
changed /mod/etc/init.d/S00swapper and rebooted

All you needed was:

Code:

humax# /mod/etc/init.d/S00swapper stop
humax# /mod/etc/init.d/S00swapper start

having edited the file. A reboot is unnecessary.

btw, why a swap file, not a swap partition?

Because the stock firmware does the partitioning and doesn't use a swap file.
There is no practical difference anyway since 2.6 kernel, and files are easier to manage than partitions, as you have just proved.

Dinosaur · Nov 21, 2019

Code:
All you needed was:

Code:

humax# /mod/etc/init.d/S00swapper stop humax# /mod/etc/init.d/S00swapper start

having edited the file. A reboot is unnecessary.

Not quite all. I would still need to delete the old swap file.

Anyway, my way was simple too.

Dinosaur · Nov 21, 2019

Success. rsync has survived a recording.

Can I suggest a change to the next release to increase the size of the swap file.

prpr · Nov 21, 2019

Dinosaur said:
Not quite all. I would still need to delete the old swap file.

I was forgetting my enhanced version did things (properly) that the repository version doesn't.

Dinosaur said:
Can I suggest a change to the next release to increase the size of the swap file.

The trouble is that significantly increasing the size significantly increases the delay in startup. But it could be made configurable.

/df · Nov 21, 2019

Especially if the swap size is configurable, there should be a check that the swap file isn't using too much of the disk, perhaps reducing the configured setting to leave at least a certain fraction or actual amount, whichever is larger, of free space. Obviously this is more relevant to HDs, where all recordings, not just the streamer file, share space with the swap file.

Dinosaur · Nov 22, 2019

The trouble is that significantly increasing the size significantly increases the delay in startup. But it could be made configurable.

It should only delay startup the first time it runs. The bit which takes the time is the dd filling the file with nulls. This only needs to run once for each new size.

Especially if the swap size is configurable, there should be a check that the swap file isn't using too much of the disk, perhaps reducing the configured setting to leave at least a certain fraction or actual amount, whichever is larger, of free space. Obviously this is more relevant to HDs, where all recordings, not just the streamer file, share space with the swap file.

This doesn't apply, as the usual delay is minimal. On the HDR the swap file seems to be the only file on a 10GiB partition - so it doesn't share space.
The package exits immediately without doing anything on an HD.

Here's a slight change which allows anyone to configure the size by setting a constant. Arguably, anyone who can't find the file to edit shouldn't change it. So not quite anyone...

Code:

#!/bin/sh
set +x
########### /mod/etc/init.d/S00swapper ###########

# size of swapspace in MiB.  Don't exceed the size of the partition  (~10GiB)
swapsize=1024

swappref='/mnt/hd3/.swap'
swapfile="$swappref$swapsize"

[ "`cat /etc/model`" = "HDR" ] || exit 0

case "$1" in
    start)
    [ -f $swapfile ] || { rm "$swappref"* &2>&1 >/dev/null
                          dd if=/dev/zero of=$swapfile bs=1M count=$swapsize
                          mkswap $swapfile
                        }
    swapon $swapfile
    ;;
    stop)
    swapoff $swapfile
    ;;
    *)
    exit 1
    ;;
esac

exit 0

It needs further modification to make it safe when stopping and starting manually to change the swap file size. Rebooting is safe.

edited a bit for copy and paste errors in code

Black Hole · Nov 22, 2019

Dinosaur said:
On the HDR the swap file seems to be the only file on a 10GiB partition - so it doesn't share space.

Not true:

Code:

HDRFOX4# ls mnt/hd1                                                                              
dvbepg  epg.db
HDRFOX4# ls mnt/hd2                                                                              
My Music      My Video      dms_cds.db    thumbnail
My Photo      Tsr           mod           virtual_disk
HDRFOX4# ls mnt/hd3                                                                              
Streamer_down_file    cookie.txt            maintenance.boot.log                                  
HDRFOX4# ls mnt/hd4
HDRFOX4# ls -lah mnt/hd3                                                                          
drwxr-xr-x    2 root     root        4.0K Apr 25  2017 .
drwxr-xr-x    7 root     root          84 Mar 15  2016 ..
-rw-rw-rw-    1 root     root      128.0M Nov 16 13:39 .swap0                                    
-rw-r--r--    1 root     root      339.7M Nov  9  2017 Streamer_down_file                        
-rw-r--r--    1 root     root         530 Jul 20  2013 cookie.txt                                
-rw-r--r--    1 root     root        2.1K Apr 25  2017 maintenance.boot.log                      
HDRFOX4#

Streamer_down_file is the buffer the TV Portal uses for streaming, and from where we can save off complete streamed media files.

hd4 seems to be empty though.

What I don't understand is what the (empty) virtual_disk folder is doing in hd2. Is that a hang over from a previous virtual disk before it became a WebIF dependency?

Dinosaur · Nov 22, 2019

I've changed it to swapoff any active swap file before removing it.

Code:

#!/bin/sh
set +x
########### /mod/etc/init.d/S00swapper ###########

# size of swapspace in MiB.
swapsize=1024

swappref='/mnt/hd3/.swap'
swapfile="$swappref$swapsize"

[ "`cat /etc/model`" = "HDR" ] || exit 0

case "$1" in
    start)
    [ -f $swapfile ] || {
            cat /proc/swaps | while read fname type rest; do
                [ ! "$type" = "file" ] || swapoff "$fname"
        done

        rm "$swappref"* &2>&1 >/dev/null
         dd if=/dev/zero of=$swapfile bs=1M count=$swapsize
        mkswap $swapfile
    }

    swapon $swapfile
    ;;
    stop)
    swapoff $swapfile
    ;;
    *)
    exit 1
    ;;
esac

exit 0

/df · Nov 22, 2019

Dinosaur said:
...
The package exits immediately without doing anything on an HD.
...

I was referring to the "enhanced version" mentioned by @prpr, which is extended for HDs to make things like youtube-dl work more reliably. It's linked from the package thread quoted above by @BH.

I know that I put the file space check in when I did a similar enhancement and it was a real pain to do using the bare CF Busybox tools, so I must have had a good reason.

Black Hole said:
...
Streamer_down_file is the buffer the TV Portal uses for streaming, and from where we can save off complete streamed media files.

hd4 seems to be empty though.

It's an unused mount point. You won't get far trying to create files there!

What I don't understand is what the (empty) virtual_disk folder is doing in hd2. Is that a hang over from a previous virtual disk before it became a WebIF dependency?

It's the virtual device folder remounted; I suppose, into a known place.

Code:

# see /mod/etc/init.d/S89virtual-disk l.5

if [ -d /media/drive1/mod ]; then
      vdisk=/media/drive1/virtual_disk
else
      vdisk=/mnt/hd2/virtual_disk
fi

# this would be better
case  "$(cat /etc/model)" in 
HD)
      vdisk=/media/drive1/virtual_disk
      ;;
HDR)
      vdisk=/mnt/hd2/virtual_disk
      ;;
*)
      exit 1
esac

prpr · Nov 22, 2019

For reference, my swapper enhanced version:
https://hummy.tv/forum/threads/yout...com-or-other-video-platforms.8462/post-122502

rsync push to humax stalling

New Member

Well-Known Member

New Member

Well-Known Member

New Member

May contain traces of nut

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

New Member

Well-Known Member

Well-Known Member

New Member

May contain traces of nut

New Member

Well-Known Member

Well-Known Member