Webif crop feature stresses the processor

[/df]

Is the box running a real-time Linux? If not (and it almost certainly isn't), there is no control over when a current process will suspend to allow even a higher priority process to get in (Real-time OSes are specifically written to guarantee service response times for processes injecting a high-priority interrupt). It strikes me this may be the problem rather than priorities themselves - the Humax process doesn't expect to be competing for resources, so they had no need to use an expensive RT OS.
-
However, no harm in trying it. Bumping up the Humax process priority is a great idea (instead of fiddling with all the CF processes).

The idea here is about IO priorities which apply within the Linux IO subsystem, separately from process scheduling priorities, even if the default IO priority is set from the process priority. Clearly the Humax Linux isn't a hard real-time version (I suppose building a CF with a hard real-time kernel falls into the category of term project), but it does have the O(1) scheduler with soft real-time scheduling (see section 5.8 here). There may well be a view, as with Br***t, that soft real-time scheduling is RINO (Real-time In Name Only). A simple ps -Al* will show the "nice" values of processes and in particular will show the less nice (higher priority, NI=0) character of the humaxtv process compared with CF processes (typically NI=10)

The Linux kernel with the CFQ IO scheduler can manage (synchronous) IO operations based on IO priorities internally regardless of the process scheduler.

It's easier than that, put a script in /mod/boot/bootstrap.d/ and it will be called with the process ID of the humaxtv process just after it is started.

Cool, so something like this as /mod/boot/bootstrap.d/ioboost:

#!/bin/sh
type ionice || exit 0
pid=$1
[ -z "$pid" ] && pid=`pgrep humaxtv`
[ -n "$pid"] && ionice -p $pid -c1

* You need the procps package loaded, as apparently Busybox can't be bothered with scheduling, but cut -f19 -d' ' /proc/${pid_of_process}/stat will display the "nice" value from the missing ps NI column for the process whose PID is ${pid_of_process}.

 

[af123]

Cool, so something like this as /mod/boot/bootstrap.d/ioboost:

The disk is not mounted at this point so you have limited utilities (probably not pgrep) and you also need to put ionice somewhere like /var/lib/humaxtv_backup/mod and use that path for it.

 

[prpr]

If you're doing this after humaxtv startup, can it not just be done in /mod/etc/init.d/ instead, when the disk is available? It doesn't really matter when it is done.

 

[/df]

The disk is not mounted at this point so you have limited utilities (probably not pgrep) and you also need to put ionice somewhere like /var/lib/humaxtv_backup/mod and use that path for it.

Exactly, pgrep is in the Busybox package, not in the CF one, so it can't be relied on except in a package with a dependency on the BB package, or procps. But it is mentioned in /mod/boot/bootstrap.d/xrts, albeit in an unreachable code path -- in fact that script is almost all an unreachable code path (?).

If you're doing this after humaxtv startup, can it not just be done in /mod/etc/init.d/ instead, when the disk is available? It doesn't really matter when it is done.

Perhaps not -- but I suppose we'd better find out if adjusting IO scheduling works first.

 

[MymsMan]

My thoughts for initial testing would be to create multiple copies of a large file then whilst recording a HD programme delete the large files using rm and trm with no ionice, ionice -c2 -n7, and ionice -c3 timing the executions and then playing the recording to see if any pixellation is evident at the points of execution.

Part of the challenge will be to find a window when I am not actually recording something I want to watch later.

My intial testing is mixed, using ionice is mixed.
Even at the most extreme settings with HumaxTV running in the Real time class and running Copies and Deletes in the Idle class did not noticeably increase the execution time of the copies and deletes. But while pixellation was reduced it was not totally eliminated

The Linux 2.6.18 IO subsystem supports several scheduling algorithms. By default it uses anticipatory, but you can change that to the Completely Fair Queuing algorithm to which ionice applies.

The scheduling algorithms article is very interesting, especially with regard to the tuning parameters available and the normal prioritization of read over write.
To avoid pixellation we probably need a higher write priority .

 

[/df]

My intial testing is mixed, using ionice is mixed.
Even at the most extreme settings with HumaxTV running in the Real time class and running Copies and Deletes in the Idle class did not noticeably increase the execution time of the copies and deletes. But while pixellation was reduced it was not totally eliminated

The scheduling algorithms article is very interesting, especially with regard to the tuning parameters available and the normal prioritization of read over write.
To avoid pixellation we probably need a higher write priority.

1. The CFQ IO scheduler only applies to synchronous IO operations.
2. Per my earlier post, it's possible that HumaxTV bypasses the normal disk IO for recording and playing (if the disk firmware supports ATA Streaming), and it's not clear to me how the scheduling algorithm would treat such operations. WIth copy/delete at Idle, is there any difference between HumaxTV at Best-effort or None vs Real-time?
3. Unfortunately this build of the Linux kernel doesn't support block IO tracing which might otherwise provide enlightenment.
4. The strace in the CF repo might reveal the IO operations used by the HumaxTV process, but hooking into the running process strace -fF -e trace=file,desc -p $HumaxTV_pid SEGVs a load of threads in short order, crashing the process. Possibly the HumaxTV process could be launched from Maintenance mode?

Anyhow, here's an init script that could be installed (chmod +x ...) as eg /mod/etc/init.d/S00htvboost:

#!/bin/sh
# set Humax TV process to real-time priority under CFQ IO scheduler

type ionice >/dev/null || exit 1

# fn to get the device on which a file's filesystem is mounted
# would use 'df $file | grep -oE "/dev/.d."' but Busybox-1.20.2 df is confused
# by virtual-disk2
mntdev() { # pathname
    local f="$(readlink -f "$1")"
    local mpt
    local pnm
    local mlen=0
    local nlen
    
    # find longest mountpoint that's an initial substring of the pathname 
    cat "/proc/self/mounts" |
      ( while read -r dev mnt _; do
            local dev
            local mnt
            pnm="${f##"$mnt"/}"
            if [ "$pnm" != "$f" ]; then
                nlen=${#mnt}
                if [ $nlen -gt $mlen ]; then
                    mpt="$dev"
                    mlen=$nlen
                fi
            fi
        done;
        echo "${mpt:-/dev/null}" )
}

# is given IO scheduler set to CFQ?
cfqP() { # scheduler
    grep -qF "[cfq]" "$1"
}

# set CFQ in given IO scheduler
cfqset() { # scheduler
    printf "cfq">"$1"
}

# get selected scheduler (or other [selected]-type /proc setting) from file) 
get_sel() { # scheduler
    grep -oE "\[[^]]+" "$1" | cut -c 2-
}

# en/dis/able CFQ for session
cfq_toggle() {
    local dsk="$(mntdev /mod | grep -oE "/dev/.d.")"
    local sched
    local osched="/tmp/.iosched"
    sched="/sys/block/${dsk##/dev/}/queue/scheduler"
    [ -w "$sched" ] || return 1
    case "$1" in
    start)
        cfqP "$sched" ||
            { [ -r "$osched" ] || get_sel "$sched" >"$osched"; cfqset "$sched"; }
        cfqP "$sched" || return 2
        ;;
    stop)
        [ -r "$osched" ] && { cat "$osched" > "$sched"; rm "$osched"; } || return 2
        ;;
    esac
    return 0
}

# like the status command, look only for the most recent HumaxTV process
pid="$(pgrep -n humaxtv)"

case "$1" in
    start|restart)
        cfq_toggle start || exit $?
        [ -n "$pid" ] && ionice -p $pid -c1
        ;;
    stop)
        [ -n "$pid" ] && ionice -p $pid -c0
        cfq_toggle stop || exit $?
        ;; 
    *)
        exit 1 ;;
esac

 

[mihaid]

just witnessed a crop also producing major pixellation on the recording. just as well it was not a recording I will keep.

 

[Black Hole]

Here's an idea I don't recall having been suggested (but it seems only a remote possibility): the extra processing going on could produce more-than-usual RF interference, which bolts out the signal to the tuners, rather than the processing itself causing problems with the disk accesses.

 

[MymsMan]

In my testing* I was unable to significantly affect pixelation by tweaking the IO tuning parameter but I did notice that file deletion (of large recordings) was a significant cause of pixelation of recordings and also causedpixelation of playback of existing recordings.

It happened whether using rm or trm commands so even if decryption doesn't affect other recordings the clean up of the original recording at end of recording might!

So I now allow the originals to go to the dustbin rather than being deleted immediately.

* For testing I recorded 2 HD channels simultaneously and watched another HD recording to simulate the maximum load the Humax is designed to cope with.
Whilst that was happening I copied another recording and deleted large (2hour HD) recordings to simulate the webif auto process running in the background,
watching the playback showed if the commands I was issuing affected playback, whilst watching the recordings later showed if the the recordings had been affected.

I didn't try to measure the effects of other commands because I assumed copy would be the most io intensive but crop is reading and writing both .ts and .nts files and while the .nts files are small it might cause more disk head movement switching between files.

 

[MymsMan]

just witnessed a crop also producing major pixellation on the recording. just as well it was not a recording I will keep.

Was that pixellation of another recording or pixellation of the output file at the crop points?

There are two very separate issues here

1. Does running webif processes like copy/crop/delete negatively impact other recordings/playback - yes in high load situations
2. Does nicepslice (crop) have bugs that can cause sound and picture problems in the output at join points - yes but that hasn't been the focus of this thread

 

[mihaid]

Was that pixellation of another recording or pixellation of the output file at the crop points?

There are two very separate issues here

1. Does running webif processes like copy/crop/delete negatively impact other recordings/playback - yes in high load situations
2. Does nicepslice (crop) have bugs that can cause sound and picture problems in the output at join points - yes but that hasn't been the focus of this thread

it was issue 1 I was dealing with. the pixellation happened on the program being recorded (I saw it when I played it back) whilst some other recording was being cropped.