Webif crop feature stresses the processor

[/df]

Is there pixellation in a replayed recording that wasn't observed while watching the transmission live? Or is the same pixellation seen at both stages?

The latter would indicate overload within the BCM7405 chip and its onboard peripherals (eg RAM), if transmission quality has been ruled out.

The former case could also result from the "ATA streaming feature set", ie the additional drive commands that are supported by "AV drives" such as those supplied in PVRs, HDR Fox T2 in particular, and recommended for upgrading (HDR) or enabling (HD) the CF platforms.

For instance, the streaming feature set allows for read and write commands with a time limit such that the command returns even if the drive cannot complete the operation within the deadline, yielding partial results. Such events are supposed to be logged in a special SMART log.

It's a fair assumption that the Humax blob makes use of these functions for recording and playing media files, when the installed drive supports them, even if the normal higher-integrity operations are used for file manipulation, through the Broadcom multimedia framework on which it's built.

Conversely, programs built for the CF will use the normal higher-integrity operations, even if it might be appropriate to use the streaming ones (eg Mediatomb), unless they have been specially coded to use low-level disk APIs.

Suppose that a programme is being recorded using a time-limited streaming write while other activities are also using the disk. In any of the following scenarios, under the above assumptions, the streaming operation may fail softly so that its process can carry on trying to write the next set of data, resulting in a video glitch when the recording is played back:

• if more simultaneous disk operations are scheduled than the disk's firmware and memory can handle;
• if a disk error occurs for the streaming operation that can't be resolved (by remapping one or more sectors) within the deadline;
• if a disk error occurs for one of the other disk operations that causes the streaming operation to miss its deadline while the drive firmware is busy resolving the error.

 

[mihaid]

Is there pixellation in a replayed recording that wasn't observed while watching the transmission live?

yes, that is the case.

 

[/df]

So the symptoms indicate a disk issue but the disk diagnostics come up OK? Perhaps there's something else affecting the I/O path in your system, since this issue doesn't seem to affect everyone.

It might be interesting to check the read and write stream error logs. smartctl is supposed to be able to do this, but the disk I'm using here has all zeroes for these logs.

Eg:

# smartctl -l directory /dev/sdb
smartctl 6.4 2015-06-04 r4109 [7405b0-smp-linux-2.6.18-7.1] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
...
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
...
# smartctl -l gplog,0x21 /dev/sdb
smartctl 6.4 2015-06-04 r4109 [7405b0-smp-linux-2.6.18-7.1] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

General Purpose Log 0x21 [Write stream error log], Page 0-0 (of 1)
0000000: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
...
00001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

# smartctl -l gplog,0x22 /dev/sdb
smartctl 6.4 2015-06-04 r4109 [7405b0-smp-linux-2.6.18-7.1] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

General Purpose Log 0x22 [Read stream error log], Page 0-0 (of 1)
0000000: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
...
00001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

 

[hairy_mutley]

This sounds no different from the problem that I have had (and previously discussed) that stops me from using chase decrypt and ad-detect, i.e. pixelation being added to the live recording.

 

[Matthew]

I had a similar issue yesterday, but not with crop. I was recording two simultaneous HD programs and watching a 3rd, I made the mistake of emptying the bin (20GB) in the webif, that created disturbance in both recordings and the playback, so clearly the delete process is using too high a priority, I`m sure had I done delete with the box menu it would have been fine.

I have often cropped whilst recording in the past and not had this issue. I don't use chase decrypt as that just seems an un-neccessary stresser.

 

[mihaid]

It may be that I'm conflating things. I have started with cropping things but shortly after the files are cropped I transfer them via copy. It may well be that the stressing only occurs when the copy happens. It does seem that a little pattern of overstressing is beginning to emerge.

 

[mihaid]

BTW, the copy is done via the remote control gui

 

[Matthew]

BTW, the copy is done via the remote control gui

When you say copy I assume you mean move/copy (which is just a move unless your copying onto an external) that does not involve a file transfer, all it does is change flags which specify where the file lives.

Or are you doing something else?

 

[MymsMan]

BTW, the copy is done via the remote control gui

So to avoid the possibility of any impact on recordings do your copies and cropping when not recording anything.
You can also avoid Auto processing running whilst recording.
Possibly you could set up Sweeper rules to automate the copies if you are routinely copying series to another device.

 

[MymsMan]

I made the mistake of emptying the bin (20GB) in the webif, that created disturbance in both recordings and the playback, so clearly the delete process is using too high a priority,

I have also noticed problems when emptying the bin but I am not sure why it should be such a disruptive process - surely it should just be returning blocks too the free pool, we don't need a secure erase that overwrites the data blocks.

 

[mihaid]

r copying onto an external)

This

 

[mihaid]

So to avoid the possibility of any impact on recordings do your copies and cropping when not recording anything.

That's exactly what I have been doing since I realised what was happening

 

[prpr]

I have also noticed problems when emptying the bin but I am not sure why it should be such a disruptive process - surely it should just be returning blocks too the free pool

I think I remember af123 saying this was one of the deficiencies of ext3. This was supposed to have been mitigated somewhat by the use of the trm command rather than the rm command for deletion, but it still seems to cause problems.

 

[/df]

I think I remember af123 saying this was one of the deficiencies of ext3. This was supposed to have been mitigated somewhat by the use of the trm command rather than the rm command for deletion, but it still seems to cause problems.

We could try building ionice* in busybox and using it to limit the impact of emptying the bin.

Also, background tasks could consult /proc/loadavg before starting something intensive -- here's a good article on the background to this.

*Actually I see I did build it as part of busybox-1.29.3 but put that project aside because the on-box build created a busybox/awk that was unable to print a number, which I put down to the problem of the conflicting FP settings between uClibc and gcc, and set off to http://buildroot.org/.

 

[MymsMan]

We could try building ionice* in busybox and using it to limit the impact of emptying the bin.

This sounds very interesting and not just for the delete case, If we could assign the webif processes that read/rewrite entire ts files (eg decrypt, detectads, crop, copy, ...) to a lower priority we might be able to significantly reduce the pixelation problems that prompted this thread.

 

[/df]

The Linux 2.6.18 IO subsystem supports several scheduling algorithms. By default it uses anticipatory, but you can change that to the Completely Fair Queuing algorithm to which ionice applies.

# df /mod
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1            1941367728 1087036452 854331276  56% /media/drive1
# cat /sys/block/sda/queue/scheduler
noop [anticipatory] deadline cfq 
# printf cfq>/sys/block/sda/queue/scheduler
# cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq] 
#

From man ionice, we see

... before kernel 2.6.26 a process that has not asked for an io priority formally uses "none" as scheduling class, but the io scheduler will treat such processes as if it were in the best effort class. The priority within the best effort class will be dynamically derived from the cpu nice level of the process: io_priority = (cpu_nice + 20) / 5.

Looking at some processes relevant to a CF configuration,

# ps -Al
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 S     0     1     0  0  81   0 -   313 wait   ?        00:00:02 init
...
4 S     0   367   291 28  85   0 - 21171 hrtime ?        16:50:36 humaxtv
...ps -Al
1 S     0  1484     1  0  86  10 -   342 hrtime ?        00:00:25 crond
1 S     0  1531     1  0  95  10 -   874 wait   ?        00:00:00 lighttpd
1 S     0  1533  1531  0  85  10 -   985 epoll_ ?        00:01:38 lighttpd
1 S     0  1534  1531  0  85  10 -   987 epoll_ ?        00:03:40 lighttpd
...
1 S     0  1680     1  0  95  10 -   311 wait   ?        00:00:00 S54recmon
0 S     0  1683  1680  0  86  10 -   317 select ?        00:00:01 recmon
1 S     0  1714     1  0  95  10 -   311 wait   ?        00:00:00 S60parseepg
0 R     0  1717  1714  3  87  10 -  2830 -      ?        02:21:46 epg
0 S     0  1724     1  0  94  10 -   315 wait   ?        00:08:39 editmonitor
...
0 S     0 22547 22545  4  86  10 -   938 hrtime ?        00:00:00 tvdiary_status
# ionice -p 367
none: prio 0
# ionice -p 1484
none: prio 0
#

we see that by default under CFQ the humax blob runs with IO priority 4 while most others including CF processes run with priority 6. But things can change, using the attached stand-alone ionice modified from the Busybox source and built on a Humax HDR.

# ionice -h

ionice [-c 1-3] [-n 0-7] [[-p PID] | [PROG ...args]]
Change I/O priority and class

-c Class. 1:realtime 2:best-effort 3:idle
-n Priority
# ionice -p 1484 -c3
# ionice -p 1484
idle
#

Whether changing the IO scheduling parameters has any effect on perceived performance, such as recording pixellation issues, is a much trickier question that would take a deal of testing to answer.

 

[MymsMan]

It isn't clear from the ionice documentation but do child processes inherit the ionice class of their parent or is it necessary to apply it each pid?
e.g. detectads runs chaseget, ffmpeg, silence, nisesplice in a pipeline would setting ionice for detectads apply to the other processes.

Whether changing the IO scheduling parameters has any effect on perceived performance, such as recordiing pixellation issues, is a much trickier question that would take a deal of testing to answer.

My thoughts for initial testing would be to create multiple copies of a large file then whilst recording a HD programme delete the large files using rm and trm with no ionice, ionice -c2 -n7, and ionice -c3 timing the executions and then playing the recording to see if any pixellation is evident at the points of execution.

Part of the challenge will be to find a window when I am not actually recording something I want to watch later.

 

[/df]

It isn't clear from the ionice documentation but do child processes inherit the ionice class of their parent or is it necessary to apply it each pid?
e.g. detectads runs chaseget, ffmpeg, silence, nisesplice in a pipeline would setting ionice for detectads apply to the other processes.
...

Apparently they do. I set the crond to IO Class Idle and the processes started by it (eg tvdiary_status) also had Idle.

Meanwhile I also built the util-linux version of ionice, which supports setting the IO scheduling parameters of an entire process group and is generally more sophisticated.

# ionice --version
ionice (adapted from util-linux 2.33.1)
# ionice --help

Usage:
 ionice [options] -p <pid>...
 ionice [options] -P <pgid>...
 ionice [options] -u <uid>...
 ionice [options] <command>

Show or change the I/O-scheduling class and priority of a process.

Options:
 -c, --class <class>    name or number of scheduling class,
                          0: none, 1: realtime, 2: best-effort, 3: idle
 -n, --classdata <num>  priority (0..7) in the specified scheduling class,
                          only for the realtime and best-effort classes
 -p, --pid <pid>...     act on these already running processes
 -P, --pgid <pgrp>...   act on already running processes in these groups
 -t, --ignore           ignore failures
 -u, --uid <uid>...     act on already running processes owned by these users

...
My thoughts for initial testing would be to create multiple copies of a large file then whilst recording a HD programme delete the large files using rm and trm with no ionice, ionice -c2 -n7, and ionice -c3 timing the executions and then playing the recording to see if any pixellation is evident at the points of execution.

Part of the challenge will be to find a window when I am not actually recording something I want to watch later.

Understandable.

What about increasing the IO priority of the humaxtv process eg ionice -c1, ionice -c2 -n0? This could be a simpler fix, potentially just a modified /etc/init.d/S90settop that could be installed with a bind mount by a new script in /mod/boot/xinit.d, without changing any of the other CF packages that may also have been linked to pixellation issues.

 

[Black Hole]

Is the box running a real-time Linux? If not (and it almost certainly isn't), there is no control over when a current process will suspend to allow even a higher priority process to get in (Real-time OSes are specifically written to guarantee service response times for processes injecting a high-priority interrupt). It strikes me this may be the problem rather than priorities themselves - the Humax process doesn't expect to be competing for resources, so they had no need to use an expensive RT OS.

However, no harm in trying it. Bumping up the Humax process priority is a great idea (instead of fiddling with all the CF processes).

 

[af123]

potentially just a modified /etc/init.d/S90settop that could be installed with a bind mount by a new script in /mod/boot/xinit.d

It's easier than that, put a script in /mod/boot/bootstrap.d/ and it will be called with the process ID of the humaxtv process just after it is started.