Early Morning Breakup

We have proc {system busy} for testing whether a recording is in progress (or similar).

More generally we could revisit /proc/loadavg as in my earlier post (I think we found that ionice didn't help much, rather like trm vs rm).

Expensive deletion is a feature of Ext3, as discussed some time ago. I suspect that the real problem is swapping all the index blocks in and out of the limited RAM.

As the Humax settop program knows about Ext3, we still await the Ext4-based CF upgrade (imagine: we could have the "2 petabyte disk installation blog", etc).
 
Last edited:
It would be better if the empty_dustin process could (optionally?) be made to be recording aware and defer its operation until a time when it is not recording.
The other thing that strikes me about anacrontab is that things really ought to be spaced out a bit better. Here's one of mine (sorted):
Code:
1 3 tunefix-autoup /mod/sbin/tunefix-autoup
1 5 thumbnails /mod/webif/plugin/thumbnails/catchup
1 5 auto-update /mod/sbin/auto-update
1 5 diskattrs /mod/webif/lib/bin/diskattrs
1 6 empty_dustbin /mod/sbin/empty_dustbin -l
1 8 sched_backup /mod/webif/html/sched/backup/backup.jim
Three things kicking off at :05 doesn't seem that sensible. Thumbnails is quite CPU intensive too.
 
We have proc {system busy} for testing whether a recording is in progress (or similar).

More generally we could revisit /proc/loadavg as in my earlier post (I think we found that ionice didn't help, rather like trm vs rm).

Expensive deletion is a feature of Ext3, as discussed some time ago. I suspect that the real problem is swapping all the index blocks in and out of the limited RAM.

As the Humax settop program knows about Ext3, we still await the Ext4-based CF upgrade (imagine: we could have the "2 petabyte disk installation blog", etc).
Thanks for the clarification, I didn't find it earlier.
Not sure if this may be useful in checking recording status nugget recordings
Also, as a possible alternative to using rm maybe something like cat /dev/null > <filename>; rm <filename>

Edit: it looks like I was wrong as others have tried similar https://serverfault.com/questions/128012/how-to-make-rm-faster-on-ext3-linux and https://www.depesz.com/2010/04/04/how-to-remove-backups/
 
Last edited:
We have a custom queue processing system. Perhaps we should use it? Obviously some assemblyJim required.
 
Deletion can also affect live TV viewing so any change should avoid scheduling during hours when it is likely that viewing is happening
 
Has anyone ever seen a disturbance during SUI deletion? I don't think I have. Presuming not, I wonder what settop does to prevent it?

I realise undelete means simple SUI deletions aren't actually deleting anything, but sometimes I delete everything in [Recycle Bin], which is not protected by undelete.
 
Standard User Interface

I just did a stress test, recording from BBC4HD while deleting 21GB from [Recycle Bin] (taking about 7 minutes). There were no disturbances live that I noticed, but there certainly were in the recording.
 
Thanks for the clarification, I didn't find it earlier.
Not sure if this may be useful in checking recording status nugget recordings
It would be. It pulls the list of active recordings out of the Humax settop program's memory in eldritch fashion. The {system busy} class method wraps a call to the lsof Linux command (eldritch in itself) to list recording files that the Humax settop program has open.

I believe there is also some code that checks for imminent scheduled recordings (the queue system?) which is just a database query. The interface that would allow the system to detect a user's future intention to press the Record button is missing.
Yes, as expected truncating the file to 0 is equivalent from the filesystem perspective to rm <filename>. trm is intended to limit resource usage by repeatedly truncating a small portion from the end of a large file until the file has length 0. Possibly it could do that by opening the file, unlink()ing it (so it appears to be deleted from its directory, and can't easily be accessed by another process), repeatedly ftruncate()ing the file until the file has 0 length, then closing the file, causing it to be finally removed from the filesystem; this seems to match its external linkages.

The second reference is useful. It confirms what @MymsMan found with ionice (ie, no visible effect, albeit a somewhat different test case). Commenters say, plausibly, that the problem is with Ext3 listing every block in use and then unwinding that list (via index blocks, etc). Also, the author had a system described as "very beefy" for 2010, yet had trouble deleting files only a tenth of the size of a typical recording.ts, whereas the BCM7405 with 128MB RAM is very much at the non-beefy end of the scale.
...I just did a stress test, recording from BBC4HD while deleting 21GB from [Recycle Bin] (taking about 7 minutes). There were no disturbances live that I noticed, but there certainly were in the recording.
That sounds like a good argument for always using the Recycle Bin and ensuring that emptying is only done when recordings are neither due nor scheduled.
 
Just me thinking out loud. As ext3 is ext2 with journalling, could we not just use ext2 for the My Video directory? This will sacrifice robustness for possible speed improvement. Eg just remount directory as ext2 or convert the whole partition (which may cause issues for the custom fw). It's a bit of a backward step of, but I'm not sure we need the saved TV programmes to be on ext3.
I suppose with ext3, you don't need to do anything much on reboot when the box crashes.
 
Last edited:
I don't think Ext2 would be better. The problem is the individual block accounting, and I guess that's the same for both 2 and 3. You can overcome it with extents, but that makes your filesystem Ext4. The Humax settop program contains filesystem code that knows about Ext3 and would complain about extents. Problems occurred when the repo version of e2fsprogs was upgraded beyond 1.42 due to unwanted setting of the extent flag.
 
Unless I misunderstand the problem (emptying the 'deleted' files if recording) then has the required code not already been implemented here?
Capture 26-02-2021 14_48_14.jpg
 
Yes, but emptying the bin doesn't come under the category of "automatic processing" for historical reasons... at the moment, anyway.
That's what /df was hinting at in #24.
 
Yes, but emptying the bin doesn't come under the category of "automatic processing" for historical reasons... at the moment, anyway.
That's not the point. The point of post 32 is that processing exclusion is already implemented for the auto-processing queue, so the code already exists to implement the same for garbage emptying.
 
Unless I misunderstand the problem (emptying the 'deleted' files if recording) then has the required code not already been implemented here?
View attachment 5367
I am quite happy to let current auto processing run whilst recordings are in processing but would not be happy to let undelete run since it definitely can cause problems so while the framework could be used as a model I think we would need a separate set of settings to control when empty-dustbin was allowed to run .

There are benefits in running auto-delete more than once a day especially when running low on space and reducing the volume being deleted at once would reduce the impact.
 
I am quite happy to let current auto processing run whilst recordings are in progress
Me too, I have no restrictions in place on auto. I think /df wants to integrate garbage clearance into auto, and there is no reason it couldn't have its own settings within auto, but it might as well be separate while using the same framework.
 
I am quite happy to let current auto processing run whilst recordings are in processing but would not be happy to let undelete run since it definitely can cause problems
The thing I don't like about the current auto processing is that it stops everything, whereas I only want to stop the heavy I/O stuff.
i.e. I'd want youtube-dl and thumbnails to run, but not decrypt or undelete
 
Back
Top