[sweeper] Custom rules to manage recordings

[prpr]

This seems to work but as the decrypt and auto-audio are processed in the background queue and Sweeper seems to be an aynchronous background task, I'm worried that Sweeper will try to delete the original recording before the mp3 file creation is completed.

I was speculating whether adding a "Queue for deletion" option to the Queue processing would be a good idea.
You can already do "Queue for Decrypt" and "Queue for MP3", so you could serialise an asynchronous process with 3 rules (you don't seem to be able to add more than 1 action per rule) - Decrypt, MP3, Delete - in a guaranteed order.

I've added the Age condition which I assume relates to the original .ts recording age.

Yes.

Does anyone know if Sweeper checks whether the original file is in use before deleting or is this a lower level operating system check?

It appears to, yes. The Linux OS doesn't care if one process tries to delete a file that is already in use - it remains accessible to processes that already have the file open, but not to subsequent ones, and the file is then actually deleted when the usage count drops to 0.
What you really need to worry about is it being deleted before the conversion has started. 2 hours seem a bit on the mean side. I would make it a day.

 

[Luke]

What you really need to worry about is it being deleted before the conversion has started. 2 hours seem a bit on the mean side. I would make it a day.

Doesn’t the condition "File exists %basename.mp3" being true mean that the conversion has at least started?

 

[prpr]

Doesn’t the condition "File exists %basename.mp3" being true mean that the conversion has at least started?

I guess so!

 

[MymsMan]

Doesn’t the condition "File exists %basename.mp3" being true mean that the conversion has at least started?

But not necesarily completed, while the OS system will protect against deletion of file in use it won't protect against shutdowns whilst in prcocess. The queue system will restart incomplete operations on restart - providing the input file still exists!

You can check in sweeper whether a recording is queued for processing so you could defer deletion until it is no longer queued.

I was speculating whether adding a "Queue for deletion" option to the Queue processing would be a good idea.
You can already do "Queue for Decrypt" and "Queue for MP3", so you could serialise an asynchronous process with 3 rules (you don't seem to be able to add more than 1 action per rule) - Decrypt, MP3, Delete - in a guaranteed order.

One of my many started, but never completed, ideas was to create a package of additional queue actions including Queue for Copy, and Queue for Move, and Queue for Command
Archiving to an external storage medium can be slow but the current Sweeper actions run sychronously, and Queue for command would allow execution of an arbitrary command.
This would require Browse to have a text entry box for queue options to specify the destination/command options. Ideally you could use %values as in sweeper rules

 

[prpr]

OK, a bit more thought and I've come up with this:

humax /mnt/hd2/My Video/Tom # cat .sweeper
# Decrypt
action {queue decrypt}
# Audio extract and delete
action {queue {mp3 -del}}

Seems to work well. The original recording files end up in the bin (if the undelete package is installed) and the .mp3 is left in the folder.
It will be in the next WebIf release, so don't try it yet as it won't work.

 

[Black Hole]

Great! I have an immediate use for that. I've been copying .mp3 to USB (for car listening) and then deleting both manually for years.

 

[prpr]

Webif 1.5.4 available in the Beta repository for testing.

 

[phit03]

Wow! thanks for all the feedback. I'll try Webif 1.5.4 and try adding the "queue recording for" with the "decrypt" and "mp3 -del" options rules and remove the decrypt and auto-audio flags from the directory so the commands are added to the background queue and thus guaranteed to run synchronously. I hadn't realised that there was a -del option when creating mp3 files.

 

[prpr]

I hadn't realised that there was a -del option when creating mp3 files.

There wasn't. I've just created it in response to your need (and also BH's it transpires).

There are also existing "-mp3" and "-mp2" options to override whatever your default is. You can use either of those alone or in combination with "-del".

 

[phit03]

Thanks prpr, much appreciated.

 

[phit03]

Installed the beta package and set up the new rules (a copy of your example) and it works perfectly. All actions now done by Sweeper.
Once again, many thanks for your help prpr.

 

[phit03]

A couple of days in a few problems observed, possibly due to the way that sweeper is adding stuff to the auto processing Q, possibly duplicating entries. Hopefully, I've worked around them by adding extra tests in the rules to Sweeper so it doesn't try to add the programs to the Q a second time.
The other thing that I can't get my head around is what needs to be done after a rule has been updated so the changes get picked up. What happens when you update a rule? does sweeper stop processing that rule until you do something to start it again? It seems to start processing again if you disable/enable the rule but perhaps I'm being impatient and not waiting long enough. If I run "Test Config" or "Run Now" they both seem to sit there doing nothing until I disable/enable the rules.
Are rules associated with the root (Media/My Video) and lower level folders processed separately or all together?

Latest rule versions below.

 

[prpr]

A couple of days in a few problems observed

What problems?

possibly due to the way that sweeper is adding stuff to the auto processing Q, possibly duplicating entries.

"Possibly"? Evidence? Queue log screenshot?

Hopefully, I've worked around them by adding extra tests in the rules to Sweeper so it doesn't try to add the programs to the Q a second time.

Unnecessary. The decrypt script already checks whether it's encrypted or not and whether it's already in the queue or not.
Likewise, the mp3 script already checks whether the .mp3 exists, whether the .ts is decrypted or not, and whether it's already in the extraction queue or not.
This will show up if you turn the logging level up to "debug", but otherwise is silent.
I'm not sure why you still have the 2 hour rule.
Did the simple rules I posted not work? Why not?

The other thing that I can't get my head around is what needs to be done after a rule has been updated so the changes get picked up.

You just have to click the "Save changes" button and say Yes to the prompt. Nothing else.

What happens when you update a rule? does sweeper stop processing that rule until you do something to start it again?

It runs the new rule(s) at the next scan interval (default setting is 5 min.). The queue is processed at the top of the next minute if there's something to do and nothing is already in progress.

It seems to start processing again if you disable/enable the rule but perhaps I'm being impatient and not waiting long enough. If I run "Test Config" or "Run Now" they both seem to sit there doing nothing until I disable/enable the rules.

"Test Config" on my box with your rules produces this, as expected:

and similarly for Run, although I don't currently have a recording in that folder.
Settings may prevent processing running if the box is recording or about to etc.

Are rules associated with the root (Media/My Video) and lower level folders processed separately or all together?

Not sure what you mean. They all run sequentially kicked off at the same time.

 

[phit03]

Sorry for the wooly explanation, I was overwhelmed by the number of errors in the processing queue, decrypt tasks were coming up with already decrypted, which made me think that Sweeper was adding them to the queue more than once and the other was "TS file not found" for the audio extract tasks when in fact the TS file was there in the swept directory. Adding the extra rules seemed to fix the problems making me think that Sweeper wasn't checking whether they were already there.
The original simple rules worked fine when there were only a few recordings to process but because the rules weren't being run, when I eventually got them to run there were a large number of programs to process.

I'll revert to the original ruleset and setup another test with 20 radio programs to be processed and capture the screen shots and the entries in the auto log.
I'll make sure that I understand what's happening before I submit any results from the test.

These are my auto processing settings, the only anomaly I can see is that I have 30 minutes in the "...how many minutes is soon" but both the "suspend automatic processing" fields are set to NO.
Do these settings have any effect on Sweeper or only on the jobs it puts into the auto-processing Q.

 

[prpr]

the only anomaly I can see is that I have 30 minutes in the "...how many minutes is soon" but both the "suspend automatic processing" fields are set to NO.

Thus making it irrelevant, so your auto-processing will start every 10 minutes. Some things run synchronously as part of this "scan" phase (which basically looks for things to do), and others get put on the (priority) queue for later.

Do these settings have any effect on Sweeper or only on the jobs it puts into the auto-processing Q.

Sweeper is essentially a subset of things that forms part of the Webif's overall automatic processing, so they apply to everything, both "scan" and "queue processing", except the latter kicks off every minute, as explained previously.