[sweeper] Custom rules to manage recordings

I'm not sure exactly what's happening but I have a modified version of the dedup script that moves duplicates to a new folder rather than deleting them. Every couple of days I get a failed recording notification from RS and when I investigate, it appears there's a .hmt file left in the recordings folder and what looks to be the corresponding .ts file (i.e. same filename) in the duplicates folder. So it looks like sweeper is moving the file before the recording is completely finished/tidied up. Is that possible and if so what can I do to fix it?

sweeper rules attached:
 

Attachments

  • sweeper_dedup.txt
    841 bytes · Views: 8
Are these real duplicates?
Usually a stale .hmt file will be created when something is moved while being chase played or something similar but sweeper should leave any in-use files completely alone. This could be a bug in sweeper, it would be useful to see the auto.log file with the log level set to debugging.
 
Yes definitely a duplicate. I've just reset the auto.log and put the debugging mode on so I'll wait until it happens again. In the meantime, I'm not sure if these bits of info help:

1. screen grab of info of the .hmt file (clarifies date & time)
sweeper_hmt.PNG
2. screen grab of info of the .ts file in the _duplicates folder (note I have sweeper running on this folder too so as to update the filename to the episode name)
sweeper_ts.PNG
3. auto.log snippet that shows files being moved and renamed around 24/02/17 to 25/02/17
 

Attachments

  • auto.log.txt
    10.7 KB · Views: 4
Hi af123,
The problem has finally occurred again, there were 3 auto.log files created. The corrupt file was called /Family Guy/Family Guy_20170312_2305.hmt and I believe it matches an episode called "Friends of Peter G". I've attached the log files, let me know if you need any other info. Thanks.
 

Attachments

  • auto.log.zip
    164.6 KB · Views: 2
I've spotted a potential improvement in nomenclature:

In the flagged as options, there are "Encrypted" and "Encrypted On Disk". Those of us who have been around from the start know what these mean, but it is rather obtuse. How about substituting "Protected (Enc)" and "Decrypted (Dec)" respectively? (Decrypted would be the inverse of "Encrypted On Disk", and require negation of the trigger condition.)
 
Black Hole said:
I've spotted a potential improvement in nomenclature:

In the flagged as options, there are "Encrypted" and "Encrypted On Disk". Those of us who have been around from the start know what these mean, but it is rather obtuse. How about substituting "Protected (Enc)" and "Decrypted (Dec)" respectively? (Decrypted would be the inverse of "Encrypted On Disk", and require negation of the trigger condition.)
Click to expand...
Like the idea, but was this posted on the wrong thread?

Sent from my Nexus 7 using Tapatalk
 
I suspect that BH was referring to a post I made in another thread regarding sweeper detecting whether a file had been decrypted before it was copied to a NAS.
 
Black Hole said:
I've spotted a potential improvement in nomenclature:

In the flagged as options, there are "Encrypted" and "Encrypted On Disk". Those of us who have been around from the start know what these mean, but it is rather obtuse. How about substituting "Protected (Enc)" and "Decrypted (Dec)" respectively? (Decrypted would be the inverse of "Encrypted On Disk", and require negation of the trigger condition.)
Click to expand...
The automatic negation isn't supported by the framework but I agree that at least Encrypted should be changed. No update to configuration files needed, this is just the GUI presentation of the rules.
 
I assume the existing standard is just the output of the hmt utility, so then there needs to be a decision made whether to bring hmt into line and what knock-on effects that would have.

I don't understand why it isn't possible to invert the "Encrypted on Disk" input from hmt before it goes on to the rest of the sweeper processing, but then I'm not a code guru. If it was inverted so that the logic condition became "Decrypted (Dec)", then it does become necessary to update the user configuration files to insert or remove the "not" in the flag test.

If it is not possible to have "Decrypted (Dec)", how about "Protected (Enc)" and "Encrypted (/Dec)"? Not as nice.
 
Black Hole said:
I don't understand why it isn't possible to invert the "Encrypted on Disk" input from hmt before it goes on to the rest of the sweeper processing, but then I'm not a code guru.
Click to expand...
All of the conditions and actions in the GUI are generated from a schema file which you can look at as http://<humax>/plugin/sweeper/schema.js
The 'flag' condition in there is defined (with 2.1.5-6):
Code: 
   flag: {
       'class': 'all',
       type: 'select',
       desc: 'Recording Flagged as',
       idesc: 'Recording not Flagged as',
       negate: true,
       select: {
           Locked: 'Locked',
           New: 'New',
           Encrypted: 'Protected (Enc)',
           Guidance: 'Guidance',
           ODEncrypted: 'Encrypted',
           Shrunk: 'Shrunk',
           Deduped: 'De-duplicated',
           Radio: 'Radio Programme'
       },
       def: 'New'
   },

Left-hand-side of the select list is the flag as understood/reported by the hmt utility and the right-hand-side is what is displayed.
The GUI builder doesn't have a concept of a negated flag at present.
 
I've noticed rsync backing up a rogue .encrypted file. As it doesn't appear in the WebIf browser I've tried a Sweeper rule to delete it but this fails. I assume this is because it's a file extension and Sweeper only examines the filename. Is it possible to have a Sweeper condition 'extension contains'?
 
What is creating the .encrypted files and are they associated with regular recordings?

You can already use the filexists condition to check whether extensions exist.

I use fileexists %basename.thm to check whether a thumbnail file already exists for a recording
 
The .encrypted file is part of the workflow during on-the-box decryption, and if it remains once the decryption is complete it is because something interrupted the process (it's the original .ts file renamed, which is then "copied" by wget to make a new .ts).
 
Black Hole said:
The .encrypted file is part of the workflow during on-the-box decryption, and if it remains once the decryption is complete it is because something interrupted the process (it's the original .ts file renamed, which is then "copied" by wget to make a new .ts).
Click to expand...
Not that I'm aware. Automatic background processing doesn't put extra files into the video tree and the one-off decryption that you can do through the web interface media browser creates a ".decrypting" file which could get left behind if power was lost or something else went wrong.
 
That's curious, because I've seen one or two .something files left behind... maybe by ad detection chase decryption?
 
This was from a BBC2 recording so was not being decrypted by DetectAds. May have started to play recording while standard decryption was in progress.

MymsMan said:
You can already use the filexists condition to check whether extensions exist.
Click to expand...
Tried this but deletes every recording except the one I want to delete! Good job I used Sweeper test mode.
Code: 
27/06/2017 11:42 - + Sweeper processing /media/My Video/Winterwatch 1963 - The Big Freeze_20130228_2323.ts
27/06/2017 11:42 - Processing [age {> 0} fileexists *.encrypted action delete]
27/06/2017 11:42 -    age(> 0)
27/06/2017 11:42 -  ... Recording age: 37906
27/06/2017 11:42 -      MATCH
27/06/2017 11:42 -    fileexists(*.encrypted)
27/06/2017 11:42 -   FILEEXISTS(/media/My Video/*.encrypted)
27/06/2017 11:42 -      Matches(Versailles_20170623_2101.encrypted)
27/06/2017 11:42 -      MATCH
27/06/2017 11:42 -    action(delete)
27/06/2017 11:42 - ACTION: delete() [0]
 
You must log in or register to reply here.
Back
Top