• The forum software that supports hummy.tv has been upgraded to XenForo 2.1!

    This is upgrade brings a number of improvements including the ability to bookmark posts to come back to later. Please bear with us as we continue to tweak things and open a new thread for any questions, issues or suggestions in Site/Forum Issues.

[sweeper] Custom rules to manage recordings

Mark Bowler

New Member
I'm not sure exactly what's happening but I have a modified version of the dedup script that moves duplicates to a new folder rather than deleting them. Every couple of days I get a failed recording notification from RS and when I investigate, it appears there's a .hmt file left in the recordings folder and what looks to be the corresponding .ts file (i.e. same filename) in the duplicates folder. So it looks like sweeper is moving the file before the recording is completely finished/tidied up. Is that possible and if so what can I do to fix it?

sweeper rules attached:
 

Attachments

OP
af123

af123

Administrator
Staff member
Are these real duplicates?
Usually a stale .hmt file will be created when something is moved while being chase played or something similar but sweeper should leave any in-use files completely alone. This could be a bug in sweeper, it would be useful to see the auto.log file with the log level set to debugging.
 

Mark Bowler

New Member
Yes definitely a duplicate. I've just reset the auto.log and put the debugging mode on so I'll wait until it happens again. In the meantime, I'm not sure if these bits of info help:

1. screen grab of info of the .hmt file (clarifies date & time)
sweeper_hmt.PNG
2. screen grab of info of the .ts file in the _duplicates folder (note I have sweeper running on this folder too so as to update the filename to the episode name)
sweeper_ts.PNG
3. auto.log snippet that shows files being moved and renamed around 24/02/17 to 25/02/17
 

Attachments

Mark Bowler

New Member
Hi af123,
The problem has finally occurred again, there were 3 auto.log files created. The corrupt file was called /Family Guy/Family Guy_20170312_2305.hmt and I believe it matches an episode called "Friends of Peter G". I've attached the log files, let me know if you need any other info. Thanks.
 

Attachments

Black Hole

May contain traces of nut
I've spotted a potential improvement in nomenclature:

In the flagged as options, there are "Encrypted" and "Encrypted On Disk". Those of us who have been around from the start know what these mean, but it is rather obtuse. How about substituting "Protected (Enc)" and "Decrypted (Dec)" respectively? (Decrypted would be the inverse of "Encrypted On Disk", and require negation of the trigger condition.)
 
I've spotted a potential improvement in nomenclature:

In the flagged as options, there are "Encrypted" and "Encrypted On Disk". Those of us who have been around from the start know what these mean, but it is rather obtuse. How about substituting "Protected (Enc)" and "Decrypted (Dec)" respectively? (Decrypted would be the inverse of "Encrypted On Disk", and require negation of the trigger condition.)
Like the idea, but was this posted on the wrong thread?

Sent from my Nexus 7 using Tapatalk
 

Wallace

Traveler 34122
I suspect that BH was referring to a post I made in another thread regarding sweeper detecting whether a file had been decrypted before it was copied to a NAS.
 
OP
af123

af123

Administrator
Staff member
I've spotted a potential improvement in nomenclature:

In the flagged as options, there are "Encrypted" and "Encrypted On Disk". Those of us who have been around from the start know what these mean, but it is rather obtuse. How about substituting "Protected (Enc)" and "Decrypted (Dec)" respectively? (Decrypted would be the inverse of "Encrypted On Disk", and require negation of the trigger condition.)
The automatic negation isn't supported by the framework but I agree that at least Encrypted should be changed. No update to configuration files needed, this is just the GUI presentation of the rules.
 

Black Hole

May contain traces of nut
I assume the existing standard is just the output of the hmt utility, so then there needs to be a decision made whether to bring hmt into line and what knock-on effects that would have.

I don't understand why it isn't possible to invert the "Encrypted on Disk" input from hmt before it goes on to the rest of the sweeper processing, but then I'm not a code guru. If it was inverted so that the logic condition became "Decrypted (Dec)", then it does become necessary to update the user configuration files to insert or remove the "not" in the flag test.

If it is not possible to have "Decrypted (Dec)", how about "Protected (Enc)" and "Encrypted (/Dec)"? Not as nice.
 
OP
af123

af123

Administrator
Staff member
I don't understand why it isn't possible to invert the "Encrypted on Disk" input from hmt before it goes on to the rest of the sweeper processing, but then I'm not a code guru.
All of the conditions and actions in the GUI are generated from a schema file which you can look at as http://<humax>/plugin/sweeper/schema.js
The 'flag' condition in there is defined (with 2.1.5-6):
Code:
   flag: {
       'class': 'all',
       type: 'select',
       desc: 'Recording Flagged as',
       idesc: 'Recording not Flagged as',
       negate: true,
       select: {
           Locked: 'Locked',
           New: 'New',
           Encrypted: 'Protected (Enc)',
           Guidance: 'Guidance',
           ODEncrypted: 'Encrypted',
           Shrunk: 'Shrunk',
           Deduped: 'De-duplicated',
           Radio: 'Radio Programme'
       },
       def: 'New'
   },
Left-hand-side of the select list is the flag as understood/reported by the hmt utility and the right-hand-side is what is displayed.
The GUI builder doesn't have a concept of a negated flag at present.
 

marsha11

Member
I've noticed rsync backing up a rogue .encrypted file. As it doesn't appear in the WebIf browser I've tried a Sweeper rule to delete it but this fails. I assume this is because it's a file extension and Sweeper only examines the filename. Is it possible to have a Sweeper condition 'extension contains'?
 

MymsMan

Ad detector
What is creating the .encrypted files and are they associated with regular recordings?

You can already use the filexists condition to check whether extensions exist.

I use fileexists %basename.thm to check whether a thumbnail file already exists for a recording
 

Black Hole

May contain traces of nut
The .encrypted file is part of the workflow during on-the-box decryption, and if it remains once the decryption is complete it is because something interrupted the process (it's the original .ts file renamed, which is then "copied" by wget to make a new .ts).
 
OP
af123

af123

Administrator
Staff member
The .encrypted file is part of the workflow during on-the-box decryption, and if it remains once the decryption is complete it is because something interrupted the process (it's the original .ts file renamed, which is then "copied" by wget to make a new .ts).
Not that I'm aware. Automatic background processing doesn't put extra files into the video tree and the one-off decryption that you can do through the web interface media browser creates a ".decrypting" file which could get left behind if power was lost or something else went wrong.
 

Black Hole

May contain traces of nut
That's curious, because I've seen one or two .something files left behind... maybe by ad detection chase decryption?
 

marsha11

Member
This was from a BBC2 recording so was not being decrypted by DetectAds. May have started to play recording while standard decryption was in progress.

You can already use the filexists condition to check whether extensions exist.
Tried this but deletes every recording except the one I want to delete! Good job I used Sweeper test mode.
Code:
27/06/2017 11:42 - + Sweeper processing /media/My Video/Winterwatch 1963 - The Big Freeze_20130228_2323.ts
27/06/2017 11:42 - Processing [age {> 0} fileexists *.encrypted action delete]
27/06/2017 11:42 -    age(> 0)
27/06/2017 11:42 -  ... Recording age: 37906
27/06/2017 11:42 -      MATCH
27/06/2017 11:42 -    fileexists(*.encrypted)
27/06/2017 11:42 -   FILEEXISTS(/media/My Video/*.encrypted)
27/06/2017 11:42 -      Matches(Versailles_20170623_2101.encrypted)
27/06/2017 11:42 -      MATCH
27/06/2017 11:42 -    action(delete)
27/06/2017 11:42 - ACTION: delete() [0]
 
Top