Old sweeper package to file away one-off recordings

Status
Not open for further replies.
Just wondering if there are any plans to extend sweeper so it could be used for archiving purposes to external HDD's / network shares? It would seem to have all the functionality required to do this except the ability to move/copy the recordings to an external location. I currently archive to a NAS using automount and have to manually select the recordings I would like to copy and then reselect them to delete after the copy is complete. The trouble is you have no idea how long the copy will take and its not interruptable. I try to archive when the box is not recording as copying files plus making multiple recordings, auto decrypting etc, really does hammer the STB. By being hooked into sweeper it would then play nicely by not running at the same time as auto decrypt etc...
 
I'm in the process of making all my recordings from both my Hummys available on my Windows server. I use Sweeper to move all recordings more than 6 hours old into a My TV folder. Every morning these are then moved (using Robocopy) onto the Server so that all recordings can be viewed on either box via Automount shares. I am currently investigating the use of a ZyXel NAS instead ( the Server is only a bog-standard Revo mini PC that also does TV recording, weather station and CCTV! so is a bit busy...)
 
Andy Edgeworth said:
... Every morning these are then moved (using Robocopy) onto the Server ...
Click to expand...

Do you know if pulling files off via Robocopy is lower resource wise on the Hummy than pushing them out via a copy using the remote? My main issue is not knowing how long the copy/move will take and then running into recordings, auto decrypt operations etc, which can really slow the box down. This would be the main benefit of integrating the process within Sweeper, which I believe is hooked into the same WebIf background routines that auto decrypt uses.
 
Robocopy is really nasty to the Hummy! I use Sweeper to wait until my recordings are over 6 hours old, so hopefully I'm not using the Hummys then. Then I start Robocopy off at 8 am to avoid any problems...
 
I'm trying to move all films that have been shrunk and decryted to a directory called films

I thought

action {movecreate Films(Test-%Film-%ODEncrypted-%Shrunk)}

would do the trick, but that just moved everything not in a directory to Films(Test-%Film-%ODEncrypted-%Shrunk)

Can anyone tell me what I'm doing wrong please
 
You need something like these two lines.

Code: 
# If it's a film which has not yet been decrypted, stop and don't process any more rules.
genre Film flag ODEncrypted action preserve
# If it got this far then we know it has been decrypted so just check if it has been shrunk
genre Film flag Shrunk action {move Films}

Assuming you are identifying films by the genre that the broadcaster assigned in the EPG. You could also trigger on recording length with something like:

Code: 
flag ODEncrypted action preserve
genre Film flag Shrunk action {move Films}
duration {>= 120} flag Shrunk action {move Films}
 
Thanks very much af123.

My effort was not even close, completely got the wrong end of the stick.

Thanks again

EDIT

Just tried the first example and it's worked.

What a splendid chap you are.
 
Sweeper has a problem with shows with accented characters in the name. I am in Finland and the name of the 7pm news is in finnish (Seitsemän Uutiset).

For all other shows sweeper moved the programme, ok, but for the 7pm news the length in the on screen browser showed zero mins. Looking via telnet the ts file was there but hmt file was wrong. Also in the original (/media/My Videos) directory another hmt file was left over! This leftover hmt file didn’t happen with other shows. I couldn’t play the recordings since the box gave an error about zero length programme.

Eventually I noticed from the logs that sweeper was moving the 7pm news at 7pm! Normally the move happens after the show has finished (eg next time the box is switched on or another timer event starts).

Going through the code I found the problem:

Sweeper doesn't move shows which are in use. It uses the ts inuse function which uses system inuse function.

System.class inuse uses lsof to check if a file is in use, but the format of filenames with UTF-8 characters in the name is different to what sweeper uses. This was the problem: sweeper was trying to rename the show while it was being recorded. Without accents the in use checking works ok.

Below is the example lsof output – you can see the “a with dots” is output at \xc3\xa4

humax# /mod/bin/lsof | grep Se
humaxtv 202 root 24 0000 8,2 1392551000 4295000448 /mnt/hd2/My Video/Seitsem\xc3\xa4n Uutiset_20140207_1900.ts
humaxtv 202 root 25 0000 8,2 1392550952 4295000448 /mnt/hd2/My Video/Seitsem\xc3\xa4n Uutiset_20140207_1900.nts

But the string passed in from sweeper isn't in that format, but is like below:

16/02/2014 11:50 - + Sweeper processing /media/My Video/Seitsemän Uutiset_20140207_1900.ts
CMD -/mod/bin/hmt -p {/media/My Video/Seitsemän Uutiset_20140207_1900.ts}-
inuse? /media/My Video/Seitsemän Uutiset_20140207_1900.ts
system inuse /media/My Video/Seitsemän Uutiset_20140207_1900

This explains why sweeper couldn't see that the file was in use.

I fixed it (sort of) by adding some code to system.class to map the lsof output back to the format sweeper uses although there are probably more elegant ways to do this, and I’ve only mapped lower case a with dots, not other accented characters

proc {system inuse} {file} {
# Is anything using the file?
set file [file rootname [file tail $file]]
set c 0
foreach line [split [exec /mod/bin/lsof] "\n"] {
#map control characters -> their ascii equivalent
set line [string map { \\xc3\\xa4 ä} $line]
if {[string first $file $line] >= 0} { incr c }
}
if {$c > 0} { return 1 }
return 0

}
 
I proposed getting the Humax to convert ALL special characters to a - z, A - Z (Link HERE), some time ago as there are many instances where accented characters can cause problems
 
riku2 said:
Sweeper has a problem with shows with accented characters in the name. I am in Finland and the name of the 7pm news is in finnish (Seitsemän Uutiset).
Click to expand...
Thanks - that's a good analysis. I've created a test case and will come up with a general solution..
 
riku2 said:
proc {system inuse} {file} {
# Is anything using the file?
set file [file rootname [file tail $file]]
set c 0
foreach line [split [exec /mod/bin/lsof] "\n"] {
#map control characters -> their ascii equivalent
set line [string map { \\xc3\\xa4 ä} $line]
if {[string first $file $line] >= 0} { incr c }
}
if {$c > 0} { return 1 }
return 0

}
Click to expand...

Here's a general fix for system inuse that I will add to the next web interface release. It looks like sweeper already skips any files with high-bit characters in the standard video directory but not in sub-folders. I can remove that restriction too.

Code: 
...
if {![exists -proc binary]} { package require binary }
...
proc {system inuse} {file} {
        # Is anything using the file?
        set file [file rootname [file tail $file]]
 
        # Escape any unicode characters to match lsof output.
        foreach range [lreverse [\
            regexp -inline -indices -all -- {[\x80-\xff]} $file]] {
                set i [lindex $range 0]
                binary scan [string index $file $i] H2 hex
                set file [string replace $file $i $i "\\x$hex"]
        }
 
        set c 0
        foreach line [split [exec /mod/bin/lsof] "\n"] {
                if {[string first $file $line] >= 0} { incr c }
        }
        if {$c > 0} { return 1 }
        return 0
}
 
Now that the web interface libraries can properly detect in-use recordings with unicode characters in their names, I've updated sweeper so that it will no longer ignore one-off recordings with names like this.
 
Border said:
Sweeper working with É ! Thanks.
Click to expand...
Thanks for the confirmation! riku2 deserves the credit for finding the issue and the exact cause and prompting me to fix it. I still need to look at general support for unicode/utf-8 at some point but it hasn't approached the top of the list yet....
 
Status
Not open for further replies.
Back
Top