[DetectAds] Announcing DetectAds version 2

[/df]

On the HD, the wonderful ad-detection only works on already decrypted files; for encrypted files, it tries to use chaseget and reports "Content Sharing disabled".

Now that we have on-box decryption, wouldn't it be appropriate for detectads to trigger decryption internally if DLNA isn't available? Maybe a new method ts::decrypt would help with that, and could be used in the main Webif decrypt code as well?

Also, should there be an option to replace the encrypted recording, after ad-detection, with the ad-detected decrypted working copy, or is that meant to happen anyway but not working for me?

And isn't line 913 in /mod/webif/plugin/detectads/detectads.jim redundant?

 

[MymsMan]

Now that we have on-box decryption, wouldn't it be appropriate for detectads to trigger decryption internally if DLNA isn't available? Maybe a new method ts::decrypt would help with that, and could be used in the main Webif decrypt code as well?

I don't have an HD so have not bothered to investigate the similarities and differences between models and may therefore be talking codswallop.

I think it would be more appropriate to put on-box software decryption into auto-decryption for use when DLNA decryption is not available. I think this would have the benefit of minimising differences between HD/HDR allowing the full auto process to run including other processes like audio extraction that require a decrypted file and are automatically triggered on completion of decryption.

What would be useful would be a version of the software decryption that can run as a pipeline stage during recording and feeding decrypted output via stdout to the next stage of the pipe - this would allow me to replace chaseget and potentially allow chaserun processing on the HD.

Also, should there be an option to replace the encrypted recording, after ad-detection, with the ad-detected decrypted working copy, or is that meant to happen anyway but not working for me?

That should happen if you are bookmarking (with -delOrig y option) while if you are cropping you should have the decrypted, shrunk and cropped output.

And isn't line 913 in /mod/webif/plugin/detectads/detectads.jim redundant?

Quite possibly, code, like garages, become cluttered with things that possibly once had a use and could do with a periodic spring clean but there is always a lurking fear that if you throw something out you will found out the reason you were keeping it
If it ain't broke don't fix it

 

[Black Hole]

I think it would be more appropriate to put on-box software decryption into auto-decryption for use when DLNA decryption is not available.

It's not ridiculously slow... but it ain't fast either. IMO processes such as this (including ad detection) should be used sparingly on a HD-FOX, and specifically targetted on a case-by-case basis.

 

[Trev]

What 'we' really need is the equivalent of the Foxsat 'nowster's patch' that stops encryption in the first place. But I suppose this has been investigated to death for the T2

 

[MymsMan]

It's not ridiculously slow... but it ain't fast either. IMO processes such as this (including ad detection) should be used sparingly on a HD-FOX, and specifically targetted on a case-by-case basis.

While specifying recursive auto decrypt on My Video may not be a good idea for an HD if people do want to run detectads and other processes requiring decryption they should be able to do so using auto processing and not required to run them manually - if it is too much overhead they will stop.

For chaserun processing it only needs to be a little faster than recording speed.

 

[Black Hole]

For chaserun processing it only needs to be a little faster than recording speed.

Testing...

 

[Trev]

I fail to see the problem. I run detectads in chaserun. What is the problem?

 

[Black Hole]

I fail to see the problem. I run detectads in chaserun. What is the problem?

Not currently possible with a HD-FOX, because the HD-FOX has no DLNA server and therefore no means to perform chase decryption (as currently defined). The question is whether chase decryption can/should be implemented for HD-FOX using software decryption.

I've just decrypted a 1h StDef recording (1GB) on an HD-FOX in about 12 minutes, so it's quicker than I thought.

 

[MymsMan]

Apart from the builtin disk drive what are the significant differences between the HDR and HD models?
We know they can run most of the same CF software

I wondered if there was any chance of modifying an HDR firmware sufficiently to allow it install on an HD and provide a DLNA-decryption server that way

 

[prpr]

I wondered if there was any chance of modifying an HDR firmware sufficiently to allow it install on an HD and provide a DLNA-decryption server that way

Isn't that what BootHDR essentially does (in a rather ugly clunky way)?
I always thought the process could be improved, especially since I worked out how to stop and restart the humaxtv app. properly, but never had enough motivation to try it.

 

[Trev]

Not currently possible with a HD-FOX

Doh.
I must pay attention.
I must pay attention
......................

 

[Black Hole]

I wondered if there was any chance of modifying an HDR firmware sufficiently to allow it install on an HD and provide a DLNA-decryption server that way

Isn't that what BootHDR essentially does (in a rather ugly clunky way)?

Yes, but by making the HD-FOX boot the HDR-FOX firmware, the HD-FOX hardware becomes essentially non-functional (as a PVR) except for gaining DLNA server capability. The process of locating all the elements of the DLNA server code within HDR-FOX and transplanting them to HD-FOX would be extremely complex with undocumented binary.

Using software for chase decryption seems like the way to go, and makes sense for HDR-FOX too (only one fork to maintain, and no dependence on DLNA being active). It would also be possible to control the decryption rate to be real time, minimising the average processor load.

Apart from the builtin disk drive what are the significant differences between the HDR and HD models?

https://hummy.tv/forum/threads/important-the-differences-between-hd-fox-t2-and-hdr-fox-t2.1365/

 

[Black Hole]

I always thought the process could be improved, especially since I worked out how to stop and restart the humaxtv app. properly, but never had enough motivation to try it.

I think that horse has bolted. Is anyone still using BootHDR now we have software decryption? I never made it work (not that I had much motivation with three HDRs), but now there is software decryption I'm happy to use my bedroom HD-FOX for routine recording as well.

 

[Black Hole]

For chaserun processing it only needs to be a little faster than recording speed.

Testing...

I've just decrypted a 1h StDef recording (1GB) on an HD-FOX in about 12 minutes, so it's quicker than I thought.

I put another 1.03 GiB 1h StDef recording on the queue for decryption, and the queue status reports a run time of 9m2s @ 1.94 MiB/s.

 

[af123]

I put another 1.03 GiB 1h StDef recording on the queue for decryption, and the queue status reports a run time of 9m2s @ 1.94 MiB/s.

With the encryption key set to all 3s or something similar? It is a lot faster with a repeated key than without.

 

[Black Hole]

You're right - I've set it to all zeros.

 

[af123]

What 'we' really need is the equivalent of the Foxsat 'nowster's patch' that stops encryption in the first place. But I suppose this has been investigated to death for the T2

I pick it up and look from time to time but there is no easy solution for this. On the Foxsat, it was just a matter of patching a function with an obvious name to do nothing, but the foxsat is built to only encrypt freesat stuff so the logic is already in there.

 

[/df]

I don't have an HD so have not bothered to investigate the similarities and differences between models and may therefore be talking codswallop. ...

Hard to credit.

I think it would be more appropriate to put on-box software decryption into auto-decryption for use when DLNA decryption is not available. ...

That happens already: see eg /mod/webif/html/browse/decrypt/execute.jim lines 64ff.

My "work"flow is to run detectads from Sweeper rules which I'd like to have the same regardless of platform. So currently I have to queue a recording for decryption and then queue it for ad-detection.

What would be useful would be a version of the software decryption that can run as a pipeline stage during recording and feeding decrypted output via stdout to the next stage of the pipe - this would allow me to replace chaseget and potentially allow chaserun processing on the HD.

So it would, and I was wondering how to do it. Until stripts has that option, you can do this:

#!/mod/bin/jimsh

# params: recording [output]

lassign $argv rec out
# allow .ts or not
if {[file extension $rec] eq ".ts"} {
        set rec [file rootname $rec]
        }
# recording is there?
if { ! [file readable "$rec.ts"]} {
        exit 1
        }

# no [file mkfifo ...]
proc mkfifo {fifo} {
        if {[catch -signal {exec mkfifo "$fifo"} msg opts]} {
                incr opts(-level)
                return {*}$opts $msg
        }
        return $msg
}

# extract status from the list returned by wait
proc retval {waitres} {
        switch [lindex $waitres 0] {
        CHILDSTATUS {return [lindex $waitres 2]}
        default {return -1}
        }
}

proc mktmpbase {template} {
        set tempfile [file tempfile $template]
        file delete $tempfile
        return $tempfile
}

# run stripts to decrypt rec, output to out if given, or stdout
proc main {rec out} {
        # this is going to be a FIFO .ts
        set pipe [mktmpbase "decr_pipeXXXXXX"]
        if {![catch {mkfifo "${pipe}.ts"}]} { 
                # spawn decryption to FIFO
                set dpid [exec stripts -@@ "$rec" "$pipe" >@stderr &]
                if { "$out" ne "" } {
                        set outparms [list > "$out"]
                } else {
                        set  outparms {}
                }
                # spawn copy from FIFO to output
                set cpid [exec cat "${pipe}.ts" {*}$outparms &]
                # wait (for quite a long time)
                set cret [retval [wait $cpid]]
                # this should have finished already
                set dret [retval [wait $dpid]]
                # clear up (is this needed?) ...
                wait
                # ... and all the unwanted outputs (FIFO, .nts, etc)
                foreach pipefrag [glob -nocomplain "$pipe.*"] { 
                        file delete -- $pipefrag
                }
                # return a code: the stripts code if that failed, or the cat code
                if {$dret == 0} {
                        set dret $cret
                } else {
                        # if decryption failed, clear up the output
                        if { "$out" ne "" } {
                                file delete $out
                        }
                }
                return $dret
        }
        return -1
}

main $rec $out

This recording is just less than an hour (key set to all 0s):

humax# ./decrypt_sw.jim Bodyguard_20180902_2100 > qqq.ts
Processed in: 733.88s
humax# ls -l
total 5248632
-rw------- 1 root root      13888 Sep 13  2018 Bodyguard_20180902_2100.hmt
-rw------- 1 root root    5673728 Sep  8  2018 Bodyguard_20180902_2100.nts
-rw------- 1 root root      43680 Sep  8  2018 Bodyguard_20180902_2100.thm
-rw------- 1 root root 1340895232 Jan 24 22:26 Bodyguard_20180902_2100.ts
-rwx--x--x 1 root root       1744 Jan 25 12:57 decrypt_sw.jim
-rw------- 1 root root 1340895232 Jan 25 13:09 qqq.ts
humax#

That should happen if you are bookmarking (with -delOrig y option) while if you are cropping you should have the decrypted, shrunk and cropped output.

So if I set -delOrig y on the HDR (for now) I should see the decrypted icon as well? I'll have to try it.

... code, like garages, become cluttered with things that possibly once had a use and could do with a periodic spring clean but there is always a lurking fear that if you throw something out you will find out the reason you were keeping it ...

I saw the all too familiar signs of an interrupted train of thought: have a look.

And

... Is anyone still using BootHDR now we have software decryption? ...

-1
Happy to be free of the BootHDR process, ingenious though it was.

 

[/df]

... there is no easy solution for [patching out encryption]. ...

Quite possibly there's no accessible encryption code in the humax_tv blob, and encryption is part of the process off-loaded from the Linux processor to the SoC's AV pipeline, so the encrypted stream arrives from the pipeline already encrypted to be written to the recording file (and conversely for local and DLNA playback and USB copy).

 

[MymsMan]

I think that horse has bolted. Is anyone still using BootHDR now we have software decryption? I never made it work (not that I had much motivation with three HDRs), but now there is software decryption I'm happy to use my bedroom HD-FOX for routine recording as well.

Having read your guide I'll agree that the bootHDR based approach is probably a non starter at this time but I think point 9 is overly pessimistic:

DON'T FORGET that references to decrypted download options, decrypt-in-place from the WebIF OPT+ menu, and unencrypt (automated decryption in the background) DO NOT APPLY to the HD-FOX. Sorry. Nothing we can do about it. Once you do have a decrypted recording though, options to convert to MP3 and extract MPG should become available (install ffmpeg).

There is a lot to we can (now) do about it.

Auto-processing can already handle the -direct option for software decryption so it should be a trivial change to autodecrypt to automatically apply that flag when running on an HD machine with DLNA inactive,

With that change and the recommendation to use repeating keys the HD becomes almost as fully functional as the HDR, albeit a bit slower.

The other limitations, access to the TSR and portal buffers are much less important especially with youtuble-dl for iplayer acces.