[DetectAds] Announcing DetectAds version 2

[MymsMan]

I have just uploaded detectads 0.2.0-2 and chaseget 0.1.0-0 to the package catalogue.

This is a major upgrade to the package that was originally created by njm.

Features included in this release include:

• Channel exclusion
  This allows non-commercial channels (e.g. BBC and radio) to be excluded from automatic processing
  
• Background processing from browse panel
  Multiple recordings can be added to the processing queue without waiting for each to complete
  
• Command line
  DetectAds can also be invoked from a command line or script
  
• Automatic cropping of breaks option
  Runs NiceSplice to delete the ad breaks found and shrink the recording
  
• Detect whilst recording
  Using the new chaserun function the detection process runs in parallel with the actual recording so the processing completes within a few seconds of the recording finishing. You can even, with some limitations, start to watch your program adfree whilst it is still recording.

I have also created a wiki page to describe usage of the package in more detail http://wiki.hummy.tv/wiki/DetectAds

Please add any comments, suggestions and questions to this thread

 

[Rob Singlehurst]

This looks great

Can it be setup to just go through all sub-directories (in the same way as Recursive Auto-Decrpyt) or is it used on single files? I'm using the "Traditional Processing" setting at the moment.

 

[MymsMan]

If the files have not yet been decrypted and you have specified 'Automatically process recordings following auto-decryption?' then when you turn on Auto-Decrypt for the folder then the files will be queued for detectads processing as they are decrypted.

If the files have already been decrypted then there is no automatic process to subsequently run DetectAds since their is no file flag to tell which files have/have not been processed.

Automatic processed is mainly intended for processing new recordings as they are made rather than processing the archive of pre-existing recordings.

If you do wish to process your archive it should be quite easy to write a script file that uses the CLI to add all of the recordings in a folder to the process queue. I don't have a script to process an entire folder but will look into writing one. Remember that processing an existing recording is slow so that it would take a long time to process a substantial archive.

 

[Rob Singlehurst]

Thank-you, I should have read the wiki in a but more detail I don't have many recordings at the moment as my old Hummy died and I hadn't properly understood the decryption! Will this happily work alongside the Shrink option too?
A nagging question I've also had is that if the Front Panel or remote is used to put the Hummy into Standby while any of these processes are underway does the state get remembered or does Standby get delayed until the processes have finished?

 

[MontysEvilTwin]

I am having some problems getting this up and running. From the Detectads log:

01/09/2015 12:08:01 DA(7344)- Processing item 1 (/media/My Video/Keiser Report/Keiser Report_20150901_0826.ts) from the queue
01/09/2015 12:08:01 DA(7344)-   DETECTADS: /media/My Video/Keiser Report/Keiser Report_20150901_0826.ts
01/09/2015 12:08:01 DA(7344)- adDetect pids 7356 7357
01/09/2015 12:08:01 DA(7344)- Initial bookmarks:
01/09/2015 12:08:03 DA(7344)- Final bookmarks:
01/09/2015 12:28:01 DA(24507)- Processing item 1 (/media/My Video/Keiser Report/Keiser Report_20150901_0826.ts) from the queue
01/09/2015 12:28:01 DA(24507)-   DETECTADS: /media/My Video/Keiser Report/Keiser Report_20150901_0826.ts
01/09/2015 12:28:01 DA(24507)- adDetect pids 24515 24516
01/09/2015 12:28:01 DA(24507)- Initial bookmarks:
01/09/2015 12:28:02 DA(24507)- Final bookmarks:

The file has decrypted and is in the queue. From the log it looks like the process starts and finishes in a couple of seconds, with no time to run the file analysis. This is repeated the next time a scheduled autoprocess happens (occurs 3 minutes after the start of each auto run). Manually running the analysis fails in the same way. I've tried several recordings on two machines with similar results.

 

[Rob Singlehurst]

I'm getting the same thing (I think). I invoked this manually by using the Detect Adverts in Opt+. I turned on verbose logging part way through to see what was going on (actually started it 10.30 last night).

01/09/2015 14:48:01 DA(5272)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 15:08:02 DA(8878)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 15:28:01 DA(12677)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 15:28:01 DA(12677)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 15:28:01 DA(12677)- adDetect pids 12684 12685
01/09/2015 15:28:01 DA(12677)- Initial bookmarks:
01/09/2015 15:28:03 DA(12677)- Final bookmarks:
01/09/2015 15:48:01 DA(16003)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 15:48:01 DA(16003)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 15:48:01 DA(16003)- adDetect pids 16011 16012
01/09/2015 15:48:01 DA(16003)- Initial bookmarks:
01/09/2015 15:48:02 DA(16003)- Final bookmarks:
01/09/2015 16:08:01 DA(19325)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 16:08:01 DA(19325)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 16:08:01 DA(19325)- adDetect pids 19333 19334
01/09/2015 16:08:01 DA(19325)- Initial bookmarks:
01/09/2015 16:08:02 DA(19325)- Final bookmarks:
01/09/2015 16:28:01 DA(22670)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 16:28:01 DA(22670)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 16:28:01 DA(22670)- adDetect pids 22677 22678
01/09/2015 16:28:01 DA(22670)- Initial bookmarks:
01/09/2015 16:28:02 DA(22670)- Final bookmarks:
01/09/2015 16:48:01 DA(26069)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 16:48:01 DA(26069)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 16:48:01 DA(26069)- adDetect pids 26075 26076
01/09/2015 16:48:01 DA(26069)- Initial bookmarks:
01/09/2015 16:48:02 DA(26069)- Final bookmarks:
01/09/2015 17:08:01 DA(29393)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 17:08:01 DA(29393)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 17:08:01 DA(29393)- adDetect pids 29399 29400
01/09/2015 17:08:01 DA(29393)- Initial bookmarks:
01/09/2015 17:08:02 DA(29393)- Final bookmarks:
01/09/2015 17:28:02 DA(32683)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 17:28:02 DA(32683)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 17:28:02 DA(32683)- adDetect pids 32691 32692
01/09/2015 17:28:02 DA(32683)- Initial bookmarks:
01/09/2015 17:28:03 DA(32683)- Final bookmarks:
01/09/2015 17:48:02 DA(3589)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 17:48:02 DA(3589)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 17:48:02 DA(3589)- adDetect pids 3595 3596
01/09/2015 17:48:02 DA(3589)- Initial bookmarks:
01/09/2015 17:48:05 DA(3589)- Final bookmarks:
01/09/2015 18:08:01 DA(6913)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 18:08:01 DA(6913)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 18:08:01 DA(6913)- adDetect pids 6921 6922
01/09/2015 18:08:01 DA(6913)- Initial bookmarks:
01/09/2015 18:08:02 DA(6913)- Final bookmarks:
01/09/2015 18:28:01 DA(10692)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 18:28:01 DA(10692)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 18:28:01 DA(10692)- adDetect pids 10700 10701
01/09/2015 18:28:01 DA(10692)- Initial bookmarks:
01/09/2015 18:28:02 DA(10692)- Final bookmarks:
01/09/2015 18:48:01 DA(14006)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 18:48:01 DA(14006)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 18:48:01 DA(14006)- adDetect pids 14014 14015
01/09/2015 18:48:01 DA(14006)- Initial bookmarks:
01/09/2015 18:48:02 DA(14006)- Final bookmarks:
01/09/2015 19:08:01 DA(17197)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 19:08:01 DA(17197)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 19:08:01 DA(17197)- adDetect pids 17205 17206
01/09/2015 19:08:01 DA(17197)- Initial bookmarks:
01/09/2015 19:08:02 DA(17197)- Final bookmarks:
01/09/2015 19:28:01 DA(20727)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 19:28:01 DA(20727)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 19:28:01 DA(20727)- adDetect pids 20732 20733
01/09/2015 19:28:01 DA(20727)- Initial bookmarks:
01/09/2015 19:28:02 DA(20727)- Final bookmarks:
01/09/2015 19:48:01 DA(24354)- Processing item 1 (/media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts) from the queue
01/09/2015 19:48:01 DA(24354)- DETECTADS: /media/My Video/Celebrity Big Brother/New_ Celebrity Big Brother_20150831_2100.ts
01/09/2015 19:48:01 DA(24354)- adDetect pids 24360 24361
01/09/2015 19:48:01 DA(24354)- Initial bookmarks:
01/09/2015 19:48:02 DA(24354)- Final bookmarks:

Stopped automatic processing for the time being as I'm not sure what's going on.

 

[MymsMan]

I am having some problems getting this up and running. From the Detectads log:

The file has decrypted and is in the queue. From the log it looks like the process starts and finishes in a couple of seconds, with no time to run the file analysis. This is repeated the next time a scheduled autoprocess happens (occurs 3 minutes after the start of each auto run). Manually running the analysis fails in the same way. I've tried several recordings on two machines with similar results.

Monty & Rob

Do you have the ffmpeg and libsndfile packages installed?
I need to add then to the package prereq list (the original package also required them but didn't list them so I didn't consider them when updating the control file)

DetectAds has a scheduled task that runs every 20 minutes to process any outstanding entries

To kill a queued task that is continually repeating you can either rename (via the webif) the recording that is causing problems or delete via telnet the queue database

rm /mod/etc/detectads_queue.db

 

[Rob Singlehurst]

Ta
Just added libsndfile (ffmpeg was already installed). I'll leave it a bit longer and see if it picks it up or will it need a reboot?
Ahh, it's processing working now - just checked the log file - thank-you.

And finished 5 Ads detected - New program length 2708s - about 45 mins.

 

[MymsMan]

I have updated the package catalogue to include the missing dependencies and also fix the bug causing the restart loop.
Will be available after 22:00 but no need to update if you already have installed libsndfile.

 

[Rob Singlehurst]

Great, I've set it going on another recording now.

 

[MymsMan]

Thank-you, I should have read the wiki in a but more detail I don't have many recordings at the moment as my old Hummy died and I hadn't properly understood the decryption! Will this happily work alongside the Shrink option too?
A nagging question I've also had is that if the Front Panel or remote is used to put the Hummy into Standby while any of these processes are underway does the state get remembered or does Standby get delayed until the processes have finished?

If you use the automatic crop option then the recording will be shrunk as part of the cropping process so you don't need to use the auto-shrink option but shrinking a recording first will not interfere with ad detection or cropping.

If you put the Humax in standby whilst it is processing a queued recording then the process is terminated and processing of the recording will restart from the beginning of the recording when the system is restarted.

On the other hand if you put the system into standby whilst it is processing an active recording using "process whilst recording" chaseget will attempt to turn the system back on to fully active so that it can continue to decrypt the recording.

 

[af123]

On the other hand if you put the system into standby whilst it is processing an active recording using "process whilst recording" chaseget will attempt to turn the system back on to fully active so that it can continue to decrypt the recording.

.. providing the ir package is installed.

btw, these package updates haven't ended up on the twitter feed due to the character limitation, but I'll post one by hand for people who don't follow the forum.

 

[cdmackay]

This is a hugely impressive piece of work, thanks.

one tiny niggle: from Settings:

• Only select one automatic processing option

in which case, why not make it an exclusive choice, e.g. a radio-button, or whatever they're called, as in "Write recording output to"?

[edit: it would have to be a 3-way button, of course: No; Trad; Shiny]

thanks again, nice one.

 

[Dave F.]

Hi

I've just installed this package.
I sure I'm missing something basic but I'm not getting the full number of options in the settings



What am I missing?
I'm able to process a program successfully from the browse option.

Cheers
Dave F.

 

[MontysEvilTwin]

Dave F. It looks like the settings panel for the old version. Make sure you have the latest version by updating the packages list and applying any updates. If you had the old version installed previously, you may just need to clear your browser cache.

 

[Trev]

I've been trying to follow this thread and had a good stiff read of the WiKi but still have a problem in understanding the mechanics of getting rid of the ads both manually and automatically.
To this end, I have chosen a decrypted (I have auto-decrypt ON) CH4 programme to experiment with and done the following.
1. Opt+ > Detect Adverts
2. Run Analysis now
Wait a while 'till it finishes. But it didn't seem to crop the file. There was not the original in Deleted Items.
3. Opt+ > Crop
The resulting page showed the bookmarks OK, but it indicated that it was going to crop the programme and leave the ads.
4.Invert selection > Crop.
This seemed to work OK, as the recording went down in size from abt. 731MB to 726MB buty I haven't tried playing it to 'make sure'. The new cropped file did not have -crop suffix present. I wonder why it didn't auto crop the file?

Tried another one.
1. Opt+ > Detect Adverts
2. Run Analysis now
Wait a while 'till it finishes. This worked OK and did the crop without input from me and the filename had the -crop suffix. I had a look in Deleted Items for the prog. It was there. Did an Opt+ > Crop on the deleted programme (to see what's what).

Q1 Again, the crop seemed to wanted to Cut the programme and Keep the ads. What's going on there. Is it a bug in the Crop programming, or is it me not understanding something? (Wouldn't be the first time).

I have gone to Settings > Settings for detectads package and set:
Automatically process recordings whilst recording in progress YES
Crop recording following ad detection? YES
Dustbin/delete original recording after processing?YES
Write output recording to Same as input

Q2 Will this be correct for it to automatically decrypt and crop ads from all future recordings? If this is the case, does auto-decrypt need to run separately?
Q3 I did the above detectads settings between my first experiment and second one. Is this why the first one did not automatically crop the ads?
Q4 Any thoughts on why the Opt+ > Crop command needs inverting before cropping?

 

[Rob Singlehurst]

I've not used the chaserun method yet (that's the setting you have selected). I have done some manually using the Opt+ Detect Adverts and as long as Crop is turned on in the settings then the Crop will happen automatically in the pipeline i.e. no need to Opt+ Crop.
On my "My Video" directory I have enabled Recursive Auto-Shrink and Recursive Auto-Decrypt and everything is working spot on. I also have the undelete package installed so my originals are "deleted" into the "Deleted Items" folder.
The main thing I think is you don't need to use the Opt+ Crop - this is taken care of automatically (when you have Crop recording following ad detection Enabled)

 

[Rob Singlehurst]

What happens if you start to watch a recording before allowing any processing on that file to finish? I just had a Recording Failed message when trying to do exactly that - the recording plays fine though - is that message because Hummy is confused?

 

[MymsMan]

I've been trying to follow this thread and had a good stiff read of the WiKi but still have a problem in understanding the mechanics of getting rid of the ads both manually and automatically.
To this end, I have chosen a decrypted (I have auto-decrypt ON) CH4 programme to experiment with and done the following.
1. Opt+ > Detect Adverts
2. Run Analysis now
Wait a while 'till it finishes. But it didn't seem to crop the file. There was not the original in Deleted Items.
3. Opt+ > Crop
The resulting page showed the bookmarks OK, but it indicated that it was going to crop the programme and leave the ads.
4.Invert selection > Crop.
This seemed to work OK, as the recording went down in size from abt. 731MB to 726MB buty I haven't tried playing it to 'make sure'. The new cropped file did not have -crop suffix present. I wonder why it didn't auto crop the file?

Tried another one.
1. Opt+ > Detect Adverts
2. Run Analysis now
Wait a while 'till it finishes. This worked OK and did the crop without input from me and the filename had the -crop suffix. I had a look in Deleted Items for the prog. It was there. Did an Opt+ > Crop on the deleted programme (to see what's what).

Q1 Again, the crop seemed to wanted to Cut the programme and Keep the ads. What's going on there. Is it a bug in the Crop programming, or is it me not understanding something? (Wouldn't be the first time).

I have gone to Settings > Settings for detectads package and set:
Automatically process recordings whilst recording in progress YES
Crop recording following ad detection? YES
Dustbin/delete original recording after processing?YES
Write output recording to Same as input

Q2 Will this be correct for it to automatically decrypt and crop ads from all future recordings? If this is the case, does auto-decrypt need to run separately?
Q3 I did the above detectads settings between my first experiment and second one. Is this why the first one did not automatically crop the ads?
Q4 Any thoughts on why the Opt+ > Crop command needs inverting before cropping?

Q1 Again, the crop seemed to wanted to Cut the programme and Keep the ads. What's going on there. Is it a bug in the Crop programming, or is it me not understanding something? (Wouldn't be the first time).

It is not a bug but a difference in design philosophy between the original package authors. The Silence program (which does the real work) marks the beginning and end of each ad break including those at the start and end of programme whilst nicesplice assumed that with manual bookmarking of breaks you wouldn't bother bookmarking the start and end of program.

I have gone to Settings > Settings for detectads package and set:
Automatically process recordings whilst recording in progress YES
Crop recording following ad detection? YES
Dustbin/delete original recording after processing?YES
Write output recording to Same as input

Q2 Will this be correct for it to automatically decrypt and crop ads from all future recordings? If this is the case, does auto-decrypt need to run separately?

Yes, Those are essentially the settings that I run with
You don't need auto-decrypt turned on unless you want BBC programs that are not processed by detectads to be decrypted. I don't use autodecrypt since I rarely move recordings off the humax.

Q3 I did the above detectads settings between my first experiment and second one. Is this why the first one did not automatically crop the ads?

Yes

Q4 Any thoughts on why the Opt+ > Crop command needs inverting before cropping?

See answer to Q1. Avoiding the need for a separate manual operation and inverting the selection was the reason for including the crop function within this version of DetectAds.

 

[MymsMan]

What happens if you start to watch a recording before allowing any processing on that file to finish? I just had a Recording Failed message when trying to do exactly that - the recording plays fine though - is that message because Hummy is confused?

What is the failure reason?
This can be seen on the webif recording info display or with the hmt utility via telnet.

Watching a recording whilst it is being processed should not cause any problems.

I often get spurious recording failed messages when there is a slight overlap between the end of previous recordings and the start of the next recording.