[DetectAds] Announcing DetectAds version 2

[af123]

The system should have gone into Standby 3 hours after the end of The Chase but if this was one of the late night episodes this may have been too close to the 4:20 wake up and the Humax decided to stay awake.

This is only true if people have auto-standby enabled which it seems, from another recent thread, that isn't as common as I thought.

 

[MymsMan]

I performed some 'Chaserun' detection overnight, with the unit in stand-by It didn't go well:

View attachment 1884

Both Frasier & Lorraine failed to process:
View attachment 1885

Detectads.log:

37    03/09/2015 09:06:16 DA(1340)- done...processed /media/My Video/Frasier_20150903_0901.ts in 0.362s - 0 ad breaks detected
36    03/09/2015 09:06:16 DA(1340)- Incomplete data retrieval 196157440 bytes missing
35    03/09/2015 09:01:15 DA(1340)- ==DETECTADS Chase Run: /media/My Video/Frasier_20150903_0901.ts
34    03/09/2015 09:01:15 RM(1334)-   DETECTADS: Started /media/My Video/Frasier_20150903_0901.ts for chaserun advert detection
33    03/09/2015 09:01:15 RM(1334)-   DETECTADS: Checking /media/My Video/Frasier_20150903_0901.ts (Channel 4 HD) for channel exclusion
32    03/09/2015 08:34:33 DA(1207)- done...processed /media/My Video/Lorraine_20150903_0829.ts in 0.451s - 0 ad breaks detected
31    03/09/2015 08:34:33 DA(1207)- Incomplete data retrieval 198893568 bytes missing
30    03/09/2015 08:29:34 DA(1207)- ==DETECTADS Chase Run: /media/My Video/Lorraine_20150903_0829.ts
29    03/09/2015 08:29:34 RM(1204)-   DETECTADS: Started /media/My Video/Lorraine_20150903_0829.ts for chaserun advert detection
28    03/09/2015 08:29:34 RM(1204)-   DETECTADS: Checking /media/My Video/Lorraine_20150903_0829.ts (ITV HD) for channel exclusion

Any ideas?

Could you check that you have the libsndfile package installed?
I added that to the dependency list for DetectAds around the same time that you were reporting you were getting the old version of the package installed so it is possible that you missed the latest update - the current detectads package level should be 0.2.0-3

If that isn't the problem could you turn Auto-processing log level up the highest in general settings and try again to see if we get any more clues.

 

[MontysEvilTwin]

I haven't seen that happen before, indexing of the the -dec file shouldn't impact anything. The problem I was referring to is when the input file is DLNA indexed whilst chaseget is still trying to read it.

I will try to recreate the problem but do you have the detectads log file for the period.

Log here:

02/09/2015 19:48:01 RM(9085)-   DETECTADS: Checking /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_1948.ts (Horror Channel) for channel exclusion
02/09/2015 19:48:01 RM(9085)-   DETECTADS: Started /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_1948.ts for chaserun advert detection
02/09/2015 19:48:02 DA(9110)- ==DETECTADS Chase Run: /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_1948.ts
02/09/2015 19:54:03 DA(9110)-   ad break found (0 - 151)
02/09/2015 19:59:06 RM(19565)-   DETECTADS: Checking /media/My Video/Star Trek_ The Next Generation_20150902_1959.ts (CBS Action) for channel exclusion
02/09/2015 19:59:06 RM(19565)-   DETECTADS: Started /media/My Video/Star Trek_ The Next Generation_20150902_1959.ts for chaserun advert detection
02/09/2015 19:59:07 DA(19572)- ==DETECTADS Chase Run: /media/My Video/Star Trek_ The Next Generation_20150902_1959.ts
02/09/2015 20:09:38 DA(9110)-   ad break found (947 - 1160)
02/09/2015 20:25:03 RM(13055)-   DETECTADS: Checking /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_2025.ts (Horror Channel) for channel exclusion
02/09/2015 20:25:03 RM(13055)-   DETECTADS: Started /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_2025.ts for chaserun advert detection
02/09/2015 20:25:04 DA(13093)- ==DETECTADS Chase Run: /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_2025.ts
02/09/2015 20:25:42 DA(19572)-   ad break found (1215 - 1474)
02/09/2015 20:26:12 DA(9110)-   ad break found (1997 - 2216)
02/09/2015 20:26:12 DA(9110)- /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_1948.ts deleted
02/09/2015 20:26:12 DA(9110)- done...processed /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_1948.ts in 1991.049s - 3 ad breaks detected
02/09/2015 21:08:37 DA(13093)-   ad break found (653 - 851)
02/09/2015 21:13:54 DA(13093)-   ad break found (1889 - 2072)
02/09/2015 21:14:17 DA(13093)- /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_2025.ts deleted
02/09/2015 21:14:17 DA(13093)- done...processed /media/My Video/Doctor Who/Doctor Who_ Resurrection Of The____20150902_2025.ts in 2655.448s - 2 ad breaks detected

I have a similar schedule set up tonight and I'll let you know what happens. I assumed that the problem was related to the DLNA indexing because of the timing, but this could just be a coincidence.

 

[MymsMan]

I could extend the ir package so that MymsMan can query if any physical remote button has been pressed since boot. That might give enough information to safely put the box back into standby after detectads/chaseget has done its thing.
Something like 'if lastboot reason was scheduled recording and no buttons have been pressed' might work.

It needs to be a bit more sophisticated than just 'if lastboot reason was scheduled recording and no buttons have been pressed' since switches in/out of half awake state don't affect the lastboot reason and Trev had pressed the standby button and wants it to be honoured when the recording has completed and all relevant processing has been completed.

If I attempt to do the standby in DetectAds it would probably occur just after the recording finishes and just as auto is starting up which would be a bit unfriendly so I am thinking it might be better to have a periodic task that ran between scheduled auto task and checked whether detectads, auto & possibly other significant tasks were currently running and if none were active and the other conditions were right(1) put the system into standby.

(1) I was being deliberately vague about the 'other conditions' since I am not sure that I yet understand what they all are

 

[Trev]

This is only true if people have auto-standby enabled which it seems, from another recent thread, that isn't as common as I thought.

I don't have auto-standby enabled. I started recording at 23:23 and it was scheduled to finish at 23:40. It detected the start of the first set of ads, but then included the last two ads and a trail. It cut off a few seconds before the actual end of the prog, thus missing out on the chaser's comments and the titles etc.
When I say 'filename', I am referring to what's in the WI Media list. The Chase had got -Crop Done in the middle, and the one that I had processed manually had -crop at the end.

.... and Trev had pressed the standby button and wants it to be honoured when the recording has completed and all relevant processing has been completed.

Yep, that's about the size of it. If the box is put into S/By when stuff has finished, this should fix the BBC RB and the 'being awake' prob that I had this morning?

I was being deliberately vague about the 'other conditions' since I am not sure that I yet understand what they all are

Thanks for your interest.

 

[antipodean]

>> I'm unsure how others will use this great add-on, but I can't see I'd ever need the bookmark at the beginning of the ad-break.
> That is already on the to-do list

I like to have bothmarks there, as I can see on the time line that it has correctly detected the add breaks, and so I know that I can jump to the next book mark without having to back up if it happened to miss the end of the book mark.

 

[Trev]

I notice that in the log for detect ads, there is a log entry for when I hit the power button to put the box to SBy. Could this event not be used as a flag to indicate that the user had tried to go to SBy? Or was this detectads switching it back on after I had switched it off? Either way, surely it indicates that the box has been switched to SBy as there was no such entry when I recorded Tipping Point this afternoon and let it runn without going to SBy? But then again, what do I know. ;=)

7 02/09/2015 23:40:07 DA(8222)- ad break found (888 - 997)
6 02/09/2015 23:32:25 DA(8222)- ad break found (141 - 394)
5 02/09/2015 23:28:25 - CG(8542)- System power on attempted
4 02/09/2015 23:28:20 NS(8222)- progLen = 0s, 0 bookmarks, HD = 1
3 02/09/2015 23:23:21 DA(8222)- ==DETECTADS Chase Run: /media/My Video/The Chase_ Celebrity Special_20150902_2323.ts
2 02/09/2015 23:23:21 RM(8219)- DETECTADS: Started /media/My Video/The Chase_ Celebrity Special_20150902_2323.ts for chaserun advert detection
1 02/09/2015 23:23:20 RM(8219)- DETECTADS: Checking /media/My Video/The Chase_ Celebrity Special_20150902_2323.ts (ITV HD) for channel exclusion

 

[MymsMan]

This was chaseget switching the system back on!

In the log I use CG for messages generated by chaseget DA for the main detectads component, RM for recmon, NS for nicesplice,
The number in parentheses is the associated process id - it can get quite confusing when there are multiple recordings

 

[Black Hole]

I apologise but I have not followed this thread too closely. There are a couple of things that bother me about automatic detection and cropping of ads, that I just want to throw into the pot to have knocked down or dealt with in an appropriate way.

1. What if the ad breaks are incorrectly detected. The crop function would eviscerate the recording, and it would then be down to the user to notice that before the original disappeared from the recycle bin. Therefore, for most telly (which is watch&delete), it is perfectly adequate to have a bookmark at the end of each ad break - when the ads come on, press the "next bookmark" button. If the bookmark isn't in the right place, it won't be a disaster. This is also easier to explain to the family than why their favourite programme has bits missing. The bookmark button (mostly unused) becomes the "skip ads" button.

2. How does the cropper know whether to crop to the first bookmark and then alternate, or keep to the first bookmark and then alternate? It is entirely possible that hhe recording started in the ad break before the programme, or at the start of the programme. If the implementation is to compare the amount that will be kept in the crop with either strategy and use the one that keeps the most, that won't necessarily work on a 5-min kids cartoon.

I don't think the lack of bookmarks is a satisfactory condition for Sweeper (or a recursive algorithm). Once cropped, won't the bookmarks be gone, and therefore the recording becomes liable for ad detection again? There needs to be some kind of flag, and if no other options are forthcoming I suggest a character appended to the file name.

 

[MontysEvilTwin]

To add to post #43, similar scheduled recordings tonight ran without any problems. The log is below for comparison:

03/09/2015 19:48:03 RM(4441)-   DETECTADS: Checking /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_1948.ts (Horror Channel) for channel exclusion
03/09/2015 19:48:03 RM(4441)-   DETECTADS: Started /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_1948.ts for chaserun advert detection
03/09/2015 19:48:03 DA(4451)- ==DETECTADS Chase Run: /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_1948.ts
03/09/2015 19:53:00 - CG(8552)- System power on attempted
03/09/2015 19:55:21 DA(4451)-   ad break found (122 - 217)
03/09/2015 19:59:02 RM(13486)-   DETECTADS: Checking /media/My Video/Star Trek_ The Next Generation_20150903_1959.ts (CBS Action) for channel exclusion
03/09/2015 19:59:02 RM(13486)-   DETECTADS: Started /media/My Video/Star Trek_ The Next Generation_20150903_1959.ts for chaserun advert detection
03/09/2015 19:59:02 DA(13542)- ==DETECTADS Chase Run: /media/My Video/Star Trek_ The Next Generation_20150903_1959.ts
03/09/2015 20:10:43 DA(4451)-   ad break found (939 - 1140)
03/09/2015 20:22:44 DA(13542)-   ad break found (1036 - 1294)
03/09/2015 20:25:04 RM(5942)-   DETECTADS: Checking /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_2025.ts (Horror Channel) for channel exclusion
03/09/2015 20:25:04 RM(5942)-   DETECTADS: Started /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_2025.ts for chaserun advert detection
03/09/2015 20:25:04 DA(5957)- ==DETECTADS Chase Run: /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_2025.ts
03/09/2015 20:26:18 DA(4451)-   ad break found (1956 - 2216)
03/09/2015 20:26:19 DA(4451)- /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_1948.ts deleted
03/09/2015 20:26:19 DA(4451)- done...processed /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_1948.ts in 1998.912s - 3 ad breaks detected
03/09/2015 20:35:25 DA(13542)-   ad break found (1804 - 2063)
03/09/2015 20:44:07 DA(5957)-   ad break found (830 - 1031)
03/09/2015 20:49:54 DA(13542)-   ad break found (2681 - 2940)
03/09/2015 21:02:17 DA(5957)-   ad break found (1912 - 2214)
03/09/2015 21:02:17 DA(5957)- /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_2025.ts deleted
03/09/2015 21:02:17 DA(5957)- done...processed /media/My Video/Doctor Who/Doctor Who_ The Caves Of Androzani_20150903_2025.ts in 1935.154s - 2 ad breaks detected
03/09/2015 21:05:14 DA(13542)- /media/My Video/Star Trek_ The Next Generation_20150903_1959.ts deleted
03/09/2015 21:05:14 DA(13542)- done...processed /media/My Video/Star Trek_ The Next Generation_20150903_1959.ts in 3673.82s - 3 ad breaks detected

 

[Trev]

This was chaseget switching the system back on!

Then if it has to turn it back on, surely it must know that it's been turned off if it finds it off whilst it is processing, thus turn it off again when processing finished?

 

[MymsMan]

Then if it has to turn it back on, surely it must know that it's been turned off if it finds it off whilst it is processing, thus turn it off again when processing finished?

The difficulty is knowing whether SWMBO has sat down in front of the TV in the meantime and would be upset if the box suddenly turned itself off without warning

 

[MymsMan]

I apologise but I have not followed this thread too closely. There are a couple of things that bother me about automatic detection and cropping of ads, that I just want to throw into the pot to have knocked down or dealt with in an appropriate way.

1. What if the ad breaks are incorrectly detected. The crop function would eviscerate the recording, and it would then be down to the user to notice that before the original disappeared from the recycle bin. Therefore, for most telly (which is watch&delete), it is perfectly adequate to have a bookmark at the end of each ad break - when the ads come on, press the "next bookmark" button. If the bookmark isn't in the right place, it won't be a disaster. This is also easier to explain to the family than why their favourite programme has bits missing. The bookmark button (mostly unused) becomes the "skip ads" button.

In my experience of a year of extensive use of the original DetectAds the most common failure is failure to recognise an ad break whilst false detection of part of a program is extremely rare. There is an explicit option in the settings to allow the original file to be retained and not sent to the recycle bin to provide a degree of security until you have established confidence that detection and cropping is working well in your environment.

If you don't crop and it fails to detect an ad break when you are watching and come across the break and press bookmark it unexpectedly skips the whole next program part and is quite difficult to reposition back to the correct spot. An undetected break in a cropped program is less of a nuisance - you either watch it or press skip forward a few times as in the past.

2. How does the cropper know whether to crop to the first bookmark and then alternate, or keep to the first bookmark and then alternate? It is entirely possible that hhe recording started in the ad break before the programme, or at the start of the programme. If the implementation is to compare the amount that will be kept in the crop with either strategy and use the one that keeps the most, that won't necessarily work on a 5-min kids cartoon.

nicesplice doesn't use the bookmarks, instead it is passed a set of explicit -cut commands telling it exactly which pieces to remove.

I don't think the lack of bookmarks is a satisfactory condition for Sweeper (or a recursive algorithm). Once cropped, won't the bookmarks be gone, and therefore the recording becomes liable for ad detection again? There needs to be some kind of flag, and if no other options are forthcoming I suggest a character appended to the file name.

The cropped output file has the suffix -crop added to the file name, no change is made to the input file name but could be if that was that was needed to help prevent reprocessing by sweeper

 

[Trev]

The difficulty is knowing whether SWMBO has sat down in front of the TV in the meantime...

Not a problem in my domestic situation. She doesn't know how to use the box and goes to bed before me anyway. Although I do agree, it could be a problem for others.

1. What if the ad breaks are incorrectly detected. The crop function would eviscerate the recording, and it would then be down to the user to notice that before the original disappeared from the recycle bin.

Or you find when you are actually watching the cropped programme that it has done it wrong and HWMBO gives you severe earache because she missed the last bit, and you can't be assed to find the last bit on the original.
Box on and on BBC RB again this morning

 

[damianiw]

Hi I recently tried to uninstall IR because its stopped working but it said it had a dependency on chaseget, which was required by detect Ads ?

I've removed all 3 and reinstalled just IR and although not working chaseget hasn't been reinstalled not sure what's messed up the dependencies but thought id post in case anyone finds similar

 

[Black Hole]

This kind of thing has already been mentioned in this topic.

 

[Rob Singlehurst]

This is performing extremely well, I just let it process all recordings . One thing I have noticed is that if you start to playback a recording whilst processing is still in progress there will be an orphaned .hmt file visible in the webif media browser.


I think the playback started 15 mins after the recording ended. We then stopped the playback and saw that the recording length was 45 mins (i.e. detectads had done it's business ) and started playback again. I can delete the .hmt by telnetting in, can't delete it from the webif media browser.
Ideally I would either use the chaseget method or wait until I had seen the recording length had been reduced but obviously not everyone else in the household will wait

Another thought, what would happen if you started to record a program part way through using the button on the remote - will detectads get confused and crop out the wrong parts or does it analyse the length and frequency of the "ads" to check that they look sensible?
Excellent work though MymsMan!!!

 

[MymsMan]

This is performing extremely well, I just let it process all recordings . One thing I have noticed is that if you start to playback a recording whilst processing is still in progress there will be an orphaned .hmt file visible in the webif media browser.
View attachment 1892

I think the playback started 15 mins after the recording ended. We then stopped the playback and saw that the recording length was 45 mins (i.e. detectads had done it's business ) and started playback again. I can delete the .hmt by telnetting in, can't delete it from the webif media browser.
Ideally I would either use the chaseget method or wait until I had seen the recording length had been reduced but obviously not everyone else in the household will wait

Another thought, what would happen if you started to record a program part way through using the button on the remote - will detectads get confused and crop out the wrong parts or does it analyse the length and frequency of the "ads" to check that they look sensible?
Excellent work though MymsMan!!!

The Humax creates a new but invalid hmt if a recording is deleted/moved whilst it is being played. You can get the same effect by deleting a program from the webif whilst it is playing (I accidentally did that myself this evening). Because the file already exists in the dustbin the undelete package cant move the duplicate to the dustbin it leaves it in-place. I think it would be better to move it to the dustbin with a suffixed name.
You can get around the problem by renaming the file in the webif (add a character to end) before deleting it.

I will add an inuse check before deleting the original in DetectAds but I have been too busy recently to get around to making all the changes I want to.

DetectAds does analyze the frequency of silences between ads to detect adbreaks so normally shouldn't be confused by starting/stopping during a program but if you started recording just before a break or stop just after one it might think the short program section is part of the ad break.

 

[Rob Singlehurst]

That's a much easier way of deleting the hmt, I'll try that next time it happens and I'll bear that in mind when doing manual recordings.

Thanks,

 

[Black Hole]

MymsMan: how does DetectAds interact with arbookmarks? Are pre-existing bookmarks ignored and over-ridden?

af123: how does the existence of bookmarks due to arbookmarks affect the sweeper detection? What does arbookmarks do if DetectAds has already processed a recording?