[DetectAds] Announcing DetectAds version 2

OwenMc · Sep 24, 2015

MymsMan said:
If anyone experiences significantly too short programs after cropping could they please
.....

I'd noticed a pattern with the Rugby, which seems to give the cropping a hard time
- detect ads works fine, all in the right place
- crop first suggestion would remove all the actual play
- invert it, and it LOOKS sensible, but letting it crop ends up with 46 mins, and that seems fairly consistent.

I have originals of all stages, & logs, but if you've reproduced it you're probably OK for those.

O

Black Hole · Sep 25, 2015

What is going on here? What info do you need?

...but when viewed from a networked HD-FOX by DLNA (note the length it claims to be):

Trev · Sep 25, 2015

You seem to have a lurker standing in the shadows.

prpr · Sep 25, 2015

He looks like he's about to wring BH's neck. Maybe it's Raydon? :roflmao:

Black Hole · Sep 25, 2015

The background is that bloke on Quest that modifies engines.

MymsMan · Sep 25, 2015

Black Hole said:
What is going on here? What info do you need?

View attachment 1914

...but when viewed from a networked HD-FOX by DLNA (note the length it claims to be):

View attachment 1915

Bizarre!

So the length appears to be correct when viewed via the webif, but not on the HD-FOX, size is consistent with full length.
Is it correct on the SUI of the HDR-FOX T2?
What length plays (on both machines)?
Are there any unusual message in the DetectAds log?

I don't use DLNA and don't know how information is passed across the network, is it usual to see only file name and not title?
I populate recording start, end and recorded duration fields in hmt files so I don't know where it would get the shorter time from unless it picked up the figures when chaseget started to create the -dec file.
I think that is probably the most likely cause and will be cured when I hide the -dec file during processing and preserve the original file name at completion.

MymsMan · Sep 25, 2015

MymsMan said:
I have just uploaded 0.2.1-2 to package catalogue

It should fix the problem with BBC programs being selected for ad- detection

I have not been able to recreate the problem with very short programmes after cropping or spot any bugs so I have put some more diagnostics in place in the hope of finding some clues

If anyone experiences significantly too short programs after cropping could they please:

Report the problem here, or via private conversation, with all available log data from detectads.log

Turn on debugging level, using General Settings

Rerun ad-detection on the original, undecrypted, input file using Run Analysis in Background from Browse->Detectads

Is problem reproducible? Post log data from the rerun.

Keep the input file - I may need to ask for help with testing any fixes.

Edit: I have now been able to recreate the problem and it is reproducible so I can start to debug the issue and hopefully find a fix soon.

After a session with the debugger I think I have identified and fixed the bug that was causing crop to fail with short recording.

I have uploaded version 0.2.1-3 with this fix - no other changes.

The bug is in the base nicesplice code rather than my modifications so could also affect manual cropping until the nicesplice package is updated.

MontysEvilTwin · Sep 25, 2015

I noticed an oddity when I cropped a file with nicesplice a couple of days ago, I wonder if it is the same bug? The programme start was spot on, so I just wanted to remove the end of the recording. I added a bookmark and selected crop with invertion. The crop completed, but it added some frames from the very end of the original recording to the cropped file, after the crop point. When I added a bookmark at 0 seconds to the original, so that the crop could be done without the invert option, it worked fine.

MymsMan · Sep 26, 2015

MontysEvilTwin said:
I noticed an oddity when I cropped a file with nicesplice a couple of days ago, I wonder if it is the same bug? The programme start was spot on, so I just wanted to remove the end of the recording. I added a bookmark and selected crop with invertion. The crop completed, but it added some frames from the very end of the original recording to the cropped file, after the crop point. When I added a bookmark at 0 seconds to the original, so that the crop could be done without the invert option, it worked fine.

This is highly unlikely to be the same bug but sounds to be nicesplice crop working normally,
When you inverted with a single bookmark it created a cut command with the end point time at the end of recording but the actual crop point is adjusted to a key frame so there could easily be a few frames kept at the end of recording (this could also happen using DetecatAds) when using two bookmarks without inversion it throws everything after the second bookmark away - see the nicesplice wiki pages for details

cdmackay · Sep 26, 2015

I am just now seeing:

Code:

>>> opkg upgrade
Upgrading detectads on root from 0.2.1-2 to 0.2.1-3...
Downloading http://hpkg.tv/hdrfoxt2/base/detectads_0.2.1-3_mipsel.opk.
Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for detectads:
*     chaseget (>= 0.1.1-0) *

on "upgrade all packages".

edit: I have updated the pkg list.

MymsMan · Sep 26, 2015

cdmackay said:
I am just now seeing:

Code:

>>> opkg upgrade Upgrading detectads on root from 0.2.1-2 to 0.2.1-3... Downloading http://hpkg.tv/hdrfoxt2/base/detectads_0.2.1-3_mipsel.opk. Collected errors: * satisfy_dependencies_for: Cannot satisfy the following dependencies for detectads: * chaseget (>= 0.1.1-0) *

on "upgrade all packages".

edit: I have updated the pkg list.

Oops - sorry, the chaseget updates are not ready for prime-time so I have removed the dependency

0.2.1-4 should be visible in catalog at 1am so try again in the morning

MontysEvilTwin · Sep 26, 2015

MymsMan said:
This is highly unlikely to be the same bug but sounds to be nicesplice crop working normally,
When you inverted with a single bookmark it created a cut command with the end point time at the end of recording but the actual crop point is adjusted to a key frame so there could easily be a few frames kept at the end of recording (this could also happen using DetecatAds) when using two bookmarks without inversion it throws everything after the second bookmark away - see the nicesplice wiki pages for details

Thanks for the clarification. I was aware that cuts are made at I-frames (unless you crop from the command line with the '-noAlignCuts' option) but I didn't know exactly how inverting the crop worked. There is no information about inversion in the wiki page you linked to. I vaguely recall there being something in a thread: I'll try and find it.

Black Hole · Sep 26, 2015

MymsMan said:
I don't use DLNA and don't know how information is passed across the network, is it usual to see only file name and not title?

Yes

MymsMan said:
So the length appears to be correct when viewed via the webif, but not on the HD-FOX, size is consistent with full length.
Is it correct on the SUI of the HDR-FOX T2?

Yes

MymsMan said:
What length plays (on both machines)?

It plays fine. Via DLNA, the time bar shows the correct length (even though the DLNA media list shows the strange length).

For completeness, via file share the parameters are the same as local SUI.

MymsMan said:
Are there any unusual message in the DetectAds log?

Code:

401    17/09/2015 22:05:19 RM(15552)- Recmon: -stop {/media/My Video/Fifth Gear/Fifth Gear_20150917_2059-dec}
400    17/09/2015 22:05:14 DA(6739)- =============================================================
399    17/09/2015 22:05:14 DA(6739)- done...processed /media/My Video/Fifth Gear/Fifth Gear_20150917_2059.ts in 3672.332s 01:01:12 - 5 ad breaks detected
398    17/09/2015 22:05:13 DA(6739)- End: Frame 98948 Time: 65:58  Seconds: 3958
397    17/09/2015 22:05:13 - CG(9681)- ChaseGet end /media/My Video/Fifth Gear/Fifth Gear_20150917_2059.ts 1254596608
396    17/09/2015 22:05:05 RM(15216)- Recmon: -stop {/media/My Video/Fifth Gear/Fifth Gear_20150917_2059}
395    17/09/2015 22:04:29 DA(6739)- Hearbeat: Frame 97119 Time: 64:45 Seconds: 3885
394    17/09/2015 22:01:33 DA(6739)- Hearbeat: Frame 93023 Time: 62:01 Seconds: 3721
393    17/09/2015 22:01:33 DA(6739)-   ad break found 57:29-60:31 (3449 - 3631) Frames: 86213 - 90772
392    17/09/2015 21:54:08 DA(6739)- Hearbeat: Frame 82261 Time: 54:50 Seconds: 3290
391    17/09/2015 21:51:46 DA(6739)- Hearbeat: Frame 78165 Time: 52:07 Seconds: 3127
390    17/09/2015 21:51:46 DA(6739)-   ad break found 44:50-50:37 (2690 - 3037) Frames: 67262 - 75914
389    17/09/2015 21:43:35 DA(6739)- Hearbeat: Frame 66008 Time: 44:00 Seconds: 2640
388    17/09/2015 21:40:42 DA(6739)- Hearbeat: Frame 61912 Time: 41:16 Seconds: 2476
387    17/09/2015 21:37:48 DA(6739)- Hearbeat: Frame 57816 Time: 38:33 Seconds: 2313
386    17/09/2015 21:35:27 DA(6739)- Hearbeat: Frame 53720 Time: 35:49 Seconds: 2149
385    17/09/2015 21:35:27 DA(6739)-   ad break found 28:13-34:19 (1693 - 2059) Frames: 42316 - 51469
384    17/09/2015 21:25:49 DA(6739)- Hearbeat: Frame 39270 Time: 26:11 Seconds: 1571
383    17/09/2015 21:22:53 DA(6739)- Hearbeat: Frame 35174 Time: 23:27 Seconds: 1407
382    17/09/2015 21:20:00 DA(6739)- Hearbeat: Frame 31078 Time: 20:43 Seconds: 1243
381    17/09/2015 21:17:39 DA(6739)- Hearbeat: Frame 26982 Time: 17:59 Seconds: 1079
380    17/09/2015 21:14:45 DA(6739)- Hearbeat: Frame 22886 Time: 15:15 Seconds: 915
379    17/09/2015 21:11:50 DA(6739)- Hearbeat: Frame 18790 Time: 12:32 Seconds: 752
378    17/09/2015 21:11:50 DA(6739)-   ad break found 8:13-11:02 (493 - 662) Frames: 12337 - 16539
377    17/09/2015 21:07:58 DA(6739)- Hearbeat: Frame 12287 Time: 8:11 Seconds: 491
376    17/09/2015 21:06:08 DA(6739)- Hearbeat: Frame 8191 Time: 5:28 Seconds: 328
375    17/09/2015 21:04:46 DA(6739)- Hearbeat: Frame 4095 Time: 2:44 Seconds: 164
374    17/09/2015 21:04:39 DA(6739)-   ad break found 0:00-0:46 (0 - 46) Frames: 1 - 1140
373    17/09/2015 21:04:02 - CG(9681)- ChaseGet /media/My Video/Fifth Gear/Fifth Gear_20150917_2059.ts
372    17/09/2015 20:59:01 DA(6739)- Waiting for recording 5min
371    17/09/2015 20:59:01 DA(6739)- ==DETECTADS Chase Run: /media/My Video/Fifth Gear/Fifth Gear_20150917_2059.ts
370    17/09/2015 20:59:01 RM(6732)-   DETECTADS: Started /media/My Video/Fifth Gear/Fifth Gear_20150917_2059.ts for chaserun advert detection
369    17/09/2015 20:59:01 RM(6732)-   DETECTADS: Checking /media/My Video/Fifth Gear/Fifth Gear_20150917_2059.ts (QUEST+1) for channel exclusion

PS: Bookmarks 5 & 6 are not placed at all well, and would have led to a disjointed programme if cropped, but the rest are OK. I don't think detectads coexists very well with arbookmarks - perhaps there should be a way of only running arbookmarks on a recording that has been excluded from detectads.

Trev · Sep 26, 2015

CH4 news tonight had 14byte length error :=(

MontysEvilTwin · Sep 28, 2015

Had a minor issue with detectads. The '-dec' file of a recording appears to be 36 minutes shorter than it actually is. The TS file is the correct length, but it seems that detectads stopped detecting adbreaks and writing to the hmt file early. The process didn't hang completely though as the normal clean-up steps occurred when the recording completed:

Code:

27/09/2015 21:02:34 RM(21421)-   DETECTADS: Checking /media/My Video/Kingdom of Heaven_20150927_2102.ts (Film4) for channel exclusion
27/09/2015 21:02:34 RM(21421)-   DETECTADS: Started /media/My Video/Kingdom of Heaven_20150927_2102.ts for chaserun advert detection
27/09/2015 21:02:35 DA(21424)- ==DETECTADS Chase Run: /media/My Video/Kingdom of Heaven_20150927_2102.ts
27/09/2015 21:09:01 DA(21424)-   ad break found 0:00-0:40 (0 - 40) Frames: 1 - 1001
27/09/2015 21:28:04 DA(21424)-   ad break found 17:46-23:17 (1066 - 1397) Frames: 26660 - 34922
27/09/2015 21:53:03 DA(21424)-   ad break found 43:23-48:34 (2603 - 2914) Frames: 65084 - 72850
27/09/2015 22:19:10 DA(21424)-   ad break found 69:37-74:48 (4177 - 4488) Frames: 104435 - 112197
27/09/2015 22:46:21 DA(21424)-   ad break found 96:20-101:52 (5780 - 6112) Frames: 144511 - 152791
27/09/2015 23:26:55 DA(21424)-   ad break found 137:14-142:25 (8234 - 8545) Frames: 205859 - 213629
27/09/2015 23:50:11 DA(21424)-   ad break found 160:24-165:34 (9624 - 9934) Frames: 240610 - 248362
28/09/2015 00:45:21 DA(21424)- /media/My Video/Kingdom of Heaven_20150927_2102.ts deleted
28/09/2015 00:45:21 DA(21424)- done...processed /media/My Video/Kingdom of Heaven_20150927_2102.ts in 13069.821s 03:37:50 - 7 ad breaks detected

I recovered the original recording from the deleted items folder; the duration is correct in this version.

MymsMan · Sep 28, 2015

This may be related to a problem with large file support in the libsndfile package which was recently identified by xyz321 and which should be fixed in the next version, it could also be that post-midnight Film4 couldn't sell enough advertising to fill a long enough ad-break for detection though it should have detected the end of recording break.

MontysEvilTwin · Sep 28, 2015

Probably the former, there was a missed ad break before the end.
Is xyz321 posting again? He hadn't been active on this forum for a while.

OwenMc · Sep 29, 2015

Today's manual detect ads are all failing for me with a memory allocation issue (Iwas set up to run traditional processing) - see below.

I was prompted for a package update first thing today, can't recall what it was though, but all yesterday's worked fine so I'm a tad suspicious....

Black Hole · Sep 29, 2015

IIRC there is a virtual memory extension available as a package.

MymsMan · Sep 29, 2015

OwenMc said:
Today's manual detect ads are all failing for me with a memory allocation issue (Iwas set up to run traditional processing) - see below.

I was prompted for a package update first thing today, can't recall what it was though, but all yesterday's worked fine so I'm a tad suspicious....

I have been out all day and only just got home but all of my recordings today have failed so I am also a tad suspicious that last night's update to libsndfile has not gone down too smoothly.

I will have time tomorrow to investigate properly - I probably need to recompile the Silence module with the latest version of libsndfile.

[DetectAds] Announcing DetectAds version 2

Forum Supporter

May contain traces of nut

The Dumb One

Well-Known Member

May contain traces of nut

Ad detector

Ad detector

Well-Known Member

Ad detector

Active Member

Ad detector

Well-Known Member

May contain traces of nut

The Dumb One

Well-Known Member

Ad detector

Well-Known Member

Forum Supporter

May contain traces of nut

Ad detector