[DetectAds] Announcing DetectAds version 2

[Trev]

Yep. Typo and mild dyslexia for numbers. 21.5MB
Copy/pasted

#idletime { display: none; }

into a new line of /mod/webif/html/css/EXTEA.css
Reboot
Idle: still there

 

[prpr]

Reboot what and why? All you need to do is refresh the page in the browser. Of course, you need call it EXTRA.css not EXTEA.css, unless that is another typo.
I just tried it and it works instantly for me.

 

[MymsMan]

EXTRA.css is is in the Commonly edited files list on the Diagnostics File editor page so no need to mount disks or hunt through directories

 

[Trev]

Reboot what and why?

The T2 because the message didn't disappear like magic. So tried a reboot. Is that such heinous crime that leads to a later execution or hang drawing and quartering?

Of course, you need call it EXTRA.css not EXTEA.css, unless that is another typo.

Corse it are.

 

[MontysEvilTwin]

Try clearing your browser cache.

 

[MymsMan]

That one slipped through the cracks. There are hooks that the auto processing use so I'll look at generalising them a bit so detectads can use them too.

Af123 has now generalised the status reporting interface (it took quite a few update webif updates to iron out all the wrinkles) so I have just updated DetectAds to version 0.2.3-3 and ChaseGet to version 0.1.1-6.

The webif status display should now show when DetectAds is active rather than a misleading playing indication, It will also show whether ChaseGet is actively running against the input file but the ChaseGet status can be turned of in the settings menu if you don't find it useful.

 

[af123]

I'm replying to a post that was in the web interface thread here rather than continuing the topic drift (that I started, sorry):

My thoughts were to reduce button clutter by grouping all of the folder related activities together under the folder OPT+ option and have just the list of files related buttons at the bottom, you are already in the file list at this stage so whether Dedup is at the top or bottom makes little difference to usability

Two of my pet hates are the plethora of series folders and the myriad of Auto-dothis & Recursive-dothat flags. Before FlatView came along I was a happy Flatten user and often consider returning to that simple life. When I took on DetectAds it was a design goal to avoid the need to set a Detect this folder flag every time I scheduled a new series and opted for the Channel exclusion rules instead.

However as a software provider I do recognize that people use their boxes in different ways and that sometimes you need to grit your teeth and provide something you will never use however I will draw the line at Recursive Ad detection since that would inevitably lead to a call for a No Ad detection flag and the OPT+ list grows ever longer

It's interesting to see the different ways that people use their boxes : )
Series folders suit me because I also use the tvdb integration to automatically rename episodes (as you'll see in the screenshot I posted in https://hummy.tv/forum/threads/webif-web-interface-1-3-4.7619/page-3#post-104051).

Detectads is a fantastic package that I use a fair amount but I don't have it enabled to run automatically in general. I selectively choose files to process - usually films that I'm going to keep - or I set up sweeper rules against specific series folders where the number or content of adverts is particularly annoying. I've seen other people post that they sometimes write temporary sweeper rules and run them, just to queue all files in a directory for processing.

So, I think there are three usage patterns here:

1. Always detect adverts in recordings on certain channels (whether that's during recording (chaserun) or following decryption);
2. Selectively detect adverts in manually chosen recordings;
3. Always detect adverts for particular series.

(1) is there and is the simplest set-and-forget option. Doesn't suit me - doubt I could pin down exactly why.
(2) can be done through OPT+->Detect Ads and manually queue. It would be nice to be able to select multiple files and submit them all at once through the new queue mechanism. I've also used sweeper in the past to do this via a temporary rule on a folder.
(3) Can be done with a sweeper rule against the series folder. I only have this set up for a few series but it would be nice to replace that with a folder flag. I'm not suggesting recursive automatic ad-detection, that's covered well enough by (1).

The patches to add support for the new queuing system and for a folder flag are pretty small. I'll send them to you for consideration for a future release in some form or another but I can always keep applying them to my local installation : )

 

[Trev]

Being a lazy sod, I use (1) above.
Need a poll on how people use it?

 

[bixieupnorth]

after enjoying detectads for several months i have noticed that it seems to have ceased to function at all, i haven't fiddled with any of the settings, previously it had been on the chaserun setting and seemed to work very well
any ideas/solutions gratefully accepted thanks

 

[af123]

Is content sharing still turned on in the Humax menus? It has been known to turn itself off and this will stop chaserun working.
Otherwise, have a look in the auto.log file for clues.

 

[bixieupnorth]

sorry but where would i find out about content sharing??

 

[bixieupnorth]

checking the detectads log it does indeed say content sharing disabled, so how do i reshare it please? i'm amazed at what goes on in the background

also since i bought a new pc i have been unable to watch content from pc, the log for that says permission denied, does that sound like i have wrongly added my log in details?

 

[af123]

Standard on-TV menu, it's under Internet Settings I think.

 

[prpr]

checking the detectads log it does indeed say content sharing disabled, so how do i reshare it please?

Menu, Settings, System, Internet Setting

 

[bixieupnorth]

thanks, i'll have a look

 

[bixieupnorth]

yes, that was it thanks, all working well now

 

[MymsMan]

DetectAds 02.4-1 is now available on the package repository, I will update the wiki page in the next few days

• It now uses the system queue for all deferred requests rather than maintaining its own database
• The old queue will be processed if there are outstanding requests on it but no new entries will be added and it will eventually be deleted.
• You can add Ad Detection requests to the queue using the Browse and Sweeper 'Queue for' options
• The old DetectAds Sweeper actions remain for compatibility with existing rules but now use the system queue
• The sample Sweeper rule has been updated to use the 'Queue recording for' action
• There are two new folder flags Auto DetectAds and No DetectAds
  • These work in conjunction with the existing Channel exclusions rules so it is not necessary to add the flag to each directory if you use the existing DetctAds Auto processing options (Automatically process whilst recording in progress? (chaserun) or Automatically process recording following auto-decryption? (traditional) )
  • You can use these flags to include additional directories (Auto DetectAds) where you want all (including pre-existing recordings) ad detected or exclude directories (No DetectAds) which would otherwise be subject to ad detection based on channel filters.
  • For instance you could set No DetectAds on the 'My Videos' root folder to prevent auto ad detection of one-off recordings (I don't recommend setting Auto DetectAds on 'My Videos' since this would override the channel selection filters and cause detection of all one-off recordings including those on BBC channels)
• Ad Detection is slow so be be sparing when adding batches of existing recordings to the queue
• DetectAds processing can still be deferred to, for example overnight, by setting a target start time on the settings page (remember to leave machine on or set a reminder)
• Logging for Queued requests will now be in auto.log but remain in detectads.log for detect while recording. Chaseget logging will be in chaseget.log.
• Chaserun processing remains a process while recording operation though it does create a temporary queue entry as a safeguard against failures during recording.

The Auto DetectAds flag means all recordings in a folder will be processed regardless of the recording channel.
The No DetectAds flag means none of the recordings in a folder will be processed regardless of the recording channel.
In the absence of a flag recordings will be processed during recording (chaserun) or following decryption (traditional) if they are on a channel that has not been excluded by the channel filters.

If you routinely want to detect ads in all commercial programmes as I do you don't need to use the new flags at all.
If you only want a subset of programmes processed you can use the flags to include/exclude directories overriding the channel filters

The processing is (deliberately) different from the other auto options where you have to have a flag set before any processing happens.

 

[MontysEvilTwin]

I had an issue with Detectads I'd not seen before. Log entries below:

10/03/2017 21:02:13 RM(22116)-   DETECTADS: Checking /media/My Video/Lethal Weapon/New_ Lethal Weapon_20170310_2102.ts (ITV HD) for channel exclusion
10/03/2017 21:02:14 RM(22116)-   DETECTADS: Started /media/My Video/Lethal Weapon/New_ Lethal Weapon_20170310_2102.ts for chaserun advert detection
10/03/2017 21:02:14 DA(22148)- ==DETECTADS Chase Run: /media/My Video/Lethal Weapon/New_ Lethal Weapon_20170310_2102.ts
10/03/2017 21:07:45 DA(22148)-   ad break found 0:00-0:28 (0 - 28) Frames: 1 - 705
10/03/2017 21:20:44 DA(22148)-   ad break found 12:17-16:35 (737 - 995) Frames: 18417 - 24880
10/03/2017 21:33:18 DA(22148)-   ad break found 24:49-29:07 (1489 - 1747) Frames: 37233 - 43680
10/03/2017 21:48:33 DA(22148)-   ad break found 40:10-44:29 (2410 - 2669) Frames: 60257 - 66723
10/03/2017 22:00:02 DA(22148)-   ad break found 55:11-57:21 (3311 - 3441) Frames: 82779 - 86030
Process 24634 found
10/03/2017 22:00:02 DA(22148)- Final bookmarks: 28 737 995 1489 1747 2410 2669 3311 3441
10/03/2017 22:00:04 RM(5029)-   DETECTADS: Checking /media/My Video/The Team/New_ The Team_20170310_2159.ts (More4+1) for channel exclusion
10/03/2017 22:00:04 DA(22148)- /media/My Video/Lethal Weapon/New_ Lethal Weapon_20170310_2102.ts deleted
10/03/2017 22:00:04 RM(5029)-   DETECTADS: Started /media/My Video/The Team/New_ The Team_20170310_2159.ts for chaserun advert detection
10/03/2017 22:00:05 DA(5090)- ==DETECTADS Chase Run: /media/My Video/The Team/New_ The Team_20170310_2159.ts
Removed /mod/tmp/New_ Lethal Weapon_20170310_2102-inp.ts<br>
Removed /mod/tmp/New_ Lethal Weapon_20170310_2102-inp.nts<br>
Removed /mod/tmp/New_ Lethal Weapon_20170310_2102-inp.hmt<br>
10/03/2017 22:04:50 DA(5090)-   ad break found 0:00-2:34 (0 - 154) Frames: 1 - 3848
10/03/2017 22:25:45 DA(5090)-   ad break found 19:02-24:03 (1142 - 1443) Frames: 28556 - 36066
10/03/2017 22:45:22 DA(5090)-   ad break found 39:13-43:52 (2353 - 2632) Frames: 58828 - 65798
10/03/2017 22:59:30 DA(5090)-   ad break found 53:16-57:59 (3196 - 3479) Frames: 79902 - 86969
10/03/2017 23:15:11 DA(5090)-   ad break found 68:20-73:19 (4100 - 4399) Frames: 102506 - 109985
10/03/2017 23:20:10 DA(5090)- Final bookmarks: 154 1142 1443 2353 2632 3196 3479 4100 4399
Process 7516 found
10/03/2017 23:20:12 DA(5090)- /media/My Video/The Team/New_ The Team_20170310_2159.ts deleted
10/03/2017 23:20:14 DA(5090)- done...processed /media/My Video/The Team/New_ The Team_20170310_2159.ts in 4637.41s 01:17:17 - 5 ad breaks detected
Removed /mod/tmp/New_ The Team_20170310_2159-inp.ts<br>
Removed /mod/tmp/New_ The Team_20170310_2159-inp.nts<br>
Removed /mod/tmp/New_ The Team_20170310_2159-inp.hmt<br>

There were two overlapping recordings (Lethal Weapon and The Team). Lethal Weapon could not be found in the series folder, in 'My Video' or in 'mod/tmp'. As far as I can tell it detected all the ad breaks, deleted the 'inp' files and deleted the original recording from the series folder. However, it did not replace it with the processed version. There were no 'Dec' files left in '/mod/tmp' but no log entry to say that these files had been deleted. I managed to retrieve the recording from the [Deleted Items] bin. The hmt file had been marked as ad detected, and contained the ad break bookmarks. However this file was not decrypted. Detectads must have replaced the hmt file in the original recording with the ad detected version before deleting it.

 

[MymsMan]

There were two overlapping recordings (Lethal Weapon and The Team). Lethal Weapon could not be found in the series folder, in 'My Video' or in 'mod/tmp'. As far as I can tell it detected all the ad breaks, deleted the 'inp' files and deleted the original recording from the series folder. However, it did not replace it with the processed version. There were no 'Dec' files left in '/mod/tmp' but no log entry to say that these files had been deleted. I managed to retrieve the recording from the [Deleted Items] bin. The hmt file had been marked as ad detected, and contained the ad break bookmarks. However this file was not decrypted. Detectads must have replaced the hmt file in the original recording with the ad detected version before deleting it.

That is strange and unfortunate

There has been no (intentional) change in processing the files at the end of recording in quite a while so I don't know what is going wrong.
Do you have Sweeper or anything else renaming files (e.g. to remove the New prefix)?
Does the auto.log for the period contain any clues?

 

[MontysEvilTwin]

That is strange and unfortunate

There has been no (intentional) change in processing the files at the end of recording in quite a while so I don't know what is going wrong.
Do you have Sweeper or anything else renaming files (e.g. to remove the New prefix)?
Does the auto.log for the period contain any clues?

I do use Sweeper to remove the 'New_' prefix but there is no log entry I can find to indicate that this ran. The only pertinent entries I can find in auto.log are:

10/03/2017 23:20:05 - De-queuing 80 - detectads - /mnt/hd2/My Video/Lethal Weapon/New_ Lethal Weapon_20170310_2102.ts
10/03/2017 23:20:05 - ts load failed.

It is probably impossible to debug without more information.