[DetectAds] Announcing DetectAds version 2

[Trev]

Thanks for your comprehensive reply MymsMan. Whilst I was away, I thought about the Opt+ Crop thing and had pretty much come up with the same answer insofar as you would not normally bookmark the start for cropping. I have the recursive auto-shrink and auto-decrypt set 'just in case' even though I rarely watch recordings other than on the T2.

 

[Rob Singlehurst]

What is the failure reason?
This can be seen on the webif recording info display or with the hmt utility via telnet.

Watching a recording whilst it is being processed should not cause any problems.

I often get spurious recording failed messages when there is a slight overlap between the end of previous recordings and the start of the next recording.

I can't remember exactly but it was to do with a higher priority recording had caused this one to fail. This file was being processed because I checked in the webif shortly afterwards and the recording had the -crop suffix. Stopped the playback and then started again and we are watching the cropped one now with no ads!

 

[Dave F.]

MontysEvilTwin Thanks. Yes, it needed an 'update from the internet' & F5 cache clearance. Shame that can't be done in the background.

Some of the point below may have already been discussed:
Query:
Shouldn't 'Detect Adverts' in Browse>Options be greyed out for non decrypted programs?
I'm unsure how others will use this great add-on, but I can't see I'd ever need the bookmark at the beginning of the ad-break.
The wiki mentions suffixing file names but I see no evidence of that. Where is this viewable?
Requests:
As only one option can be selected, 'whilst recording in progress' & 'following auto-decryption' should be combined
A progress bar when analysing a recording.
Icon in Browse to denote it's been processed. (Similar to 'Dec' or 'DNLA')
Prevent redetection if it's already been performed on a file.
A 'Return' button on the Channel Exclusion page.
Add 'Chaserun' to 'whilst recording in progress' in settings to tie in with the wording on wiki page.
Fussy Request:
Display the 'done' times in H:M:S

If "whilst recording in progress" is set to Yes, does it automatically decrypt even if the file has not been set to decrypt directly? This question maybe irrelevant because I'm misunderstanding how Sweeper works - I assume Sweeper doesn't move a file until it's completed recording so as it remains in the top level folder auto decrypt can't be performed & therefore chaserun can not also operate.

Thanks to all who put time & effort into this package. It's a very useful one,

 

[Rob Singlehurst]

I'm using it with the "Crop recording following ad detection?" enabled so I'm not using the bookmarks.
The file name has the -crop suffix in the webif recordings browser so it's easy to tell which files have been processed. Presumably a log is kept so that files aren't processed more than once (they aren't anyway as far as I can tell)
I've just turned on verbose logging and check the detectads.log now and again to see what's happening.

 

[MymsMan]

MontysEvilTwin Thanks. Yes, it needed an 'update from the internet' & F5 cache clearance. Shame that can't be done in the background.

The auto-update package will keep all packages up to date and ensure you have a fairly current catalogue - it would be nice if just the catalogue was periodically updated for those who don't trust auto updating

Some of the point below may have already been discussed:
Query:
Shouldn't 'Detect Adverts' in Browse>Options be greyed out for non decrypted programs?

No, You can run DetectAds on encrypted programs and it will use the chaserun process to decrypt and process the file

I'm unsure how others will use this great add-on, but I can't see I'd ever need the bookmark at the beginning of the ad-break.

That is already on the to-do list

The wiki mentions suffixing file names but I see no evidence of that. Where is this viewable?

Suffixing applies when using chasreun or cropping after detection, it will be visible in the webif and in the program title on the standard TV media list

Requests:
As only one option can be selected, 'whilst recording in progress' & 'following auto-decryption' should be combined

As the second user to request this I suppose I should consider doing this! It was more convenient to keep them separate initially - they are independent it just would be a waste of processing to run detection twice.

A progress bar when analysing a recording.

I don't do any estimation of how long processing will take so have idea of whether processing is 25%, 50% complete, I would prefer users to use automatic detection or run analysis in the background than spend ages watching a slow moving status bar so have no plans to enhance the 'Run anlaysis; now option

Icon in Browse to denote it's been processed. (Similar to 'Dec' or 'DNLA')

Very difficult, I don't know if there is a precedent for adding a new flag to the hmt file and it would take a lot of coordination with other package developers. The -crop suffix to the filename provides a visual indication when a file has been processed and cropped.

Prevent redetection if it's already been performed on a file.

See above, The initial bookmarks display will show any previous bookmarks set by a previous run, No harm is done by rerunning and if I get around to allowing detection settings to be altered you will want to be able to compare the affect of any changes

A 'Return' button on the Channel Exclusion page.

Good idea, page was cloned from Channeldel which also doesn't have a return button.

Add 'Chaserun' to 'whilst recording in progress' in settings to tie in with the wording on wiki page.

Good idea

Fussy Request:
Display the 'done' times in H:M:S

Good idea, I am not good at mental division by 60

If "whilst recording in progress" is set to Yes, does it automatically decrypt even if the file has not been set to decrypt directly? This question maybe irrelevant because I'm misunderstanding how Sweeper works - I assume Sweeper doesn't move a file until it's completed recording so as it remains in the top level folder auto decrypt can't be performed & therefore chaserun can not also operate.

Not sure I fully understand question. Chaserun is independent of auto-decrypt and sweeper. Chaserun runs whilst the program is still recording and whilst the program is still in the series folder. Output from chaserun can be sent to a different folder or the same folder as the program - it will be eligible for sweeper processing once chaserun has completed.

 

[cdmackay]

one further request, please: it would be handy to be able to set a "recursive auto detect-ads" on a particular directory.

I don't want to do auto detection on all progs: the family users here will never get their head around bookmarks (sigh). but i might like to enable it for all *my* recordings, which are already auto-filed into my own dirs.

thanks much for considering...

 

[MontysEvilTwin]

MymsMan - Re. items in the above post (#25), I thought that the bookmarks at the beginning of each ad break were needed to allow cropping. During playback you don't need them. The ones at the very start and end aren't required for cropping or playback, but they shouldn't cause any problems.
There is a precedent for using the hmt file to store extra information re. the status of the file: the shrunk icon is appended if the letter 'E' is present at 073D. I'm undecided whether or not similar is required for a Detectads-processed recording.

 

[Black Hole]

one further request, please: it would be handy to be able to set a "recursive auto detect-ads" on a particular directory.

An alternative could be to have DetectAds as a Sweeper action.

 

[af123]

An alternative could be to have DetectAds as a Sweeper action.

There isn't currently a mechanism for packages to add new sweeper actions but it sounds like a good idea to create one.
As for conditions, there isn't current a 'has already been processed by detectads' but perhaps 'has no bookmarks' would suffice?

"If recording has no bookmarks, add to detectads queue."

I think something else would be needed to prevent things being processed more than once.

For anyone who is not using chaseget but still wants ads to be detected, then there is already an option to do the processing following decryption. That covers my use case - I'm not using chaseget because I don't want my Humax to stay on for three hours in fully awake mode every time it wakes up to do a recording but usually when the kids come in from school and watch TV for an hour everything that has been recorded is dealt with.

The Sweeper option would allow people to easily run this retrospectively on existing recordings or to pick and choose with more granularity than just channel name.

 

[MymsMan]

one further request, please: it would be handy to be able to set a "recursive auto detect-ads" on a particular directory.

I don't want to do auto detection on all progs: the family users here will never get their head around bookmarks (sigh). but i might like to enable it for all *my* recordings, which are already auto-filed into my own dirs.

thanks much for considering...

With automatic cropping of ads there is nothing for your family to get their heads around - you just start playing a recording, put the remote away and watch it without any ads and no need to press any buttons - Simples!

An alternative could be to have DetectAds as a Sweeper action.

As you know enhancements to Sweeper were suggested soon after the original DetectAds package came out but they never seemed to make it very far up the to-do list. It was frustration with the lack of progress with improvements to DetectAds &/or Sweeper that lead me to look into whether I could feasibly make the changes myself and now here we are with something substantially more advanced than my original modest goals!

I think having a command as a Sweeper action would be far more powerful and generally useful option than yet another recursive folder option, hopefully now that DetectAds has a command line interface it would be simpler to actually implement in Sweeper and the suggestion will meet with more favour.

 

[Trev]

A couple of further questions if I might make so bold.
My detectads settings are as in post #16

Automatically process recordings whilst recording in progress YES
Crop recording following ad detection? YES
Dustbin/delete original recording after processing? YES
Write output recording to = Same as input

Last night I switched the box to The Chase on ITV HD, halfway through.
As my third test, I switched the box to standby to see what would happen. The display went blank and the redring went red, indicating a recording in progress.

This morning, the box was out of standby with BBC RB selected.
I have the POC set to BBC1 HD and previously this worked OK with the box still in standby in the morning and when switched on, it is on BBC1 HD as expected

Q1 Presumably the ON and BBC RB is a result of detectads. Is this expected?

If so, surely this will defeat the purpose of putting it into standby when retiring any time you put the box into standby whilst detectads is processing.
I have noticed your comment about not being able to tell whether the box is being watched or not, but I put it into standby before retiring, so obviously I was not watching it. And also your last paragraph of post #11

On the other hand if you put the system into standby whilst it is processing an active recording using "process whilst recording" chaseget will attempt to turn the system back on to fully active so that it can continue to decrypt the recording.

Presumably this means that if you have 'Automatically process recordings whilst recording in progress YES' you will not be able to achieve standby for the night whilst recording.

Q2 Would it be possible to respect the user's 'standby' command when it has finished processing the current programme (or the queue) which would possibly also fix the BBC RB 'feature'?

The recording I made (third test) recorded OK with the uncropped file in the Deleted Items folder, but the W/I Media List new filename had '-Crop Done' after the name but before the date/time.
My second test that I did via 'Process Now' had'-crop' as a suffix just before the .ts extention (as I expected from what I read).

Q3 Is this by design?

 

[MymsMan]

MymsMan - Re. items in the above post (#25), I thought that the bookmarks at the beginning of each ad break were needed to allow cropping. During playback you don't need them. The ones at the very start and end aren't required for cropping or playback, but they shouldn't cause any problems.
There is a precedent for using the hmt file to store extra information re. the status of the file: the shrunk icon is appended if the letter 'E' is present at 073D. I'm undecided whether or not similar is required for a Detectads-processed recording.

With automatic cropping of ad breaks bookmarks aren't needed at all! DetectAds builds a list of -cut options which it passes on to nicesplice so any bookmarks are ignored - this also prevents any pre-existing manually added bookmarks from confusing the process. It still adds bookmarks to the input file in case for some reason you do need to go back to the uncropped original.

If you are not planning on cropping the start of break bookmarks are of no use to you so should be optional, even with a cropped file there is often unwanted material before the start of an ad break (typically those inane viewer 'competitions') that cant be detected by DetectAds but which you might wish to skip so it would be good to still have a bookmark at the end of the ad break.

 

[af123]

Q2 Would it be possible to respect the user's 'standby' command when it has finished processing the current programme (or the queue) which would possibly also fix the BBC RB 'feature'?

I could extend the ir package so that MymsMan can query if any physical remote button has been pressed since boot. That might give enough information to safely put the box back into standby after detectads/chaseget has done its thing.
Something like 'if lastboot reason was scheduled recording and no buttons have been pressed' might work.

 

[Trev]

Or perhaps. IF lastbutton=standby, THEN switch to standby? (after processing)
Bear in mind I know nothing about all this

 

[Dave F.]

MymsMan - Re. items in the above post (#25), I thought that the bookmarks at the beginning of each ad break were needed to allow cropping.

To clarify, I was processing pre-recorded programs & hadn't set them to crop as I was still unsure how detectads fully worked. When crop is used, my point is moot.

 

[MymsMan]

A couple of further questions if I might make so bold.
My detectads settings are as in post #16

Automatically process recordings whilst recording in progress YES
Crop recording following ad detection? YES
Dustbin/delete original recording after processing? YES
Write output recording to = Same as input

Last night I switched the box to The Chase on ITV HD, halfway through.
As my third test, I switched the box to standby to see what would happen. The display went blank and the redring went red, indicating a recording in progress.

This morning, the box was out of standby with BBC RB selected.
I have the POC set to BBC1 HD and previously this worked OK with the box still in standby in the morning and when switched on, it is on BBC1 HD as expected

Q1 Presumably the ON and BBC RB is a result of detectads. Is this expected?

If so, surely this will defeat the purpose of putting it into standby when retiring any time you put the box into standby whilst detectads is processing.
I have noticed your comment about not being able to tell whether the box is being watched or not, but I put it into standby before retiring, so obviously I was not watching it. And also your last paragraph of post #11 Presumably this means that if you have 'Automatically process recordings whilst recording in progress YES' you will not be able to achieve standby for the night whilst recording.

Q2 Would it be possible to respect the user's 'standby' command when it has finished processing the current programme (or the queue) which would possibly also fix the BBC RB 'feature'?

The recording I made (third test) recorded OK with the uncropped file in the Deleted Items folder, but the W/I Media List new filename had '-Crop Done' after the name but before the date/time.
My second test that I did via 'Process Now' had'-crop' as a suffix just before the .ts extention (as I expected from what I read).

Q3 Is this by design?

The current process of switching the box into full-awake mode and leaving it alive at the end of processing is far from ideal and I do want to improve it in the future. The new Idle time display has given me some ideas but I need to discuss them with AF123 since it would probably require some new System class services for optimum effect.

The system should have gone into Standby 3 hours after the end of The Chase but if this was one of the late night episodes this may have been too close to the 4:20 wake up and the Humax decided to stay awake.

'Crop done' should only appear in the Program Title and not in the file name, -crop appears in the file name. The title does become part of the file name if you use the remote control to edit the program title.

 

[Dave F.]

MontysEvilTwin Thanks. Yes, it needed an 'update from the internet' & F5 cache clearance. Shame that can't be done in the background.
The auto-update package will keep all packages up to date and ensure you have a fairly current catalogue - it would be nice if just the catalogue was periodically updated for those who don't trust auto updating

Hmm... I have auto-update installed. When did you upload the latest version? It was installing v0.0.2-2 yesterday PM.

Some of the point below may have already been discussed:
Query:
Shouldn't 'Detect Adverts' in Browse>Options be greyed out for non decrypted programs?
No, You can run detecads on encrypted programs and it will use the chaserun process to decrypt and process the file

Ah, I see. I read "DetectAds can be run in this mode triggered from auto when auto-decryption completes." & assumed it was for all processes. Could this be clarified in the 'Chaserun' section of the wiki for dullards like me who are a bit slow on the uptake?

I'm unsure how others will use this great add-on, but I can't see I'd ever need the bookmark at the beginning of the ad-break.
That is already on the to-do list

Please see my previous message.

A progress bar when analysing a recording.
I don't do any estimation of how long processing will take so have idea of whether processing is 25%, 50% complete, I would prefer users to use automatic detection or run analysis in the background than spend ages watching a slow moving status bar so have no plans to enhance the 'Run anlaysis; now option

...The initial bookmarks display will show any previous bookmarks set by a previous run, No harm is done by rerunning and if I get around to allowing detection settings to be altered you will want to be able to compare the affect of any changes

I understand. It will be a bit pointless after users have processed existing programs

 

[MontysEvilTwin]

I had set up a unit to automatically process while recording, with bookmarks only (no crop). I was recording two programmes at the same time. They had similar start times, but one was 30 minutes long, the other an hour. Chaserun was working fine, processing both recordings at the same time but when the 30 minute programme finished the '-Dec' file being created for the hour programme was indexed by the DLNA server. This stopped the rest of the recording from being appended to the '-Dec' file. I read that it is known that DLNA indexing can interfere with the process. I wonder if it is possible to have an option in the settings menu to limit chaserun to one analysis at a time?

 

[Dave F.]

I performed some 'Chaserun' detection overnight, with the unit in stand-by It didn't go well:



Both Frasier & Lorraine failed to process:


Detectads.log:

37    03/09/2015 09:06:16 DA(1340)- done...processed /media/My Video/Frasier_20150903_0901.ts in 0.362s - 0 ad breaks detected
36    03/09/2015 09:06:16 DA(1340)- Incomplete data retrieval 196157440 bytes missing
35    03/09/2015 09:01:15 DA(1340)- ==DETECTADS Chase Run: /media/My Video/Frasier_20150903_0901.ts
34    03/09/2015 09:01:15 RM(1334)-   DETECTADS: Started /media/My Video/Frasier_20150903_0901.ts for chaserun advert detection
33    03/09/2015 09:01:15 RM(1334)-   DETECTADS: Checking /media/My Video/Frasier_20150903_0901.ts (Channel 4 HD) for channel exclusion
32    03/09/2015 08:34:33 DA(1207)- done...processed /media/My Video/Lorraine_20150903_0829.ts in 0.451s - 0 ad breaks detected
31    03/09/2015 08:34:33 DA(1207)- Incomplete data retrieval 198893568 bytes missing
30    03/09/2015 08:29:34 DA(1207)- ==DETECTADS Chase Run: /media/My Video/Lorraine_20150903_0829.ts
29    03/09/2015 08:29:34 RM(1204)-   DETECTADS: Started /media/My Video/Lorraine_20150903_0829.ts for chaserun advert detection
28    03/09/2015 08:29:34 RM(1204)-   DETECTADS: Checking /media/My Video/Lorraine_20150903_0829.ts (ITV HD) for channel exclusion

Any ideas?

 

[MymsMan]

I had set up a unit to automatically process while recording, with bookmarks only (no crop). I was recording two programmes at the same time. They had similar start times, but one was 30 minutes long, the other an hour. Chaserun was working fine, processing both recordings at the same time but when the 30 minute programme finished the '-Dec' file being created for the hour programme was indexed by the DLNA server. This stopped the rest of the recording from being appended to the '-Dec' file. I read that it is known that DLNA indexing can interfere with the process. I wonder if it is possible to have an option in the settings menu to limit chaserun to one analysis at a time?

I haven't seen that happen before, indexing of the the -dec file shouldn't impact anything. The problem I was referring to is when the input file is DLNA indexed whilst chaseget is still trying to read it.

I will try to recreate the problem but do you have the detectads log file for the period.