[DetectAds] Announcing DetectAds version 2

[peterworks]

The same scenario has happened again - not queued for shrinking. I had set the logging to 'debugging information' so, if you think it worth progressing, I have attached auto log

Could it possibly be the same as the problem I was having ?

 

[MymsMan]

Could it possibly be the same as the problem I was having ?

No, this problem is with chaserun processing during recording.

Chaserun processing creates a temporary queue entry as a backup in case of failure during processing

This failure occurred at the very end of processing after the input file was deleted as the -crop file was being renamed to the input name.
What I don't understand at the moment is why the -crop file wasn't left behind if the rename failed

 

[Ian Manning]

I'm a newcomer to T2 custom firmware, and am trying out DetectAd. I have only attempted 2 ITV recordings since selecting "
Automatically process whilst recording in progress? (chaserun)", and both have failed with messages similar to the one below:

03/04/2017 22:15:12 - Retry limit exceded for /mnt/hd2/My Video/New_ Inside The Tube_ Going____20170403_2126.ts cant process file
03/04/2017 22:45:13 - DetectAds /mnt/hd2/My Video/Nightly Show with Bradley Walsh/New_ Nightly Show with Bradley Walsh_20170403_2202.ts Incomplete data retrieval 1006481408 bytes missing (00:27:39)
03/04/2017 22:15:12 - DetectAds /mnt/hd2/My Video/New_ Inside The Tube_ Going____20170403_2126.ts Incomplete data retrieval 1425735680 bytes missing (00:33:22)
03/04/2017 22:45:13 - Retry limit exceded for /mnt/hd2/My Video/Nightly Show with Bradley Walsh/New_ Nightly Show with Bradley Walsh_20170403_2202.ts cant process file

Any ideas for how to diagnose this?

 

[MymsMan]

Can you turn on "Auto-processing log level Debugging information" on the settings page
Then can you post or send me via PM the detectads.log, chaseget.log and auto.log files for the period that covers a failing recording

 

[Ian Manning]

Can you turn on "Auto-processing log level Debugging information" on the settings page
Then can you post or send me via PM the detectads.log, chaseget.log and auto.log files for the period that covers a failing recording

Will do. Where are those logs usually stored?

 

[MymsMan]

Most logs are in the /mod/tmp/ directory but they can be viewed and downloaded on the Diagnostics webif page.

 

[renzz]

I've finally decided to give Detectads a go (not sure why I've not done so before really), and I've read through the comprehensive, if slightly overwhelming, wiki and a large part of this thread, but I have one question that I can't find an answer to.

If I use chaserun AND want to chaseplay a recording with ads removed, is my only option to play the -crop file with the ads removed? Is there no way to chaseplay the recording with bookmarks marking the ads instead? I'm only thinking about the scenario where Detectads erroneously removes part of the programme (although I understand this is rare) - with the -crop file there is no way to skip back or forward to view something that has been removed as an ad, but with bookmarks you obviously can.

Forgive me if this has been answered before, but I've only been playing with this for a few days and so far I can't see a way to do this.

 

[MymsMan]

If I use chaserun AND want to chaseplay a recording with ads removed, is my only option to play the -crop file with the ads removed? Is there no way to chaseplay the recording with bookmarks marking the ads instead?

Unfortunately not.
Whilst the programme is recording the Humax is frequently updating the hmt file with the current recording progress information and this updating overwrites any attempts by DetectAds to update the bookmark during recording. It can only add the bookmarks when the Humax is not recording or playing the file,

 

[Trev]

I find that the 'safest' method of using detectads is just to detect the end of the ads with no crop then use 'jump to bookmark' to skip the ads when you replay later. This doesn't directly answer your question (MymsMan did that above) but it ensures that the crop does not erroneously crop a wanted portion of the recording, thus necessitating a hunt through deleted items and then a hunt through to find where you were. I must admit, when I used the crop feature, I only got a couple of erroneous crops, but that was when HWMBO was watching, and she got peed off when I had to hunt for where we were at.

 

[Ian Manning]

I have the instant replay set to 15 seconds, but when I try to use it on a recording which has been cropped to remove ads it skips back to near the the start of the recording (regardless of how far into the recording I was). Is this a known issue?

 

[MymsMan]

Is this a known issue?

Yes and No,
I haven't experienced skipping back to the beginning but I have seen, occasionally, problems when fast forwarding across a cropped ad break.
I think nicesplice can introduce discontinuities into the nts file when cropping which can sometimes cause errors when skipping but I haven't investigated in detail.

Skipping is especially unreliable when chase playing a -crop file since the Humax only see the portion of the .nts that exists when the play back starts.

 

[Ian Manning]

Yes and No,
I haven't experienced skipping back to the beginning but I have seen, occasionally, problems when fast forwarding across a cropped ad break.
I think nicesplice can introduce discontinuities into the nts file when cropping which can sometimes cause errors when skipping but I haven't investigated in detail.

Skipping is especially unreliable when chase playing a -crop file since the Humax only see the portion of the .nts that exists when the play back starts.

This happens when playing a completed recording, rather than chase play. It seems to happen alot, but I'm not sure if it happens on every recording. I'll keep an eye on it...

 

[Trev]

Then don't crop. use the 'bookmark end of ads' option and the 'Jump to bookmark' facility to skip ads. The 'skip back' 15 seconds works fine for me using that setup.
But that doesn't solve the possible 'problem' with the crop that you have 'found'.

 

[Ian Manning]

So I stopped cropping, and tried using the "Only Bookmark ends of ad breaks" option. It doesn't seem to insert any bookmarks at all. Other settings are:
Automatically process whilst recording in progress? (chaserun) YES
Start processing recording queue ASAP
Only Bookmark ends of ad breaks? YES
Crop recording following ad detection? NO
Delete (move to bin) original recording after processing? YES
Write output recording to folder: Same as input

Any idea why it might not be creating bookmarks?

 

[Ian Manning]

Here's the log output:
4970 02/05/2017 20:01:42 RM(31498)- Recmon: -delete {/media/My Video/ [FlatView]/New_ Save Money_ Good Food_20170502_1931}
4969 02/05/2017 20:00:45 RM(30309)- Recmon: -move {/media/My Video/Save Money_ Good Food/New_ Save Money_ Good Food_20170502_1931-dec}
4968 02/05/2017 20:00:44 RM(30306)- Recmon: -move {/media/My Video/Save Money_ Good Food/New_ Save Money_ Good Food_20170502_1931}
4967 02/05/2017 20:00:44 RM(30252)- Recmon: -move {/media/My Video/Save Money_ Good Food/New_ Save Money_ Good Food_20170502_1931-dec}
4966 Removed /mod/tmp/New_ Save Money_ Good Food_20170502_1931-inp.hmt<br>
4965 Removed /mod/tmp/New_ Save Money_ Good Food_20170502_1931-inp.nts<br>
4964 Removed /mod/tmp/New_ Save Money_ Good Food_20170502_1931-inp.ts<br>
4963 02/05/2017 20:00:43 DA(28086)- =============================================================
4962 02/05/2017 20:00:43 DA(28086)- done...processed /media/My Video/Save Money_ Good Food/New_ Save Money_ Good Food_20170502_1931.ts in 1583.307s 00:26:23 - 2 ad breaks detected
4961 02/05/2017 20:00:43 DA(28086)- Removing item 77 (/media/My Video/Save Money_ Good Food/New_ Save Money_ Good Food_20170502_1931.ts) from the queue
4960 02/05/2017 20:00:41 RM(30221)- Recmon: -move {/media/My Video/Save Money_ Good Food/New_ Save Money_ Good Food_20170502_1931}
4959 02/05/2017 20:00:41 DA(28086)- /media/My Video/Save Money_ Good Food/New_ Save Money_ Good Food_20170502_1931.ts deleted
4958 02/05/2017 20:00:40 DA(28086)- -cut 0 340 -cut 15060 17380
4957 02/05/2017 20:00:40 DA(28086)- Final bookmarks: 34 1738
4956 02/05/2017 20:00:40 DA(28086)- After process loop
4955 02/05/2017 20:00:40 DA(28086)- End: Frame 43460 Time: 28:58 Seconds: 1738
4954 02/05/2017 20:00:40 DA(28086)- status@= End 37659- 43460 ( 25:06- 28:58), 5802 ( 3:52.1), 5801 ( 3:52), [ 0]
4953 02/05/2017 20:00:40 DA(28086)- ad break found 25:06-28:58 (1506 - 1738) Frames: 37660 - 43460
4952 02/05/2017 20:00:40 DA(28086)- cut@= Cut 37660- 43460 ( 25:06- 28:58), 5801 ( 3:52.0), 0 ( 0:00), [ 0]
4951 02/05/2017 20:00:40 DA(28086)- info@> Cluster 37660- 43460 ( 25:06- 28:58), 5801 ( 3:52.0), 15690 ( 10:28), [ 11]

 

[MymsMan]

Any idea why it might not be creating bookmarks?

Not really,
According to the log the only bookmarks would be at 34 seconds and the very end, no middle ad break was found
Were those bookmarks created?

Ad breaks sometimes are not found if they are too short or don't contain sufficient ads

 

[Ian Manning]

Not really,
According to the log the only bookmarks would be at 34 seconds and the very end, no middle ad break was found
Were those bookmarks created?

Ad breaks sometimes are not found if they are too short or don't contain sufficient ads

Yes there are bookmarks before and after the start of the show. I've also just noticed that the ad break in the middle consisted only of ads for 2 other ITV programmes (just over a minute in total) - i.e. no "real" ads as such. Presumably that's why an ad break wasn't detected? How are ads usually identified by DetectAds?

 

[MymsMan]

DetectAds detects the silences between ads, trailers often don't have the silences so are not detected as separate from the program.
To be recognised as an ad break there must be a minimum number of silences and a minimum break length

 

[Black Hole]

Yes there are bookmarks before and after the start of the show.

They could be from arbookmarks. Do you have that installed?

 

[Ian Manning]

DetectAds detects the silences between ads, trailers often don't have the silences so are not detected as separate from the program.
To be recognised as an ad break there must be a minimum number of silences and a minimum break length

Interesting. Does that mean that silences in the show itself might be incorrectly interpreted as ads?

I don't have arbookmarks installed - should I have?