Crop fails on manual use of Detect Adverts

[paultry]

My first attempted use of Detect Adverts. It failed on one film in the same way, so I'm trying it on a second film. They are SD films stored on the internal HDD in /media/My Video.

Using Web-IF Browse, I use OPT+ Decrypt - which works, then OPT+ Detect Adverts, which successfully creates the bookmarks. I check bookmark location in the HDR UI by using the remote to play, skip through the bookmarks, then stop playback, and press the remote media button to exit. I return to Web-IF browse and use OPT+ crop. On that screen I see the correct portions in red to be cropped out and press "perform crop operation" (with "save new bookmarks?" ON). I get "file cropping complete". Back in the Web-IF browse the file is the same size, and in the HDR UI it still has the adverts (and bookmarks) OPT+ crop is not greyed out. The original file has been automatically moved to an "original" (or similar) folder and is the same length, size and has adverts.

Sharing is on. In case it is relevant, in the Web-IF settings for detectads package I have the radio button "Folder flag, Sweeper or No automatic processing of recordings" selected. There's 17% of internal HDD empty. I've rebooted (i.e. button on standby, HDD parked, switched off at back, then back on) and tried OPT+ crop again but it's the same result.

Please, what on earth am I doing wrong?

 

[Black Hole]

I get "file cropping complete".

Is that after a reasonable time for processing, or immediately?

Assuming the latter, I have seen processing errors like this (other situations) and IIRC it's the result of some bug, but trying to find the relevant report might be tricky...

 

[Black Hole]

I think you should be warned: I realise you're toying with these facilities for the first time, but "crop" (for example) is a blunt instrument. Sure, it cuts&splices the video file at strategic points (i-frames), but it does not patch up the .nts sidecar file – which contains time index information for the HDR-FOX transport controls.

That said, it is possible to reconstruct the sidecar files subsequently.

 

[paultry]

On no attempt did it take more than a few seconds IIRC - I assumed that finding the ads was the hard bit, so it didn't set off alarm bells.

I guess I might try to do it in a non-manual way, as a workaround. I bought specifically the HDR a long time ago because I'd read about the rather extraordinary CF - and damn it, when I finally install it I can't get one of the best features working! Weird, when others use it routinely.

 

[paultry]

I think you should be warned: I realise you're toying with these facilities for the first time, but "crop" (for example) is a blunt instrument. Sure, it cuts&splices the video file at strategic points (i-frames), but it does not patch up the .nts sidecar file – which contains time index information for the HDR-FOX transport controls.

That said, it is possible to reconstruct the sidecar files subsequently.

I'm geting confused now. I thought a standard use case for detectads was to crop the adverts out - and that many users just let it do it automatically. Is cropping not common?

And I have no idea what the implications are of those other consequences!

 

[Black Hole]

Weird, when others use it routinely.

What we're actually doing is running ad-detection on-the-fly, during recording and not as a separate process, and then just setting a post-advert bookmark for manual skip.

WebIF >> Settings >> Setttings for detectads package

 

[Black Hole]

I thought a standard use case for detectads was to crop the adverts out - and that many users just let it do it automatically. Is cropping not common?

I'm not sure what the common usage is – the tools were created by the various enthusiasts on the back of the CF framework and released on a take-it-or-leave-it basis. Playback of cropped files is sometimes problematic, so I know a fair number of users gave up with cropping and just use the bookmarks for skip.

This is a side-issue however. The basic manual crop operation should work, and we should be able to fix it.

 

[paultry]

Ah, OK,

What we're actually doing is running ad-detection on-the-fly, during recording and not as a separate process, and then just setting a post-advert bookmark for manual skip.

WebIF >> Settings >> Setttings for detectads package

I read that on-the-fly might stretch resources so I opted for a more vanilla approach. I can run detectads overnight. Bookmarking the adverts is a huge luxury, I just liked the idea of a bit more disk space as well - but as has been noted previously here, probaly by you, that's more of an inner journey.

 

[Black Hole]

Shot in the dark... do you have swapper installed? It helps when demands on RAM exceed what's available.

Belay that: swapper is default.

 

[Luke]

On no attempt did it take more than a few seconds IIRC - I assumed that finding the ads was the hard bit, so it didn't set off alarm bells.

When opening OPT+ Crop it gives an estimated time for how long the crop operation will take. This is usually fairly accurate for all types of recording (i.e. HD, SD and audio).

When running crop it displays a progress bar. Are you saying that is not being displayed?

I return to Web-IF browse and use OPT+ crop. On that screen I see the correct portions in red to be cropped out and press "perform crop operation" (with "save new bookmarks?" ON). I get "file cropping complete". Back in the Web-IF browse the file is the same size, and in the HDR UI it still has the adverts (and bookmarks) OPT+ crop is not greyed out.

Depending on the sort order you have set the file position may move position within the list of files. E.g. if you are viewing by "reverse date" and crop the file which is positioned at the bottom of the list, after cropping it will appear at the top of the list on the web-if. Are you sure you are looking at the correct file on the web-if?

I return to Web-IF browse and use OPT+ crop. On that screen I see the correct portions in red to be cropped out and press "perform crop operation" (with "save new bookmarks?" ON). I get "file cropping complete". Back in the Web-IF browse the file is the same size, and in the HDR UI it still has the adverts (and bookmarks) OPT+ crop is not greyed out.

Even when working as expected, it will usually not be greyed out as cropping with "save new bookmarks" will result in the cropped file having bookmarks and that option is only greyed out if there are no bookmarks.

 

[paultry]

When opening OPT+ Crop it gives an estimated time for how long the crop operation will take. This is usually fairly accurate for all types of recording (i.e. HD, SD and audio).
View attachment 7261
When running crop it displays a progress bar. Are you saying that is not being displayed?

View attachment 7262

Depending on the sort order you have set the file position may move position within the list of files. E.g. if you are viewing by "reverse date" and crop the file which is positioned at the bottom of the list, after cropping it will appear at the top of the list on the web-if. Are you sure you are looking at the correct file on the web-if?

Even when working as expected, it will usually not be greyed out as cropping with "save new bookmarks" will result in the cropped file having bookmarks and that option is only greyed out if there are no bookmarks.

No, that progress bar never shows, the beige message box (in your image "please do not interrupt...) flashes up with I think a bit more text than that, but too quick to read. It says something about the crop being complete half way up the page. In order to check if this behaviour happens the first time I run crop on a file, I decrypyted another file, ran AdDetect on it, edited the bookmarks until they made sense (at the end of the recording there were too many), checked it with playback and the ran crop from Web-IF. Still the same behaviour.

I am definitely checking the right files afterwards.

 

[Black Hole]

No, that progress bar never shows, the beige message box (in your image "please do not interrupt...) flashes up with I think a bit more text than that, but too quick to read. It says something about the crop being complete half way up the page. In order to check if this behaviour happens the first time I run crop on a file, I decrypyted another file, ran AdDetect on it, edited the bookmarks until they made sense (at the end of the recording there were too many), checked it with playback and the ran crop from Web-IF. Still the same behaviour.

As I mentioned up-thread I have definitely seen this kind of thing before.

 

[Black Hole]

...but I can't find it.

Try something very simple: make a short recording, decrypt it, set a couple of bookmarks, crop it. Document and report every stage so we can see the details – things you might not consider relevant. Even an unusual character in the name of a recording can trigger a bug.

 

[paultry]

...but I can't find it.

Try something very simple: make a short recording, decrypt it, set a couple of bookmarks, crop it. Document and report every stage so we can see the details – things you might not consider relevant. Even an unusual character in the name of a recording can trigger a bug.

I have just reinstalled detectads nicesplice and then tried two more files (2 parts of the same film, sometimes HDR does that when there's news etc in the middle). After reinstall I decrypted, ran detectads, checked bookmark positions on the UI, deleted a couple (this is not true for all the previous files, I left some files as is) and then got the same behaviour on trying to crop.

I thought I was pretty thorough in my original post, I don't know what the further details are that would help. Are there logs that could be useful? I've already had a sift through them but without direction nothing stands out. The filenames are as they came off the TV, and it's at least 4 different files this has failed on - so that particular example doesn't seem the cause.

Maybe I should reinstall CF. AFAIK that means remove it, reinstall stock, then reinstall CF.

 

[Black Hole]

I thought I was pretty thorough in my original post

The point is we're not looking over your shoulder and can't see what you're seeing but not noticing. The screen grabs in Luke's post should give you the idea. You are not disclosing details such as programme names either. It all matters (or might matter). Please post the information displayed on the WebIF front page bottom right.

In the absence of guru-level suggestions, it feels like a Jim problem to me.

Maybe I should reinstall CF. AFAIK that means remove it, reinstall stock, then reinstall CF.

You don't need to go that far, the firmware itself won't need refreshing (only the elements on HDD). WebIF >> Diagnostics >> CFW Reset

 

[MymsMan]

Is anything shown in webif-error.log (viewable via the diagnostics page) following a crop attempt?

As an experiment try a Detectads run with cropping within detectads and see if that runs (can specify -crop y -debug 2 in "Process options" if you don't want to change the global setting)

 

[paultry]

It's working! I'm running some more tests, will come back when they're done tomorrow.

 

[Black Hole]

It would be helpful to know what you did to get it working (if anything). CFW Reset?

 

[paultry]

The point is we're not looking over your shoulder and can't see what you're seeing but not noticing. The screen grabs in Luke's post should give you the idea. You are not disclosing details such as programme names either. It all matters (or might matter). Please post the information displayed on the WebIF front page bottom right.

In the absence of guru-level suggestions, it feels like a Jim problem to me.


You don't need to go that far, the firmware itself won't need refreshing (only the elements on HDD). WebIF >> Diagnostics >> CFW Reset

So, SUCCESS!

I did a CFW Reset. Crop failed again. Because I couldn't read the message that flashed up as it failed, this time I screen recorded it, then slowed down replay in VLC, froze it, and took a screenshot, see attached, but it reads:

"Processing The Hitman's Wife's Bodyguard_20241116_2200 (inverted: 0)
This recording already exists within _original
Cannot continue."

with "cropping complete" above it, as is usual.

I deleted the corresponding file in the _original folder, repeated the crop and it worked. To a layman it looks like crop decides it has already run if there is a copy of the file in the _original folder - irrespective of why it is there.

I now realise that the CFW Reset probably had no effect. All along, it seems that decrypt puts a copy of the encrypted file in an "_original" folder. Detectads doesn't care, but crop does, hence the message above, so the file in _original needs deleting before a crop. When crop does succeed, it also leaves a copy in _original, uncropped. This behaviour survives a back-switch restart.

If this is standard behaviour, then it's just a question of knowing about it. The combination of no documentation of this slightly unexpected behaviour, and the message being unreadable, made for a headache. Is it possible for a Dev to tweek crop behaviour so it keeps the message on screen a bit longer? And maybe include extra text "..until that is deleted" on the last line. That would probably get everyone over the line.

Can anyone else replicate this behaviour?

Many thanks to Luke and MymsMan for their help.

 

[Black Hole]

To a layman it looks like crop decides it has already run if there is a copy of the file in the _original folder - irrespective of why it is there.

No, what happened is the backup copy of the original uncropped file failed to be created as expected, so the process took the least risky option and aborted. Under normal circumstances this wouldn't happen, but the invisibility of error messages definitely doesn't help.

If this is standard behaviour

Not standard, but the result of meddling fingers while you were exploring. An edge case.

Can anyone else replicate this behaviour?

No need, it will happen any time you try to re-run a crop without tidying up from the previous run properly.