Beta [webif] Web Interface 1.4.x

Status
Not open for further replies.
af123

af123

Administrator
Staff member
I've just uploaded webif 1.4.0 to the beta repository along with a number of updated plugins to go with it.

I'd like to extend my thanks MymsMan for being instrumental in inspiring the massive improvement this version brings to automatic processing and for his help with design and testing.

Here are the packages now available to anyone with the beta repository enabled:
  • arbookmarks 1.0.2-1
  • badnts 1.0.1-1
  • detectads 0.2.4-0
  • disable-dso 0.3-1
  • disable-ota 1.0.3-4
  • flatten 1.0.7-2
  • flatview 1.0.5-1
  • newk 1.0.5-1
  • sweeper 2.1.5-4
  • webif 1.4.0
This list includes a new version of the detectads package, which has been updated for the new queuing system (I'll let MymsMan talk more about it).
disable-ota and disable-dso are now real-time-scheduling aware and will remove OTA & DSO events from the schedule without requiring a reboot.

1.4.0 Release Notes
New:
  • Complete overhaul of the background auto-processing system thanks to inspiration and help from MymsMan;
    • Its work has been split into two separate tasks - media scanner and queue runner;
    • The media scanner adds jobs to the queue rather than processing them as it runs. This makes its run-time shorter and more predictable. It still performs fairly quick actions such as expire and de-duplication in-line;
    • Rather than scanning the recording tree once for each action (decrypt, dedup, shrink, etc.), the new media scanner scans the recording hierarchy just once per run. This change alone has halved the run-time (although it depends on the directory structure to some extent);
    • The default automatic processing scan interval has been changed from every 20 minutes to every 10;
    • Most existing automatic processing plugins (sweeper,flatten,flatview,newk) will work unchanged with the new system but updates are available so that they can take advantage of the new framework;
    • The badnts,arbookmarks and detectads (if using post-decrypt mode) plugins will not work properly and must be upgraded;
    • The web interface built-in actions (decrypt, shrink, etc.) are now implemented as plugins, just like any other extension package;
    • Background automatic tasks such as decryption and shrink now appear in the queue;
  • The queue diagnostic screen has been updated:
    • Shows more information about tasks;
    • Allow a task to be manually put on hold or re-submitted to the queue;
    • Provide links back to the media browser for each queue entry;
  • A link to the queue screen can be added to the toolbar (new option in Settings->Auto Processing);
  • It's now possible to manually specify MP2 or MP3 audio when manually adding files to the queue from the browse screen;
  • Viewing a schedule backup now shows favourites from the backup file too;
  • New hook to allow plugins to take action following the creation of a new directory (used by flatten to mark new folders as noflatten).
Fixes:
  • If a recording fails to decrypt, a system warning will only be generated after three attempts;
  • Restoration of favourite channels has been fixed;
  • The status panel was not correctly reporting MP3 extraction jobs;
  • Fix pkg _load method to handle packages with a missing description;
  • Fix duplicate suppression in system notifications for events with no associated date (such as system crashes);
  • Fix manual event scheduling (was reporting duplicate event).
Other Changes:
  • The . filename used for recursive flags has changed. Recursion is now indicated by the addition of an R (it was previously lower-cased);
  • Detect and report the CFW 3.11 kernel versions;
  • New {ts tsr} method to check if a recording was time-shifted.
 
I don't know how af123 does it! A major rewrite of auto processing, whilst simultaneously creating a new boot settings package, updating numerous other packages, maintaining the forums and reorganising the package list :doublethumbsup:

I struggle to find the time to make modest changes to a single package :( (but I did do the hard work of coming up with the idea that caused the rewrite :p )

The changes to DetectAds are fairly modest:
  • It now uses the system queue for all deferred requests rather than maintaining its own database
  • The old queue will be processed if there are outstanding requests on it but no new entries will be added and it will eventually be deleted.
  • You can add Ad Detection requests to the queue using the Browse and Sweeper 'Queue for' options
  • The old DetectAds Sweeper actions remain for compatibility with existing rules but now use the system queue
  • The sample Sweeper rule has been updated to use the 'Queue recording for' action
  • There are two new folder flags Auto DetectAds and No DetectAds
  • These work in conjunction with the existing Channel exclusions rules so it is not necessary to add the flag to each directory if you use the existing DetctAds Auto processing options (Automatically process whilst recording in progress? (chaserun) or Automatically process recording following auto-decryption? (traditional) )
  • You can use these flags to include additional directories (Auto DetectAds) where you want all (including pre-existing recordings) ad detected or exclude directories (No DetectAds) which would otherwise be subject to ad detection based on channel filters.
  • For instance you could set No DetectAds on the 'My Videos' root folder to prevent auto ad detection of one-off recordings (I don't recommend setting Auto DetectAds on 'My Videos' since this would override the channel selection filters and cause detection of all one-off recordings including those on BBC channels)
  • Ad Detection is slow so be be sparing when adding batches of existing recordings to the queue
  • DetectAds processing can still be deferred to, for example overnight, by setting a target start time on the settings page (remember to leave machine on or set a reminder)
  • Logging for Queued requests will now be in auto.log but remain in detectads.log for detect while recording. Chaseget logging will be in chaseget.log.
  • Chaserun processing remains a process while recording operation though it does create a temporary queue entry as a safeguard against failures during recording.
 
Last edited:
MymsMan said:
These work in conjunction with the existing Channel exclusions rules so it is not necessary to add the flag to each directory if you use the existing DetctAds Auto processing options...

(I don't recommend setting Auto DetectAds on 'My Videos' since this would override the channel selection filters and cause detection of all one-off recordings including those on BBC channels)
Click to expand...
I find these two statements mutually contradictory.
 
Black Hole said:
I find these two statements mutually contradictory.
Click to expand...
Lets try again in the hope the wordsmiths can come up with a better phrasing.

The Auto DetectAds flag means all recordings in a folder will be processed regardless of the recording channel.
The No DetectAds flag means none of the recordings in a folder will be processed regardless of the recording channel.
In the absence of a flag recordings will be processed during recording (chaserun) or following decryption (traditional) if they are on a channel that has not been excluded by the channel filters.

If you routinely want to detect ads in all commercial programmes as I do you don't need to use the new flags at all.
If you only want a subset of programmes processed you can use the flags to include/exclude directories overriding the channel filters

The processing is (deliberately) different from the other auto options where you have to have a flag set before any processing happens.
 
Last edited:
My HDRs Auto Updated to Web If v1.4. One of them crashed within seconds of start-up and a warning on the screen had the usual statement of 'some packages have been disabled'. I chose to run the fix-flash packages diagnostic. However, the log said something to the effect of 'unable to download Web If'. When I clicked OK on the log box, I was given the HDD directory tree, no Web If.

I then rebooted the HDR via the front button. On logging in to the HDR, I was faced with another message 'Download and Install the Web If'. I did so and all is back to normal.

I can replicate this every time I run fix-flash packages the same happens.
 
Last edited:
MymsMan said:
The Auto DetectAds flag means all recordings in a folder will be processed regardless of the recording channel.
The No DetectAds flag means none of the recordings in a folder will be processed regardless of the recording channel.
In the absence of a flag recordings will be processed during recording (chaserun) or following decryption (traditional) if they are on a channel that has not been excluded by the channel filters.
Click to expand...
Right, got it. It's the "in conjunction with" that was the confusing point, whereas the folder flags completely override the channel exclusions. I agree, not a good idea to set the flags on My Video - just leave that to the channel exclusions.

Can sweeper tell when a recording has been processed, so a task isn't queued a second (and third, and fourth...) time?
 
Black Hole said:
Can sweeper tell when a recording has been processed, so a task isn't queued a second (and third, and fourth...) time?
Click to expand...
Yes, the Sweeper Queue recording for action won't add a second entry to the queue and there is also a 'Queued for' condition that can be included in the condition list. Once processing is complete there are the existing hmt flags to show that a recording has been decrypted, shrunk or ad_detected

Off hand I am not sure how you would code a sweeper rule to check whether audio or video extraction has been performed - probably a file exists check with a suitable template
 
Working well here with detectads just set to process my Timeless folder : )
Screenshot%202017-01-18%2022.23.15.png
 
My Queue Tasks screen is approx two screens wide which means I have to scroll to the right to see the Last Update column. Would it be possible to truncate the Shrink log output to show only, for instance, '2.66 GiB in 144.579 seconds - 18.82 MiB/s Saved'. The more detailed information is available in the auto.log should it be required.
Also the 'Last media scan:' is hardly visible - could this be slightly more bold ?
 
The bottom of my queued tasks log says this:
Last media scan: Thu Jan 19 11:15:12 GMT 2017 (00:01:20 ago) - scanning every 0 minutes.

Is it supposed to say "0 minutes"? Seems an odd figure. What is it actually doing?
 
peterworks said:
My Queue Tasks screen is approx two screens wide which means I have to scroll to the right to see the Last Update column. Would it be possible to truncate the Shrink log output to show only, for instance, '2.66 GiB in 144.579 seconds - 18.82 MiB/s Saved'. The more detailed information is available in the auto.log should it be required.
Click to expand...
You can put this into EXTRA.css if you want the details column to wrap. I prefer the short view with having to scroll right for details.

Code: 
td.queuelog { white-space: normal !important; }
Also the 'Last media scan:' is hardly visible - could this be slightly more bold ?
Click to expand...
It's supposed to be unobtrusive, really just a debugging tool. You can highlight it with the mouse to make it more visible.
 
prpr said:
The bottom of my queued tasks log says this:
Last media scan: Thu Jan 19 11:15:12 GMT 2017 (00:01:20 ago) - scanning every 0 minutes.

Is it supposed to say "0 minutes"? Seems an odd figure. What is it actually doing?
Click to expand...
0 means it's using the default of 10 minutes. I'll update it to say that!
 
Favourites restore still seems to be problematical.
I did the following:
1. A manual backup of the schedule
2. A retune
3. A restart (from the remote)
4. A restore of the backup taken in 1 above
5. A restart from within WebIf
The schedule restored okay, however, during the restore a lot of entries displayed the original lcn(?) and then showed and updated the entry with the changed lcn(?). I think AUL was displayed but cannot quite remember.
The restore of Favourites did not work correctly. It created the correct number of Favourite Groups with their correct group names and the correct number of entries in each group. The entries in the groups, however, were incorrect. In fact some just showed a zero.
I don't know if there is a restore log and/or whether it would be useful.
I suspect I will not be able to replicate this until there are changes requiring a retune.
 
Last edited:
I was unsure whether to report this as I cannot replicate but...
Yesterday I recorded something between 16:00 and 17:00 pm from standby. I turned on the pvr at 17:00 and, when checking the queue 20 minutes later found that the program had been shrunk successfully but was unable to decrypt (see attached extract from browsing the folder and from auto.log). Auto processing was checking every minute and coming up with the 'protected flag' defer reason.
On going to the WebIf restart option the 'may not be able to restart, may need manual intervention' message appeared. I restarted anyway and I did have to manually intervene. Decryption was successful on the first run of auto processing.
Checking the auto log from the overnight processing (I have set it to do all processing from the previous evenings recordings between 04:00 and 05:00) everything went smoothly.

Capture20-01-2017-17.25.46.jpg Capture20-01-2017-17.26.18.jpg
 
Unprotect works as a recmon triggered process at the end of recording so have a look at the recmon.log to see if there is anything there.

With the new Auto structure it would probably worth retrying unprotect from within Decrypt processing if it hasn't happened by the first retry
 
Tks MymsMan - nothing in the recmon.log since 17/01/17 at 17:05 as below:

4 at file "/mod/etc/recmon.d/rslog", line 16
3 /mod/etc/recmon.d/rslog:16: Error: invalid command name "0"
2 at file "/mod/etc/recmon.d/webiflog", line 16
1 /mod/etc/recmon.d/webiflog:16: Error: invalid command name "0"
 
Status
Not open for further replies.
Back
Top