Beta [webif] 1.4.9-6

Status
Not open for further replies.
prpr

prpr

Well-Known Member
The Auto processing queue has "How many days should completed entries stay in the queue?" set to 7 on my boxes.
Yet I have entries going back at 3 least months in the log. Something is broken...
 
Auto log entries for this morning:

213716/05/2021 10:18:02 - Auto de-queue processed 0 items in 0.161 seconds.
213216/05/2021 10:18:02 - Auto de-queue starting
212916/05/2021 10:17:02 - Auto de-queue processed 0 items in 0.127 seconds.
212416/05/2021 10:17:02 - Auto de-queue starting
212116/05/2021 10:16:02 - Auto de-queue processed 0 items in 0.142 seconds.
211616/05/2021 10:16:02 - Auto de-queue starting
211316/05/2021 10:15:02 - Auto de-queue processed 0 items in 0.135 seconds.
210816/05/2021 10:15:02 - Auto de-queue starting
 
My queue database goes back to 17th Feb in one case, 18th Feb (with some errant older entries) in another, and 17th April in the third. Whatever triggered this seems to have been around 16th Feb (and maybe I didn't update the third HDR for a couple of months).
 
I'm not clear where the log (as opposed to the queue) comes into it – log entries are persistent until the log file exceeds the preset size limit and a new log file is started (and even then the old log file persists until it exceeds the number of log files to keep).
 
It's not related to the firmware (which has zero knowledge of such stuff). It's something to do with the WebIf processing, hence the thread title.
Presumably database maintenance is done somewhere in the Auto processing. Will have to have a hunt, unless @/df gets there first :).
 
prpr said:
Presumably database maintenance is done somewhere in the Auto processing.
Click to expand...
Yes, in /mod/webif/lib/auto/deq and the part in question is handled by /mod/webif/lib/queue.class
Specifically, this part is the cause of the problem:
Code: 
{ {
                delete from queue
                where status in ('COMPLETE', 'FAILED')
                and submitted < %s    
        } [expr [clock seconds] - 86400 * $days]
        }
I have no idea how to fix it, but you can't bolt command execution in there like that, nor substitute %s.
This is the stuff to blame.
 
I gave up (after battling with the useless Jim manual, which I loathe most of the time) and reverted it:
Code: 
cd /mod
wget https://git.hpkg.tv/hummypkg/webif/commit/7834dae075b029b8ac804efa2252cf2dc8ed22c2.diff -O q.diff
patch -p1 -R <q.diff
rm q.diff

I'm afraid to say that this patch smacks of trying to be too clever and obviously this part of it hasn't been tested. The rest of it seems to be fine though, as far as I can tell.
 
The idea of this bit (ll. 149+ of the new code) is that we run a list of queries instead of running each separately. This means that the whole lot can be protected by a transaction and either succeed or fail together.

Let's work it through. At ll.132+ in the class function proc {queue dbqueryl} {query_list {txn_mode ""}} we run each query
Code: 
 			foreach q $query_list {
				$db query {*}$q
			}
So query_list is a list of items, each of which is a list of arguments to the Jim SQLite query database method, which can bind parameters like so:
Code: 
# Eg: '$db' is a handle to an open DB with a table 'tab' having an integer column 'col' 
$db query { select * from tab where col > %s } 12
# will run 
# select * from tab where col > 12;
In the problem code, there are 3 constant queries followed by one that has to be computed each time. The problem, I believe, is that [command substitution] doesn't happen inside {}, so the computed query isn't being constructed properly.

The list and its 4th item have to be constructed instead of being written out, something like this:
Code: 
 		return [queue dbqueryl [list { {
			update queue
			set status = 'INTERRUPTED',
			log = 'Job will be retried automatically.',
			retries = retries + 1,
			interrupts = interrupts + 1
			where status = 'RUNNING'
		} } { {
			update queue
			set status = 'FAILED',
			log = 'Too many interrupts.'
			where status = 'INTERRUPTED'
			and interrupts >= 5
		} } { {
			update queue
			set status = 'PENDING'
			where status = 'DEFER'
		} } [list {
			delete from queue
			where status in ('COMPLETE', 'FAILED')
			and submitted < %s
		} [expr [clock seconds] - 86400 * $days]
		] ] ]
An alternative is to use subst which forces an additional evaluation:
Code: 
 		return [queue dbqueryl [subst { { {
			update queue
			set status = 'INTERRUPTED',
			log = 'Job will be retried automatically.',
			retries = retries + 1,
			interrupts = interrupts + 1
			where status = 'RUNNING'
		} } { {
			update queue
			set status = 'FAILED',
			log = 'Too many interrupts.'
			where status = 'INTERRUPTED'
			and interrupts >= 5
		} } { {
			update queue
			set status = 'PENDING'
			where status = 'DEFER'
		} } { {
			delete from queue
			where status in ('COMPLETE', 'FAILED')
			and submitted < %s
		} [expr [clock seconds] - 86400 * $days]
		} } ] ]
This approach would cause problems if the list being processed contained $ or [ outside the command that was intended to be evaluated, but it doesn't (yet).
 
Last edited:
I have some completed queue entries for last night on a machine with the code revised as above (1st version) and a 1 day limit for completed entries. Let's see what happens.

[... time passes ...]

And the answer is: There are no tasks in the queue.

Fixed in https://git.hpkg.tv/hummypkg/webif/pulls/46.
 
Last edited:
prpr said:
...
I'm afraid to say that this patch smacks of trying to be too clever and obviously this part of it hasn't been tested. The rest of it seems to be fine though, as far as I can tell.
Click to expand...
Possibly, but the rationale is in my post above. Beta test gladly accepted!

You are a bit harsh on the Jim manual (considering how much open source is only documented in its code), but you can always send PRs to Steve ...
 
Not sure when the next WebIf update is so I thought I might have a go at patching. Is there any guidance that I can follow ? Also I see 'You are not authorized to merge this pull request.' on @/df 's link.
Can I log in using the same login info that I use for the forum ?
 
The authentication prompt is just saying that @af123 (or someone else authorised) has to carry out the process of merging the pull request into the main branch, but that doesn't affect you. Although it's not relevant, the login details are different.

If you append .diff to that link, you get a URL that gives the text of the required patch file.

In your telnet or WebShell session, make /mod the current directory. Then wget -O- 'https://git.hpkg.tv/hummypkg/webif/pulls/46.diff' | patch -p1.
 
Last edited:
peterworks said:
Not sure when the next WebIf update is
Click to expand...
1.4.9-6 should have been released to the main repository by now. It's been nearly 6 months since the last one.
1.4.9-7 should have been in beta, and also released to the main repository by now.
There's loads of other stuff in beta that has been there for months, perhaps even a year now.
 
I tried patching using the WebShell and Putty but it failed with the following output:
H2# cd /mod/webif
H2# wget -O- 'https://git.hpkg.tv/hummypkg/webif/pulls/46.diff' | patch -p1
--2021-06-29 10:19:06-- https://git.hpkg.tv/hummypkg/webif/pulls/46.diff
Resolving git.hpkg.tv... 2a00:5600:1600::50, 89.248.55.75
Connecting to git.hpkg.tv|2a00:5600:1600::50|:443... failed: Address family not supported by protocol.
Connecting to git.hpkg.tv|89.248.55.75|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 779 [text/plain]
Saving to: 'STDOUT'

- 100%[===================>] 779 --.-KB/s in 0s

2021-06-29 10:19:07 (5.05 MB/s) - written to stdout [779/779]

patching file webif/lib/queue.class
patch: can't open 'webif/lib/queue.class': No such file or directory
H2#

More help needed please !
 
Status
Not open for further replies.
Back
Top