[webif] Web Interface 1.4.x

Status
Not open for further replies.
/df said:
The Restore function doesn't restore custom favourites list names (WebIf 1.4.8-2, rsvsync 1.1.12).

As well as restoring the itemText it's necessary to set the itemValue, which controls whether the custom name is used.

This is the SQL command in the /mod/boot/rsvsync binary:
Code: 
update setup.TBL_MENUCONFIG set itemText = '%s' where itemName = 'FAV_CUSTOM_STR0%d'
And this is what it should do:
Code: 
update setup.TBL_MENUCONFIG set itemText = '%s', itemValue = 1 where itemName = 'FAV_CUSTOM_STR0%d'
Click to expand...
Although this appeared to be fixed and the suggested SQL is in the rsvsync v1.13 binary, it still seems not to be working in some cases (after a forced retune, eg). Apparently I had to restore the schedule twice to get the favourites list names to be shown.

After a forced retune, I had favourites lists named "Favourite 1" and "Favourite 2", which was disappointing.

I restored a saved schedule:
Code: 
Listing scheduled events from auto-2020-Aug-07-02:08... 
Backup version 2 
Disable OTA (BBC Red Button - Fri Aug 07 04:20:00 BST 2020 - 04:40)
BBC News (BBC ONE Lon - Mon Aug 10 04:29:00 BST 2020 - 04:30)
Favourites 1 'Faves'
Favourites 2 'FavesHD'
Favourites 3 'Favourite 3'
Favourites 4 'Favourite 4'
Favourites 5 'Favourite 5'
...
When that was saved, nos. 3-5 didn't appear in the on-screen UI (Guide>Blue).

After restoring and restarting once:
Code: 
# sqlite3 /var/lib/humaxtv/setup.db 
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> select * from TBL_MENUCONFIG where itemName like 'FAV_CUSTOM_STR0%';
35|FAV_CUSTOM_STR01|0|Favourite 1|
36|FAV_CUSTOM_STR02|0|Favourite 2|
sqlite>
{/code]
And after restoring again but before restarting:
[code]
# sqlite3 /var/lib/humaxtv/rsvp.db 
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> .tables
fav       favnames  pending   skip    
sqlite> select * from favnames ;
1|Faves
2|FavesHD
3|Favourite 3
4|Favourite 4
5|Favourite 5
sqlite>
And then after restarting again, it's all fine.

Unsurprisingly this is also a problem with auto-schedule-restore.

Could there be an issue where the first restore fails to set the custom names but somehow populates previously missing fields in TBL_MENUCONFIG and the update then succeeds?
 
Last edited:
I created a script for usqlite to run at xinit-time ahead of rsvsync; then ran Menu>Installation>Automatic Search; then restart. At that point auto-schedule-restore ran and rebooted, as expected. After this the favourite group names, as well as the other backed up data, had been correctly restored.

The /mod/boot/xinit.d/favnames script, though it's the two SQL statements introduced by insert and updatewith fav that matter.
Code: 
#!/bin/sh

PATH=${0%/*}/..:$PATH

# for testing, arg1 can set DB dir
dbs=${1:-/var/lib/humaxtv}

attach() { # db_path db_alias
	printf "attach '%s' as %s" "$1" "$2"
}

tables() { # db_alias
	printf "select name from %s.sqlite_master where type='table'" "$1"
}

# check for favnames table
if usqlite "$(attach "$dbs/rsvp.db" x); $(tables x);" <&- | grep -q 'favnames'; then
	# process the pending/restored favnames 
	# as this will be executed at xinit, limit to SQLite 3.7.5:
	#	no WITH, no UPDATE set column-name-list = row-value
	usqlite <<EOM
$(attach "$dbs/rsvp.db" rsvp);
$(attach "$dbs/setup.db" setup);
begin transaction;
update setup.TBL_MENUCONFIG set itemValue=0 where itemName like 'FAV_CUSTOM_STR0%';
create temp view fav as 
	select *, 'FAV_CUSTOM_STR0'||idx as fname, 'Favourite '||idx as ftext from rsvp.favnames;
create temp view ridx as 
	select max(itemIdx - _rowid_) as offs from setup.TBL_MENUCONFIG;
insert into setup.TBL_MENUCONFIG                                              
        select (-1), f.fname, 0, f.ftext, NULL from fav f 
		where f.fname not in (select itemName from setup.TBL_MENUCONFIG);
update setup.TBL_MENUCONFIG
	set itemIdx = _rowid_ + (select offs from ridx)
	where itemIdx = -1;
update setup.TBL_MENUCONFIG 
	set itemValue = 1, itemText = (select f.name from fav f where itemName = f.fname)
	where itemName in (select fname from fav);
drop table rsvp.favnames;
commit;
select 'Favourite group names:';
select itemName, itemText from TBL_MENUCONFIG where itemName like 'FAV%';
EOM
fi
if usqlite "$(attach "$dbs/rsvp.db" x); $(tables x);" <&- | grep -q 'fav'; then
	# process the pending/restored favourites 
	usqlite <<EOM
$(attach "$dbs/rsvp.db" rsvp);
$(attach "$dbs/channel.db" channel);
$(attach "$dbs/setup.db" setup);
begin transaction;
insert into channel.TBL_FAV select * from rsvp.fav;
drop table rsvp.fav;
update setup.TBL_MENUCONFIG set itemValue = 0
	where itemName like 'FAV_CUSTOM_STR0%' and
	    (select count(hSvc) from channel.TBL_FAV f
		 where itemName = 'FAV_CUSTOM_STR0'||f.eFavgroup) <= 0;
commit;
select 'Favourite groups:';
select itemText || ' (' || case itemValue when 0 then 'dis' else 'en' end || 'abled) has ' || count(hSvc) 
    from channel.TBL_FAV f, setup.TBL_MENUCONFIG s 
	where s.itemName = 'FAV_CUSTOM_STR0'||f.eFavgroup group by eFavGroup;
EOM
fi

The final state:
Code: 
# sqlite3 /mod/boot/../setup.db 
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> select * from TBL_MENUCONFIG where itemName like 'FAV%';
35|FAV_CUSTOM_STR01|1|Faves|
36|FAV_CUSTOM_STR02|1|FavesHD|
sqlite>
As identified in post #445, it's important to initialise the itemIdx when inserting any missing rows.
And even more significantly, unlike the version tested above, the SQlite 3.7.5 library that is used at xinit-time doesn't support WITH or UPDATE SET column-name-list = row-value. Hence the SQL part of the script has been rewritten.
 
Last edited:
Queue processing can fail if the settings database (table settings in /mod/etc/webif.db) is locked, as the first thing it does when scanning or dequeuing is to try to fetch its log setting:
Code: 
at file "/mod/lib/jim/oo.tcl", line 73
at file "/mod/webif/lib/settings.class", line 111
/mod/webif/lib/settings.class:111: Error: database is locked
in procedure '<reference.<setting>.00000000000000000000>' called at file "/mod/webif/lib/auto/scan", line 11
Perhaps this should be trapped and the operation deferred? Maybe create settings::safe_get that will either exit with diagnostic or return a setting.
 
/df said:
Queue processing can fail if the settings database (table settings in /mod/etc/webif.db) is locked, as the first thing it does when scanning or dequeuing is to try to fetch its log setting:
Code: 
at file "/mod/lib/jim/oo.tcl", line 73
at file "/mod/webif/lib/settings.class", line 111
/mod/webif/lib/settings.class:111: Error: database is locked
in procedure '<reference.<setting>.00000000000000000000>' called at file "/mod/webif/lib/auto/scan", line 11
Perhaps this should be trapped and the operation deferred? Maybe create settings::safe_get that will either exit with diagnostic or return a setting.
Click to expand...
This problem can affect any process that accesses settings not just queue and there are hundreds of calls to settings scattered across the webif and packages with very few calls catching any errors in settings.

So yes, settings should catch the db access calls and retry after a second, an error should only be raised if the db remain locked after 5 retries.
 
@StanLaurel carried out a mass decryption in which two recordings failed to be decrypted by the accelerated method but could be decrypted using the direct method.
StanLaurel said:
...
View attachment 4996
Click to expand...
I realised I also had one of these fine messes:
Code: 
18/10/2020 17:28:20 - De-queuing 36552 - decrypt - /mnt/hd2/My Video/The Handmaid's Tale/The Handmaid's Tale S2/The Handmaid's Tale_20180729_2102.ts
18/10/2020 17:28:22 - decrypt:  DECRYPT: /mnt/hd2/My Video/The Handmaid's Tale/The Handmaid's Tale S2/The Handmaid's Tale_20180729_2102
18/10/2020 17:28:22 - decrypt:  DLNA: http://127.0.0.1:9000/web/media/147.TS
18/10/2020 17:33:05 - decrypt:  /mnt/hd2/My Video/The Handmaid's Tale/The Handmaid's Tale S2/The Handmaid's Tale_20180729_2102.ts - File did not decrypt properly.
But:
Code: 
# wget --progress=dot:giga -O "The Handmaid's Tale_20180729_2102.decr.ts" http://127.0.0.1:9000/web/media/147.TS
--2020-10-18 19:02:09--  http://127.0.0.1:9000/web/media/147.TS
Connecting to 127.0.0.1:9000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1810875840 (1.7G) [video/ts]
Saving to: 'The Handmaid\'s Tale_20180729_2102.decr.ts'

     0K ........ ........ ........ ........  1% 7.52M 3m45s
 32768K ........ ........ ........ ........  3% 5.37M 4m25s
 ...
1703936K ........ ........ ........ ........ 98% 7.93M 4s
1736704K ........ ........ ........ ......  100% 8.09M=3m19s

2020-10-18 19:05:28 (8.69 MB/s) - 'The Handmaid\'s Tale_20180729_2102.decr.ts' saved [1810875840/1810875840]

# ls -l The\ Handmaid\'s\ Tale_20180729_2102*
-rw------- 1 root root 1810875840 Oct 18 19:05 The Handmaid's Tale_20180729_2102.decr.ts
-rw-r--r-- 1 root root      13888 Aug  2  2018 The Handmaid's Tale_20180729_2102.hmt
-rw-r--r-- 1 root root    6017088 Jul 29  2018 The Handmaid's Tale_20180729_2102.nts
-rw-r--r-- 1 root root      43680 Aug  2  2018 The Handmaid's Tale_20180729_2102.thm
-rw-r--r-- 1 root root 1810875840 Jul 29  2018 The Handmaid's Tale_20180729_2102.ts
#
I wonder if whatever decides about the propriety of the decryption is somehow confused?
 
Last edited:
MymsMan said:
This problem can affect any process that accesses settings not just queue and there are hundreds of calls to settings scattered across the webif and packages with very few calls catching any errors in settings.

So yes, settings should catch the db access calls and retry after a second, an error should only be raised if the db remain locked after 5 retries.
Click to expand...
Like this?
 
/df said:
I wonder if whatever decides about the propriety of the decryption is somehow confused?
Click to expand...
Auto decryption quite often fails but will succeed when retried later, but there are a few files where it fails persistently
The test used is the length returned is exactly equal to the original file length.

In detectads I use a tolerance of 30 second worth of recording before requeing the recording to retry decryption and detection later since the last few seconds tend to be the credits and following ads so not usually critical.
 
What I see is that the test stripts -qE, at l.160 of webif/lib/auto/plugin/decrypt/queue.hook may return 1 (encrypted) for what was supposed to be a DLNA-decrypted file. But that seems to be because the recording was imported from another machine without decrypting and before the source machine's key was trivialised to match the test machine.

The DLNA streaming operation doesn't contain a check that the key is valid for the recording. Now that we know how to do that (stripts -/), should the key be checked before wasting a load of MB? This would clarify why some recording keeps getting decrypted without actually being decrypted. The code used for the direct case could be pulled out: test 1 key (system encryptionkey ?) for DLNA, up to 3 for direct. In fact if the DLNA key doesn't work but one of the other known keys matches, the job could be switched to direct decryption.

The other case referenced above would be explained if the system key had been changed after recording, since direct decryption tries other keys and did manage to decrypt.
 
/df said:
The other case referenced above would be explained if the system key had been changed after recording, since direct decryption tries other keys and did manage to decrypt.
Click to expand...
It would, but it seems unlikely that anyone reporting this issue isn't going to mention that they've been importing encrypted recordings or messing about with the keys. I guess there could be a problem if there is a failure mechanism in the boot-time imposition of a custom key, but again only for those using a custom key.
 
Isn't trivialising the key now SOP? The example that failed was from 2015, so well before the key breakthrough. The failure using DLNA together with success using stripts doesn't seem to admit another explanation.
 
/df said:
The code used for the direct case could be pulled out: test 1 key (system encryptionkey ?) for DLNA, up to 3 for direct. In fact if the DLNA key doesn't work but one of the other known keys matches, the job could be switched to direct decryption.
Click to expand...
Are you advocating having 3 configurable decryption keys in the WebIf then? That seems like a good idea to me. It would save messing around with the encryption key setting.
 
prpr said:
Are you advocating having 3 configurable decryption keys in the WebIf then? That seems like a good idea to me. It would save messing around with the encryption key setting.
Click to expand...
Not specifically. Is this a worthwhile use case?

The queued direct decryption code tries the active key (as revealed by nugget) and keys derived in two other ways, which may actually be the same, ie the native key for the box: see https://git.hpkg.tv/df/webif/src/branch/df-tsgetkey-patch/webif/lib/ts.class l.833 on.
 
prpr said:
...
You've got a bit of an indentation error in there (849-854)!
Click to expand...
Well, you know what I think about that. In this case not only does the IDE (Gitea) not apply the desired format, it has an unstoppable auto-indent function that breaks any correctly formatted code pasted into its editor.
 
/df said:
...
The queued direct decryption code tries the active key (as revealed by nugget) and keys derived in two other ways, which may actually be the same, ie the native key for the box: see https://git.hpkg.tv/df/webif/src/branch/df-tsgetkey-patch/webif/lib/ts.class l.833 on.
Click to expand...
Here's a show that was supposed to have been recorded with the all-zeroes key: contrary to expectation it fails the test with stripts -/; the modified queued decryption code tries direct decryption, and it works (and the program is viewable!).
Code: 
21/10/2020 16:37:02 - De-queuing 36559 - decrypt - /mnt/hd2/My Video/Budgie/Budgie S1E01 Out.ts
21/10/2020 16:37:03 - decrypt:  DECRYPT: /mnt/hd2/My Video/Budgie/Budgie S1E01 Out
21/10/2020 16:37:04 - decrypt:  Direct decryption
21/10/2020 17:40:55 - decrypt:  Removing/binning old copy.
21/10/2020 17:40:59 -     OK - 1.1 GiB in 3835.172 seconds - 300.23 KiB/s - 
21/10/2020 17:40:59 -   ARBOOKMARK: /mnt/hd2/My Video/Budgie/Budgie S1E01 Out
21/10/2020 17:40:59 - Recording was not padded, cannot process.
21/10/2020 17:40:59 - Processed in: 0.06s
21/10/2020 17:40:59 - Done...
Whereas episode 2 and later of the same test OK with stripts -/:
Code: 
# for ff in /mnt/hd2/My\ Video/Budgie/Budgie*.ts; do 
> stripts -/ "00000000000000000000000000000000" "${ff%.ts}" 2>&1 | grep -v "Processed in:"
> done
Recording is not encrypted.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
Encryption key is correct for this recording.
#
This example was deleted as part of the decryption (no undelete). Let's find out if other recordings might be affected:
Code: 
# cat /tmp/ckenc
#!/bin/sh
main() {
        x="$(stripts -q/ "$1" "${2%.*}" 2>&1)"
        if echo "$x" | grep -q "not encrypted"; then 
                return 0
        elif [ "$x" = 1 ]; then
                return 0
        fi 
        return 1
}
main "$@"
#  find /mnt/hd2/My\ Video -name '*.ts' ! -exec /tmp/ckenc "00000000000000000000000000000000" "{}" \; -print
/mnt/hd2/My Video/Civilisations/bad?/Civilisations s00e06 - 6_9_ First Contact.ts
/mnt/hd2/My Video/Exploring China_ A Culinary Adventure/Exploring China_ A Culinary Adventure_20170121_1158.ts
/mnt/hd2/My Video/The Handmaid's Tale/The Handmaid's Tale S2/The Handmaid's Tale_20180729_2102.decr.ts
/mnt/hd2/My Video/The Handmaid's Tale/The Handmaid's Tale S2/_org/The Handmaid's Tale_20180729_2102.ts
/mnt/hd2/My Video/The Handmaid's Tale/The Handmaid's Tale S2/The Handmaid's Tale_20180729_2102.ts
...
#
The list contains various cases:
  1. corrupt shows, maybe recorded when an old disk was failing, or when the aerial feed was broken;
  2. orphan shows, accidentally imported without decrypting, or believing that the source machine's key had been trivialised;
  3. shows that were encrypted with the native key before it was trivialised;
  4. shows that were encrypted with the native key after it should have been trivialised - maybe a crash that needed fix-flash-packages;
  5. shows whose .ts was generated from .mp4 using ffmpeg.
I suppose the Budgie episode was case #4. It would be good if the last case could be caught.
 
prpr said:
Are you advocating having 3 configurable decryption keys in the WebIf then? That seems like a good idea to me. It would save messing around with the encryption key setting.
Click to expand...
How about /mod/etc/keys as a list of other keys (hex, one per line) to try? That should fix the case of occasional "foreign" material, allowing it to be direct-decrypted without major changes.

Obviously there could be a WebIf interface to the file though I would have thought that the sort of users who are going to use the feature would be just as happy to edit a file.

I don't think there's a reasonable way to change the system key temporarily since you'd have to eliminate the risk that new material gets recorded with the temporary key.
 
You need to understand how decryption works in CF.

Nowadays, using the stripts utility, we can decrypt any material recorded on Humax HD/HDR-Fox T2 given the key, but the best speed for that is slower than real-time; with the HD, it's all that's possible.

The Humax settop box program uses a key that's loaded into the program to drive high-speed encryption and decryption as part of the AV processing chain off-loaded from the Linux CPUs. Any material that is recorded and played back directly or using the DLNA server (in the HDR version) is using that key to process the material in real time. The accelerated decryption used in WebIf by default on the HDR saves a recording using its DLNA URL.

So if it was desired to perform accelerated decryption using some foreign encryption key, that key would have to be loaded into the settop program (for which a mechanism exists), but then the procedure would have to ensure that no other decryption or encryption would be run while the foreign key was in use, since only one key is active at any time: that means: no other accelerated decryption, such as a queued operation; no detectads; no playing back of recordings; no recording.

I think a good compromise is the current proposal:
  • accelerated decryption is always available on the HDR, if the active key matches the encrypted programme;
  • otherwise, or if it is selected, "direct" (software) decryption is used;
  • for direct decryption, each of these keys is checked and used if it matches: the active key, the native key, the saved active key, any key in the file /mod/etc/keys.
This would apply to queued decryption because Opt+>Decrypt (a) uses different code (-:() (b) shares a problem with any long-running Opt+> operation: the web server drops the page that is supposed to be updating with progress, making it appear that the operation has finished, although it actually continues and generally completes without ever clearing up after itself.
 
Last edited:
Status
Not open for further replies.
Back
Top