Beta My little projects

Here's something I just extracted from a 32s recording of a filtered .ts file:
Code:
$ dvbsnoop -s ts -if subs.ts -tssubdecode|grep Timestamp
                   ==> PTS: 2225001696 (0x849ed4e0)  [= 90 kHz-Timestamp: 6:52:02.2410]
                   ==> PTS: 2225008896 (0x849ef100)  [= 90 kHz-Timestamp: 6:52:02.3210]
                   ==> PTS: 2225131296 (0x84a0cf20)  [= 90 kHz-Timestamp: 6:52:03.6810]
                   ==> PTS: 2225138496 (0x84a0eb40)  [= 90 kHz-Timestamp: 6:52:03.7610]
                   ==> PTS: 2225325696 (0x84a3c680)  [= 90 kHz-Timestamp: 6:52:05.8410]
                   ==> PTS: 2225332896 (0x84a3e2a0)  [= 90 kHz-Timestamp: 6:52:05.9210]
                   ==> PTS: 2225520096 (0x84a6bde0)  [= 90 kHz-Timestamp: 6:52:08.0010]
                   ==> PTS: 2225527296 (0x84a6da00)  [= 90 kHz-Timestamp: 6:52:08.0810]
                   ==> PTS: 2225912496 (0x84acbab0)  [= 90 kHz-Timestamp: 6:52:12.3610]
                   ==> PTS: 2225919696 (0x84acd6d0)  [= 90 kHz-Timestamp: 6:52:12.4410]
                   ==> PTS: 2226106896 (0x84afb210)  [= 90 kHz-Timestamp: 6:52:14.5210]
                   ==> PTS: 2226114096 (0x84afce30)  [= 90 kHz-Timestamp: 6:52:14.6010]
                   ==> PTS: 2226445296 (0x84b4dbf0)  [= 90 kHz-Timestamp: 6:52:18.2810]
                   ==> PTS: 2226452496 (0x84b4f810)  [= 90 kHz-Timestamp: 6:52:18.3610]
                   ==> PTS: 2226510096 (0x84b5d910)  [= 90 kHz-Timestamp: 6:52:19.0010]
                   ==> PTS: 2226535296 (0x84b63b80)  [= 90 kHz-Timestamp: 6:52:19.2810]
                   ==> PTS: 2226600096 (0x84b738a0)  [= 90 kHz-Timestamp: 6:52:20.0010]
                   ==> PTS: 2226625296 (0x84b79b10)  [= 90 kHz-Timestamp: 6:52:20.2810]
                   ==> PTS: 2226636096 (0x84b7c540)  [= 90 kHz-Timestamp: 6:52:20.4010]
                   ==> PTS: 2226657696 (0x84b819a0)  [= 90 kHz-Timestamp: 6:52:20.6410]
                   ==> PTS: 2226675696 (0x84b85ff0)  [= 90 kHz-Timestamp: 6:52:20.8410]
                   ==> PTS: 2226704496 (0x84b8d070)  [= 90 kHz-Timestamp: 6:52:21.1610]
                   ==> PTS: 2226877296 (0x84bb7370)  [= 90 kHz-Timestamp: 6:52:23.0810]
                   ==> PTS: 2226898896 (0x84bbc7d0)  [= 90 kHz-Timestamp: 6:52:23.3210]
                   ==> PTS: 2226967296 (0x84bcd300)  [= 90 kHz-Timestamp: 6:52:24.0810]
                   ==> PTS: 2226985296 (0x84bd1950)  [= 90 kHz-Timestamp: 6:52:24.2810]
                   ==> PTS: 2226992496 (0x84bd3570)  [= 90 kHz-Timestamp: 6:52:24.3610]
                   ==> PTS: 2227093296 (0x84bebf30)  [= 90 kHz-Timestamp: 6:52:25.4810]
                   ==> PTS: 2227118496 (0x84bf21a0)  [= 90 kHz-Timestamp: 6:52:25.7610]
                   ==> PTS: 2227136496 (0x84bf67f0)  [= 90 kHz-Timestamp: 6:52:25.9610]
                   ==> PTS: 2227154496 (0x84bfae40)  [= 90 kHz-Timestamp: 6:52:26.1610]
                   ==> PTS: 2227233696 (0x84c0e3a0)  [= 90 kHz-Timestamp: 6:52:27.0410]
                   ==> PTS: 2227298496 (0x84c1e0c0)  [= 90 kHz-Timestamp: 6:52:27.7610]
                   ==> PTS: 2227395696 (0x84c35c70)  [= 90 kHz-Timestamp: 6:52:28.8410]
                   ==> PTS: 2227406496 (0x84c386a0)  [= 90 kHz-Timestamp: 6:52:28.9610]
                   ==> PTS: 2227431696 (0x84c3e910)  [= 90 kHz-Timestamp: 6:52:29.2410]
                   ==> PTS: 2227449696 (0x84c42f60)  [= 90 kHz-Timestamp: 6:52:29.4410]
                   ==> PTS: 2227467696 (0x84c475b0)  [= 90 kHz-Timestamp: 6:52:29.6410]
                   ==> PTS: 2227579296 (0x84c629a0)  [= 90 kHz-Timestamp: 6:52:30.8810]
                   ==> PTS: 2227597296 (0x84c66ff0)  [= 90 kHz-Timestamp: 6:52:31.0810]
                   ==> PTS: 2227615296 (0x84c6b640)  [= 90 kHz-Timestamp: 6:52:31.2810]
                   ==> PTS: 2227633296 (0x84c6fc90)  [= 90 kHz-Timestamp: 6:52:31.4810]
                   ==> PTS: 2227644096 (0x84c726c0)  [= 90 kHz-Timestamp: 6:52:31.6010]
                   ==> PTS: 2227662096 (0x84c76d10)  [= 90 kHz-Timestamp: 6:52:31.8010]
                   ==> PTS: 2227680096 (0x84c7b360)  [= 90 kHz-Timestamp: 6:52:32.0010]
                   ==> PTS: 2227698096 (0x84c7f9b0)  [= 90 kHz-Timestamp: 6:52:32.2010]
                   ==> PTS: 2227723296 (0x84c85c20)  [= 90 kHz-Timestamp: 6:52:32.4810]
                   ==> PTS: 2227734096 (0x84c88650)  [= 90 kHz-Timestamp: 6:52:32.6010]
                   ==> PTS: 2227752096 (0x84c8cca0)  [= 90 kHz-Timestamp: 6:52:32.8010]
                   ==> PTS: 2227770096 (0x84c912f0)  [= 90 kHz-Timestamp: 6:52:33.0010]
                   ==> PTS: 2227791696 (0x84c96750)  [= 90 kHz-Timestamp: 6:52:33.2410]
                   ==> PTS: 2227827696 (0x84c9f3f0)  [= 90 kHz-Timestamp: 6:52:33.6410]
                   ==> PTS: 2227849296 (0x84ca4850)  [= 90 kHz-Timestamp: 6:52:33.8810]
                   ==> PTS: 2227867296 (0x84ca8ea0)  [= 90 kHz-Timestamp: 6:52:34.0810]
                   ==> PTS: 2227892496 (0x84caf110)  [= 90 kHz-Timestamp: 6:52:34.3610]
                   ==> PTS: 2227899696 (0x84cb0d30)  [= 90 kHz-Timestamp: 6:52:34.4410]
                   ==> PTS: 2227928496 (0x84cb7db0)  [= 90 kHz-Timestamp: 6:52:34.7610]
The rest of it is left as an exercise for the reader.
 
a bit of research indicates that it is in the data stream (sorry Black Hole!) but it is done as a sort of timed overlay using the old teletext system. There is some sort of time base and the overlays are shown relative to that. So it is the timebase that must be varying - which was my suspicion but assumed there was ACII text involved. Next I need to understand what causes the wandering and whether the base is start of recording or relative to a later point (virtual bookmark or adverts or something)
The merging of these 2 images would almost certainly be done in HW. Now it could be down to the HW if it sent a 'batch' of timed merges and the HW lost (or ignored) its time base; however that raises the Q of why some channels seem reliable and others not
 
Here's something I just extracted from a 32s recording of a filtered .ts file:
Code:
$ dvbsnoop -s ts -if subs.ts -tssubdecode|grep Timestamp
                   ==> PTS: 2225001696 (0x849ed4e0)  [= 90 kHz-Timestamp: 6:52:02.2410]
                   ==> PTS: 2225008896 (0x849ef100)  [= 90 kHz-Timestamp: 6:52:02.3210]
                   ==> PTS: 2225131296 (0x84a0cf20)  [= 90 kHz-Timestamp: 6:52:03.6810]
                   ==> PTS: 2225138496 (0x84a0eb40)  [= 90 kHz-Timestamp: 6:52:03.7610]
                   ==> PTS: 2225325696 (0x84a3c680)  [= 90 kHz-Timestamp: 6:52:05.8410]
                   ==> PTS: 2225332896 (0x84a3e2a0)  [= 90 kHz-Timestamp: 6:52:05.9210]
                   ==> PTS: 2225520096 (0x84a6bde0)  [= 90 kHz-Timestamp: 6:52:08.0010]
                   ==> PTS: 2225527296 (0x84a6da00)  [= 90 kHz-Timestamp: 6:52:08.0810]
                   ==> PTS: 2225912496 (0x84acbab0)  [= 90 kHz-Timestamp: 6:52:12.3610]
                   ==> PTS: 2225919696 (0x84acd6d0)  [= 90 kHz-Timestamp: 6:52:12.4410]
                   ==> PTS: 2226106896 (0x84afb210)  [= 90 kHz-Timestamp: 6:52:14.5210]
                   ==> PTS: 2226114096 (0x84afce30)  [= 90 kHz-Timestamp: 6:52:14.6010]
                   ==> PTS: 2226445296 (0x84b4dbf0)  [= 90 kHz-Timestamp: 6:52:18.2810]
                   ==> PTS: 2226452496 (0x84b4f810)  [= 90 kHz-Timestamp: 6:52:18.3610]
                   ==> PTS: 2226510096 (0x84b5d910)  [= 90 kHz-Timestamp: 6:52:19.0010]
                   ==> PTS: 2226535296 (0x84b63b80)  [= 90 kHz-Timestamp: 6:52:19.2810]
                   ==> PTS: 2226600096 (0x84b738a0)  [= 90 kHz-Timestamp: 6:52:20.0010]
                   ==> PTS: 2226625296 (0x84b79b10)  [= 90 kHz-Timestamp: 6:52:20.2810]
                   ==> PTS: 2226636096 (0x84b7c540)  [= 90 kHz-Timestamp: 6:52:20.4010]
                   ==> PTS: 2226657696 (0x84b819a0)  [= 90 kHz-Timestamp: 6:52:20.6410]
                   ==> PTS: 2226675696 (0x84b85ff0)  [= 90 kHz-Timestamp: 6:52:20.8410]
                   ==> PTS: 2226704496 (0x84b8d070)  [= 90 kHz-Timestamp: 6:52:21.1610]
                   ==> PTS: 2226877296 (0x84bb7370)  [= 90 kHz-Timestamp: 6:52:23.0810]
                   ==> PTS: 2226898896 (0x84bbc7d0)  [= 90 kHz-Timestamp: 6:52:23.3210]
                   ==> PTS: 2226967296 (0x84bcd300)  [= 90 kHz-Timestamp: 6:52:24.0810]
                   ==> PTS: 2226985296 (0x84bd1950)  [= 90 kHz-Timestamp: 6:52:24.2810]
                   ==> PTS: 2226992496 (0x84bd3570)  [= 90 kHz-Timestamp: 6:52:24.3610]
                   ==> PTS: 2227093296 (0x84bebf30)  [= 90 kHz-Timestamp: 6:52:25.4810]
                   ==> PTS: 2227118496 (0x84bf21a0)  [= 90 kHz-Timestamp: 6:52:25.7610]
                   ==> PTS: 2227136496 (0x84bf67f0)  [= 90 kHz-Timestamp: 6:52:25.9610]
                   ==> PTS: 2227154496 (0x84bfae40)  [= 90 kHz-Timestamp: 6:52:26.1610]
                   ==> PTS: 2227233696 (0x84c0e3a0)  [= 90 kHz-Timestamp: 6:52:27.0410]
                   ==> PTS: 2227298496 (0x84c1e0c0)  [= 90 kHz-Timestamp: 6:52:27.7610]
                   ==> PTS: 2227395696 (0x84c35c70)  [= 90 kHz-Timestamp: 6:52:28.8410]
                   ==> PTS: 2227406496 (0x84c386a0)  [= 90 kHz-Timestamp: 6:52:28.9610]
                   ==> PTS: 2227431696 (0x84c3e910)  [= 90 kHz-Timestamp: 6:52:29.2410]
                   ==> PTS: 2227449696 (0x84c42f60)  [= 90 kHz-Timestamp: 6:52:29.4410]
                   ==> PTS: 2227467696 (0x84c475b0)  [= 90 kHz-Timestamp: 6:52:29.6410]
                   ==> PTS: 2227579296 (0x84c629a0)  [= 90 kHz-Timestamp: 6:52:30.8810]
                   ==> PTS: 2227597296 (0x84c66ff0)  [= 90 kHz-Timestamp: 6:52:31.0810]
                   ==> PTS: 2227615296 (0x84c6b640)  [= 90 kHz-Timestamp: 6:52:31.2810]
                   ==> PTS: 2227633296 (0x84c6fc90)  [= 90 kHz-Timestamp: 6:52:31.4810]
                   ==> PTS: 2227644096 (0x84c726c0)  [= 90 kHz-Timestamp: 6:52:31.6010]
                   ==> PTS: 2227662096 (0x84c76d10)  [= 90 kHz-Timestamp: 6:52:31.8010]
                   ==> PTS: 2227680096 (0x84c7b360)  [= 90 kHz-Timestamp: 6:52:32.0010]
                   ==> PTS: 2227698096 (0x84c7f9b0)  [= 90 kHz-Timestamp: 6:52:32.2010]
                   ==> PTS: 2227723296 (0x84c85c20)  [= 90 kHz-Timestamp: 6:52:32.4810]
                   ==> PTS: 2227734096 (0x84c88650)  [= 90 kHz-Timestamp: 6:52:32.6010]
                   ==> PTS: 2227752096 (0x84c8cca0)  [= 90 kHz-Timestamp: 6:52:32.8010]
                   ==> PTS: 2227770096 (0x84c912f0)  [= 90 kHz-Timestamp: 6:52:33.0010]
                   ==> PTS: 2227791696 (0x84c96750)  [= 90 kHz-Timestamp: 6:52:33.2410]
                   ==> PTS: 2227827696 (0x84c9f3f0)  [= 90 kHz-Timestamp: 6:52:33.6410]
                   ==> PTS: 2227849296 (0x84ca4850)  [= 90 kHz-Timestamp: 6:52:33.8810]
                   ==> PTS: 2227867296 (0x84ca8ea0)  [= 90 kHz-Timestamp: 6:52:34.0810]
                   ==> PTS: 2227892496 (0x84caf110)  [= 90 kHz-Timestamp: 6:52:34.3610]
                   ==> PTS: 2227899696 (0x84cb0d30)  [= 90 kHz-Timestamp: 6:52:34.4410]
                   ==> PTS: 2227928496 (0x84cb7db0)  [= 90 kHz-Timestamp: 6:52:34.7610]
The rest of it is left as an exercise for the reader.
Ignore that - duh!
 
Last edited:
I recorded a .ts file on the Humax, decrypted it, transferred it to my Pi (but it could have been my main Linux PC), converted it from 192 to 188 byte packets, worked out what the subtitles PID was (from the PAT and PMT, or just using experience from the PIDs in the .ts), filtered only those packets out of the original .ts into subs.ts, then ran the command as shown. Simples.
 
There is some sort of time base and the overlays are shown relative to that.
So, perhaps stating the obvious, as later or other makes of boxes are able to handle this correctly the problem must be a feature of the older Humax code systems.
 
Last edited:
So, perhaps stating the obvious, as later or other makes of boxes are able to handle this correctly the problem must be a feature of the older Humax code.
Is that obvious? Is it not at least as likely that the hardware is different on those boxes?
 
Is that obvious? Is it not at least as likely that the hardware is different on those boxes?
Regardless of whether the problem is in the humax hardware or software I think that is unlikely that we can modify the humaxes playback of subtitles from the custom firmware environment,

If VLC can handle the subtitles correctly a possible solution might be to investigate subtitles on the video players built into smart TVs or casting VLC output to a smart TV / Fire / Roku / Now TV stick
 
I think that is unlikely that we can modify the humaxes playback of subtitles from the custom firmware environment,
Agreed, but the OP wants to reverse-engineer the Humax firmware, not be constrained by the existing CF environment.
 
I decrypt my recordings on the HDR. If I wish to watch with synced subtitles I use VLC on another device to playback.
I should have said Kodi and VLC.They both show DVB subtitles in sync.

Some recent threads regarding subtitles

Which suggests a rough order from worst to best as
HDR FOX T2 & 1010S
HDR 1800T/2000T
HDR 5000T
FVP 4000T/5000T, Aura, Manhatten T-3R
 
Last edited:
Which suggests a rough order from worst to best as
HDR FOX T2 & 1010S
HDR 2000T
HDR 5000T
Aura, Manhatten T-3R
I assume by HDR 5000T you mean FVP-5000T (and the FVP-4000T) and my experience was that its subtitle playing was at the same level as the Aura.
 
Apparently I already exhausted all the ideas I had about the unsynced subtitles issue (3).

When the latest or next speech-to-subtitle system using some generative "AI" tech turns out to be prone to rewrite the dialogue
as it goes along, it won't matter anyway.

Regarding (4), that's eminently achievable with a command-line SMTP client. I used to use blat for a similar application, but others exist and one would pick the least broken one that is easy build for the HD/R platform (or just write one in Jim TCL, since the protocol is not too complex).

Then you need:
  • critically, a mail server to which your SMTP will be sent (many people will have an account with one or many email addresses provided by their ISP)
  • some configuration that controls the notification process and which sender address and SMTP server to use when email notifications are enabled
  • a WebIf settings hook to set up the configuration.
 
There is already an ssmtp package which provides a simple sender (too simple as port 25 mail seems to be disappearing due to the spam problem) but it still seems to work for me, once you get your head round its configuration.
 
  • Like
Reactions: /df
IIUC email destination servers now check the originator of the email before they decide whether to accept it. Would that not be a barrier to just sending ad-hoc emails?
 
Apparently I already exhausted all the ideas I had about the unsynced subtitles issue (3).

When the latest or next speech-to-subtitle system using some generative "AI" tech turns out to be prone to rewrite the dialogue
as it goes along, it won't matter anyway.

Regarding (4), that's eminently achievable with a command-line SMTP client. I used to use blat for a similar application, but others exist and one would pick the least broken one that is easy build for the HD/R platform (or just write one in Jim TCL, since the protocol is not too complex).

Then you need:
  • critically, a mail server to which your SMTP will be sent (many people will have an account with one or many email addresses provided by their ISP)
  • some configuration that controls the notification process and which sender address and SMTP server to use when email notifications are enabled
  • a WebIf settings hook to set up the configuration.
ta for all that!
sort of agreed on the ST issue (I had read your thread) but I still like to understand why something is impossible - already learnt quite a bit about it that may have been obvious to others but wasn't to me
I have yet to figure out if htere is even a standard procedure for notifying the alerts that appear in the banner so I can tee it off there
I run a mail server here so no probs there ;-)
I appreciate the config issue - 1st crack your walnut :) I rather assumed the WebIf hooks are rich enough for that once it gets beyond the hard wiring
 
I have yet to figure out if htere is even a standard procedure for notifying the alerts that appear in the banner so I can tee it off there
That is the easy bit
from Jim code you would write system notify "$msg"
the system class can be found in /mod/webif/lib/system.class
It writes the msg to /mod/tmp/notify.log which is read and the banner displayed whenever a web page is generated (using more class calls)

Theoretically you could modify the system notify procedure to generate an email every time a message is notified but that might genereate too many emails.
It might be easier to write a standalone programme that periodiaclly (triggered by crontab) that reads /mod/tmp/notify.log and sends an email for all outstanding messages that haven't yet been acknowledged on a web page.

It would be nice if this could be done via the RS server to avoid the complexities of configuring smtp but we dont currently have the ability to modify code on the server.
 
Last edited:
A simple, and I think valuable, change would be to also post notify messages, to rs log and /mod/tmp/activity.log

Using rs log would allow you to see the notify messages anywhere in the world if on holiday or remotely monitoring for an elderly relative.
/mod/tmp/activity.log would provide a longer term record of the notify messages which currently disappear without trace once the banner is acknowledged. Unless you think to take a screen shot it can be difficult to remember the details of the alert box later.
 
That is the easy bit
from Jim code you would write system notify "$msg"
the system class can be found in /mod/webif/lib/system.class
It writes the msg to /mod/tmp/notify.log which is read and the banner displayed whenever a web page is generated (using more class calls)

Theoretically you could modify the system notify procedure to generate an email every time a message is notified but that might genereate too many emails.
It might be easier to write a standalone programme that periodiaclly (triggered by crontab) that reads /mod/tmp/notify.log and sends an email for all outstanding messages that haven't yet been acknowledged on a web page.

It would be nice if this could be done via the RS server to avoid the complexities of configuring smtp but we dont currently have the ability to modify code on the server.
where poss, I'd like to make any mods local if poss (for my own benefit) and echo to the RS if reqd. Partly because I haven't yet tasted the RS and partly it is not always available - us wot live in the sticks often lose power and connectivity (power cover is easy; have lots of UPS, connectivity is harder/more expensive)
 
Even if you don't have a mail server, it seems like sSMTP can manage if you set its config file like this GMail example:
Code:
root=myemailaddress@gmail.com
mailhub=smtp.gmail.com:587
AuthUser=mygmailusername
AuthPass=mypassword
UseSTARTTLS=YES
Since you'd be sending the email through your known server (if you are), mail sent across the internet won't be spam-hammered.

I think blat's command-line interface is much more straightforward but sSMTP is trying to to be a drop-in for sendmail and blat is only a Windows thing (and C++, which is normally bad for on-box compilation at least). A shell wrapper could be used, or there's smtp-cli, which looks like a Linux equivalent but is written in Perl, so in principle it can be run except that Perl programs typically import a whole load of modules that aren't included in the CF Perl and/or have some unbuildable dependency, and here it is, the first significant code line:
Code:
# perl
use strict;
use IO::Socket::INET;
Can't locate loadable object for module IO in @INC (@INC contains: /mod/lib/perl5/5.10.0/mipsel-linux /mod/lib/perl5/5.10.0 /mod/lib/perl5/site_perl/5.10.0/mipsel-linux /mod/lib/perl5/site_perl/5.10.0 .) at /mod/lib/perl5/5.10.0/mipsel-linux/IO/Handle.pm line 263
Compilation failed in require at /mod/lib/perl5/5.10.0/mipsel-linux/IO/Handle.pm line 263.
BEGIN failed--compilation aborted at /mod/lib/perl5/5.10.0/mipsel-linux/IO/Handle.pm line 263.
Compilation failed in require at /mod/lib/perl5/5.10.0/mipsel-linux/IO/Socket.pm line 11.
BEGIN failed--compilation aborted at /mod/lib/perl5/5.10.0/mipsel-linux/IO/Socket.pm line 11.
Compilation failed in require at /mod/lib/perl5/5.10.0/mipsel-linux/IO/Socket/INET.pm line 11.
BEGIN failed--compilation aborted at /mod/lib/perl5/5.10.0/mipsel-linux/IO/Socket/INET.pm line 11.
Compilation failed in require at - line 2.
BEGIN failed--compilation aborted at - line 2.
#
(A similar problem exists with Python, where the SSL module is present but out-of-date and hasn't been rebuilt).

But only ~950 lines of unusually readable Perl, so quite suitable for translation to Jim.

There are other interesting things in /mod/tmp/activity.log (probably should have been .../var/...). Maybe we need a rsyslog-ish arrangement, which would allow some remote syslog server to select and send messages to email ("-ish arrangement" because the standard rsyslog daemon might be too big/hard to build).
 
Back
Top