Custom firmware and Ireland Saorview

Black Hole · Feb 15, 2013

The EPG data prioritises data for the current channel and multiplex, and for today, transmitting that data more frequently than other multiplexes far into the future. It takes 15 mins to get the whole data set.

Luke · Feb 16, 2013

Where do get the 15 minutes figure from?

Black Hole · Feb 16, 2013

We established it a long time ago, I believe from the specifications. The EPG refresh wake-up period was set because of it, but me being an engineer said 20 mins instead of 15 so there is no chance of missing a bit.

Luke · Feb 17, 2013

How was it established?

Black Hole · Feb 17, 2013

Dunno. It was a long time ago. I'll see if I can find it (might have been a different forum).

dma · Feb 17, 2013

I'm not sure if this helps, but I can see my /tmp/epgd.log updating approx every 16 mins:

Code:

humax# ls -l /tmp/epgd.log
-rw-rw-rw-    1 root     root          1223 Feb 17 14:18 /tmp/epgd.log
 
humax# ls -l /tmp/epgd.log
-rw-rw-rw-    1 root     root          1311 Feb 17 14:34 /tmp/epgd.log

Looking inside the log you can see it generates a "last" time (epoch time - number of secs since 1-Jan-1970) during each epg update:

Code:

humax# cat /tmp/epgd.log
epgd: DB time 1361060739, last 0
epgd: Regenerating.
epgd: Done in 20 seconds.
epgd: DB time 1361099625, last 1361060739
epgd: Regenerating.
epgd: Done in 19 seconds.
epgd: DB time 1361100430, last 1361099625
epgd: Regenerating.
epgd: Done in 19 seconds.
epgd: DB time 1361101420, last 1361100430
epgd: Regenerating.
epgd: Done in 18 seconds.
epgd: DB time 1361102365, last 1361101420
epgd: Regenerating.
epgd: Done in 20 seconds.
epgd: DB time 1361103198, last 1361102365
epgd: Regenerating.
epgd: Done in 18 seconds.
epgd: DB time 1361104259, last 1361103198
epgd: Regenerating.
epgd: Done in 18 seconds.
epgd: DB time 1361105100, last 1361104259
epgd: Regenerating.
epgd: Done in 19 seconds.
epgd: DB time 1361105942, last 1361105100
epgd: Regenerating.
epgd: Done in 19 seconds.
epgd: DB time 1361107012, last 1361105942
epgd: Regenerating.
epgd: Done in 18 seconds.
epgd: DB time 1361107899, last 1361107012
epgd: Regenerating.
epgd: Done in 18 seconds.
epgd: DB time 1361108766, last 1361107899
epgd: Regenerating.
epgd: Done in 18 seconds.
epgd: DB time 1361109729, last 1361108766
epgd: Regenerating.
epgd: Done in 22 seconds.
epgd: DB time 1361110558, last 1361109729
epgd: Regenerating.
epgd: Done in 22 seconds.
epgd: DB time 1361111594, last 1361110558
epgd: Regenerating.
epgd: Done in 23 seconds.

If you subtract the "last" time from each update you get this approximation:

1361109729 - 1361108766 = 963 (16.05 mins)
1361108766 - 1361107899 = 867 (14.45 mins)
1361107899 - 1361107012 = 887 (14.78 mins)
1361107012 - 1361105942 = 1070 (17.83 mins)

So it looks like the update is anywhere between 14 and 18 mins. The variance might be due to variable processing time of each epg update, and I guess what the Humax box is doing at the time (maybe resources are being diverted for recording, transferring/streaming, etc...)

I'd say you're right by picking a conservative figure of 20 minutes.

Hope this helps.

Black Hole · Feb 17, 2013

Thanks for that. I will continue to try to track down the specification data.

af123 · Feb 17, 2013

dma said:
I'm not sure if this helps, but I can see my /tmp/epgd.log updating approx every 16 mins:

I'm afraid that just the fact that the EPG daemon only checks for EPG database updates every 15 minutes.
The log does show, though, that the database has updated every time it checks.

Black Hole said:
Thanks for that. I will continue to try to track down the specification data.

http://www.etsi.org/deliver/etsi_en/300400_300499/300468/01.11.01_60/en_300468v011101p.pdf

stripts already tells us what percentage of packets in a broadcast stream contain Event Information Table (EIT) data - it could be modified to dump out the timestamps on those I suppose..

Border · Feb 18, 2013

Black Hole said:
If the database file was in the same partition the updates would be written to the same file area (or some unused space if the file needed to expand). Keeping the file in its own partition means it shouldn't suffer from fragmentation.

The EPG is stored on the separate 1GB partition. Is that partition divided further and specific areas allocated to a channel number?

For example
First MB : Channel 1
Second MB : Channel 2
Third MB : Channel 3

and so on

800 MB : Channel 800.

If this was the case would it mean that, for example, when the Guide is on Channel 4 and updating the CH4 EPG (4th Channel on EPG) that it writes the data to the 4th MB sector of the EPG partition.

Is there a way to see if this is happening this way?

Black Hole · Feb 18, 2013

I think you'll find it is stored as an SQLite database - that's how the WebIF extracts the content. There are tools in the packages for performing SQL queries.

dma · Feb 18, 2013

Yeah, plus you get a sqlite3 binary which lets you query the database.

For example, you can load up the epg.db database file:

Code:

humax# sqlite3 /mnt/hd1/epg.db
SQLite version 3.7.5
Enter ".help" for instructions
Enter SQL statements terminated with a ";"

It only has one table when I list them with ".tables" - it's called epg:

Code:

sqlite> .tables
epg

You can then see what columns are in this table:

Code:

sqlite>pragma table_info(epg);
0|service_id|integer|0||0
1|event_id|integer|0||0
2|start|integer|0||0
3|end|integer|0||0
4|duration|integer|0||0
5|name|text|0||0
6|text|text|0||0
7|warning|text|0||0
8|content_type|integer|0||0
9|content|text|0||0
10|event_crid|text|0||0
11|series_crid|text|0||0
12|rec_crid|text|0||0

...and you can search. In this example I'm looking for a program with "Professionals" in the name and displaying the start time, end time and duration. You'll notice satrt/end is in epoch (number of seconds since 1-Jan-1970) and duration is also in seconds:

Code:

sqlite> select name, start, end, duration from epg where name like('%Professionals%');
The Professionals|1361168100|1361171100|3000
The Professionals|1361184300|1361188200|3900
The Professionals|1361199300|1361202900|3600
The Professionals|1361236200|1361239500|3300
The Professionals|1361253600|1361256600|3000
The Professionals|1361270100|1361274000|3900
The Professionals|1361285700|1361289300|3600
The Professionals|1361340900|1361343900|3000
The Professionals|1361372100|1361375700|3600
The Professionals|1361427300|1361430300|3000
The Professionals|1361459100|1361462700|3600
The Professionals|1361513700|1361516400|2700
The Professionals|1361530200|1361533800|3600
The Professionals|1361544900|1361548500|3600
The Professionals|1361585100|1361588400|3300
The Professionals|1361600700|1361603700|3000
The Professionals|1361621100|1361624700|3600
The Professionals|1361804400|1361808000|3600

So Border, you could write your own software to query the EPG (and even change entries if you wished)!

af123 · Feb 18, 2013

Black Hole said:
I think you'll find it is stored as an SQLite database - that's how the WebIF extracts the content. There are tools in the packages for performing SQL queries.

Nearly : )
The Humax software periodically dumps a copy of the EPG data in raw form* to a file called /mnt/hd1/dvbepg/epg.dat which is in that partition. The epg package, which is part of the custom firmware, converts that to the SQLite3 database in the same partition every 15 minutes if it has changed (which it almost always has).
The Webif uses a combination of the raw EIT data and SQLite database to provide its content.

You can use the epg utility to directly query the raw file too:

Code:

humax# epg search Professionals
----------------------------------------------------------
                    Service ID: 28032
                      Event ID: 6031
                    start_date: 0xdc15 6:15:00 (Mon Feb 18 06:15:00 2013)
                      duration: 0:50:00
                     encrypted: 0
                          Name: The Professionals
                          Text: Spy Probe: Drama series about two criminal-intelligence agents. Bodie and Doyle pose as potential hitmen to infiltrate an organisation that hires out killers. [AD,S,SL]
                       Warning: Contains acts of violence and strong language. 
                  Content Type: Undefined (15)
                    Event CRID: /558448
                   Series CRID: /418207
... more entries follow ...

* The EIT extracted from the broadcast stream.

Black Hole · Feb 18, 2013

So Border might be right in the way the EIT data is partitioned then.

I'm pretty confident of my original assertion: the process is unlikely to bypass the file system and use raw disk access, so it is probably set up as a random (as opposed to serial) file access (and, if it were on a shared partition, the OS would be perfectly capable of managing it).

af123 · Feb 18, 2013

It's just a file on a general purpose filesystem. I suspect the Humax software uses an in-memory copy of the data and just dumps it periodically so that it can reload it at startup in order that the guide works straight away.

The HD works perfectly well without anywhere to dump the data, although it will if a suitable drive is connected.

dma · Feb 19, 2013

Yes af123, I noticed your epg utility takes the pain out of querying the epg database. It's far much easier to use than sql query.

I did also notice that when I flick between Soarview and Freeview channels, the number of rows in the EPG table stays the same. So I'm convinced all the data is there, the Humax just seems capable of displaying the on-screen EPG from only one transmitter at a time.

I was wondering that it might be possible to make a utility to display the EPG from both transmitters as, say HTML. I just don;t understand how you can get the channel numbers. The service_id seems to be mapped to a provider, e.g. BBC rather than an actual channel, e.g. BBC One.

Ezra Pound · Feb 19, 2013

It looks like each TV channel has a different service_id, e.g. Channel 5 = 8500 and Channel 5+1 = 13024 :-

Code:

humax# epg search 'Killers Behind Bars'
----------------------------------------------------------
                    Service ID: 13024
                      Event ID: 21504
                    start_date: 0xdc17 21:00:00 (Wed Feb 20 21:00:00 2013)
                      duration: 0:58:00
                    encrypted: 0
                          Name: Killers Behind Bars: The Untold...
                          Text: ...Story. Using cutting-edge criminology techniques to try and link unsolved cold-case murders to convicted monsters, Professor David Wilson looks further into the case of child killer, Robert ... [S]
                  Content Type: Social/Political/Economic (8)
                    Event CRID: /V5ZZZ
                  Series CRID: /RBBC
----------------------------------------------------------
                    Service ID: 8500
                      Event ID: 38835
                    start_date: 0xdc17 20:00:00 (Wed Feb 20 20:00:00 2013)
                      duration: 0:58:00
                    encrypted: 0
                          Name: Killers Behind Bars: The Untold...
                          Text: ...Story. Using cutting-edge criminology techniques to try and link unsolved cold-case murders to convicted monsters, Professor David Wilson looks further into the case of child killer, Robert ... [S]
                  Content Type: Social/Political/Economic (8)
                    Event CRID: /V5ZZZ
                  Series CRID: /RBBC
humax#

Border · Feb 19, 2013

EDIT ( Clutter Removed)

Could I review an earlier theory on the EPG partition (post 69 I think)

Black Hole · Feb 19, 2013

Border said:
Also on the var/lib/humaxtv/channel.db is this part of the cf

I take it that's a question. No, I don't think it's part of the CF.

af123 · Feb 19, 2013

Could I review an earlier theory on the EPG partition (post 69 I think)

No, that isn't how it works.

Each MUX transmits a complete copy of the Event Information Table (EIT) for its region periodically; this is multiplexed in with all of the channels and other tables and contains EPG data for all channels. It is this table which is dumped to disk by the Humax in raw form (the epg.dat file). This table is also used for the EPG and for the Accurate Recording (AR) feature (it watches for a flag to become set in the EIT data which indicates that the programme is running, although it could also use the RST if it is present*).

The problem with multi-region (ignoring the Saorview part at the moment and considering two different freeview regions!) is that switching channels to the other region will result in a different EIT being received. An EIT in which the service IDs for the channels are different (i.e. BBC One in region 1 will have a different service ID to BBC One in region 2)

* - It should be obvious why AR isn't reliable if you have muxes from multiple regions tuned into your box.

The gory details can be found in the link I posted earlier:

http://www.etsi.org/deliver/etsi_en/300400_300499/300468/01.11.01_60/en_300468v011101p.pdf

dma · Feb 19, 2013

You're right - I just needed to look in channel.db...

Code:

humax# sqlite3 /mnt/hd1/epg.db
SQLite version 3.7.5
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select name, service_id from epg where name like('%Death in Para%');
Death in Paradise|4221
Death in Paradise|4221
Death in Paradise|17597
Death in Paradise|17597

You can see the service_id for Death in Paradise is 4221; and if you look in channel.db, you can see that 4221 maps to BBC One:

Code:

humax# sqlite3 /var/lib/humaxtv/channel.db
SQLite version 3.7.5
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select usLcn, szSvcName, usSvcId from tbl_svc order by usLcn;
1|BBC ONE NI|4221
2|BBC TWO NI|4285
3|UTV|8276
4|Channel 4|8384
5|Channel 5|8500

Custom firmware and Ireland Saorview

May contain traces of nut

Well-Knwοn Мember

May contain traces of nut

Well-Knwοn Мember

May contain traces of nut

Member

May contain traces of nut

Administrator

Member

May contain traces of nut

Member

Administrator

May contain traces of nut

Administrator

Member

Well-Known Member

Member

May contain traces of nut

Administrator

Member