Custom firmware and Ireland Saorview

Black Hole said:
If the original .hmt file is not pure ASCII, transferring it masquerading as a .txt might not have achieved fidelity.
Click to expand...
From Ezra's post, the hmt tool seems to be able to parse it ok so I think it's fine.

There is only limited EPG data in that file and no synopsis. I would guess that either the Saorview EPG data is different from standard Freeview so the process that copies the EPG data over to the .hmt file isn't working, or the EPG cache data that the Humax is holding at the time of the recording is Freeview data so it can't find the programme to copy the data across.

It's inconsistent that the Humax is able to show the synopsis on a live programme though.

The good news is that there is some information in there that could be used to find the programme in the EPG data for retrospective population, as long as the cache file on the disk relates to Saorview.

Can you test something from the command line?

Here's a recording that my box made today and the output of the hmt command run against it.

Code: 
humax# hmt Tree\ Fu\ Tom_20130202_1726
Format:SD
Title:Tree Fu Tom
Channel:71 (CBeebies)
Folder:/mnt/hd2/My Video/Tree Fu Tom/
Filename:Tree Fu Tom_20130202_1726
Genre:Children (80)
EPG:Rickety Rescue: Animated fantasy adventure show. When the impulsive behaviour of Tom and friends gets them into trouble it's up to Rickety to leap to the rescue! [S]
Raw file is encrypted on disk.
 
Flags: SD,New,Unlimited Copies,ODEncrypted,
Copy count:0
 
Scheduled start:1359825900 (Sat Feb  2 17:25:00 2013)
Scheduled duration:1200
Recording start:1359825960 (Sat Feb  2 17:26:00 2013)
Recording end:1359827320 (Sat Feb  2 17:48:40 2013)
Duration:1360
Play resumes at: 4 seconds in.
 
Service ID (SID):4672
Transport Stream ID (TSID):4170
Originating Network ID (ONID):9018
Programme Map Table PID (PMTPID):800
Video PID:401
Audio PID:402
Bookmarks:0 =

The fields I'm interested in are Service ID and Scheduled start.
Here's an EPG search for the programme in question. Note that I've added 1 to the scheduled start time so that I only get the relevant programme.

Code: 
humax# epg -S4672 -@1359825901 dump
----------------------------------------------------------
                    Service ID: 4672
                      Event ID: 58840
                    start_date: 0xdc05 17:25:00 (Sat Feb  2 17:25:00 2013)
                      duration: 0:20:00
                    encrypted: 0
                          Name: Tree Fu Tom
                          Text: Rickety Rescue: Animated fantasy adventure show. When the impulsive behaviour of Tom and friends gets them into trouble it's up to Rickety to leap to the rescue! [S]
                  Content Type: Children (5)
                    Event CRID: /SQ0UG
                  Series CRID: /L5C9UE

Can you try this process on a new recording?
If this doesn't work then try tuning into a Saorview channel for a while and then running the EPG search again. Hopefully the synopsis should be retrievable and then it can be patched into the hmt file.

If it's practical then a package could be built which automatically patches the synopsis into the HMT file when a recording finishes.

If the EPG data cache isn't present on disk then it could still be extracted from the TS stream as long as you're prepared to decrypt each recording first. Not as straightforward though due to the way the EIT is embedded - and wouldn't work in conjunction with shrink.
 
These are the results relating to the second hmt file called sample 2
Code: 
Service ID: 1101
                      Event ID: 39852
                    start_date: 0xdc05 17:00:00 (Sat Feb  2 17:00:00 2013)
                      duration: 1:01:00
                    encrypted: 0
                          Name: Hairspray
                          Text:
                  Content Type: Undefined (0)
                    Event CRID: /46040345

I tuned onto the Saorview region watched a Saorview show. the EPG on the box was the saorview EPG. On the web application it was still the Freeview EPG. Turned off box. Webif EPG now Saorview. I have just recorded a new show and the Saorview EPG is viewable through webif.
 
Can you run it again with dumpraw instead of dump and in debug mode?

Code: 
humax# epg -d2 -S1101 -@12345... dumpraw
 
Done. Regarding the unusual packaging Blackhole mentioned in a previous post that the Humax box does not recognize the 'fada'. Because some of the program descriptions on Saorview are in Irish they may have had to use additional character sets to represent the accents on the vowels. Additional characters (small and large case) may have been required.

In Irish/Gaelic there are accents on upper and lower case vowels. This would mean a completely different character set may be required for the EPG. The accents used are
á é í ó ú Á É Í Ó Ú

Some of these may feature regularly and even the national television station has them in its name (i.e. RTÉ). There is also a channel dedicated to the Irish language (TG4) similar to S4C. As a result code in the EPG may sometimes appear like random characters or broken code for example
RTÉJr may appear like
RTÉ jr or
RTÉ ONE appears like
RTÉ ONE

This is the way Saorview entries appears in the webif EPG also.
 
Having analysed the EPG data file that you sent it seems that the Saorview EPG data stream encodes the programme synopsis within an extended event descriptor whereas the standard Freeview encodes it in a short event descriptor, along with the programme title. The Saorview data only includes the programme title in the short event descriptor.

From the freeview specification:

The extended event descriptor provides a detailed text description of an event, which may be used in addition to the short event descriptor. More than one extended event descriptor can be associated to allow information about one event greater in length than 256 bytes to be conveyed. Text information can be structured into two columns, one giving an item description field and the other the item text. A typical application for this structure is to give a cast list, where for example the item description field might be "Producer" and the item field would give the name of the producer.
Click to expand...

I have just updated the epg package to properly handle these extended event descriptors which is the first step to improving the situation for you.

Code: 
humax# epg -f saorview.dat -@1360454101 -S 1232 dump
----------------------------------------------------------
                    Service ID: 1232
                      Event ID: 10616
                    start_date: 0xdc0c 23:00:00 (Sat Feb  9 23:00:00 2013)
                      duration: 1:00:00
                     encrypted: 0
                          Name: Black Beats
                          Text: Travel through the decades with music from Motown and contemporary R&B inlcuding funk, soul and hip hop. Presented by Annabelle Brossard.
                  Content Type: Undefined (0)

Now that the epg tool can extract this field it should be possible to patch up any Saorview recordings as they complete, providing the current EPG cache file is the right one. I don't know how likely that is to be the case - some experimentation will be required.

I'll write a package that triggers whenever a recording ends and tries to patch the information so you can do more testing if you're willing. I can't do much from here unfortunately : )
 
af123 said:
Now that the epg tool can extract this field it should be possible to patch up any Saorview recordings as they complete, providing the current EPG cache file is the right one. I don't know how likely that is to be the case - some experimentation will be required.

I'll write a package that triggers whenever a recording ends and tries to patch the information so you can do more testing if you're willing. I can't do much from here unfortunately : )
Click to expand...

Thanks af123

I have installed the epg package and now the Saorview program synopsis on the webif EPG is visible. It would be good if this could be used to put the info on the saved file.

I am using 1.02.20 and CF 2.15. I thought today that maybe some of the Humax firmware updates since then may have included a fix for this. I tried for a while to update to 1.02.32 but couldnt get access so I rolled it back to 1.02.20. When I have time I will try latest version (1.02.32) to see if it manages the synopsis data.
 
I've just uploaded a test package called epgpatch. The idea is that it monitors recordings on the box and whenever a recording finishes it checks to see if it has a real synopsis against it and if not then it tries to fix it by searching the EPG database.

I'm not sure how well this will work as I can't really test it end to end here but the bits I have tested seem ok.

Can you give it a try?

To fix individual recordings, as long as it is still in the EPG cache, you can run the epgpatch command directly against it without any file extension.

Code: 
humax# /mod/sbin/epgpatch /media/My\ Video/QI\ XL_20130204_0020
Processing '/media/My Video/QI XL_20130204_0020'
No patch required.
 
I have installed epgpatch.
When the file is saved there is a program synopsis but it seems to be cut short. On one file it is 48 characters long and on another 54. This is visible when you click the i button on a selected file. However the synopsis still remains blank when the i button is double clicked during playback.
The same shortened synopsis appears in the webif when you check the program data.

Also I noticed on a manual recording where the recording went over two files that the file name was that of the program running at the start of the recording. During playback if you double clicked on the i button the name of the program shown (with no synopsis) was that of the second show.

The full program synopsis is visible on the webif EPG.

I have tried 1.02.32 and it did not make any difference to the Saorview saved programs.
 
I've uploaded a new version of epgpatch which should be better. It fixes the truncation issue and attempts to patch up the EPG information for recordings that span multiple EPG entries. You might need to reboot after installation for the new version to start working.

Can you give that one a go with a test recording?
 
epgpatch 1.01 installed and running.
When you click on the recorded file and select 'i' the full program synopsis from Saorview files is visible.
When playing back the program and double clicking on the 'i' button the synopsis is blank so I presume this process gathers the information differently.

I installed epgpatch and rebooted. For a while the EPGs from both regions Saorview and Freeview were visible. I then was going to try record simultaneously Saorview and Freeview programs.

As soon as I started to record the Saorview program the Freeview EPG disappeared.It allows recording simultaneously from both regions but only one regions EPG is now visible at a time.

I haven't been able to get both back on simultaneously. Have tried reinstalling epgpatch and rebooting. I even tried removing poweron-chanel in case that was interfering with it.

On the webif EPG only one region EPG currently showing
 
af123 thanks for the work on epgpatch. I see some parts (recmon) caused a bit of a commotion.

I have my boxes set to power on on the Saorview Chanel at 5 a.m. for a few minutes that seems to switch the epg database (DB) over to Saorview for the day allowing the cf to grab the synopsis on recorded files. I have also scheduled Freeview recordings through the day and the Humax box pulls the short series descriptor as normal from the signal even though the epg db is presumably still set to Saorview.

If I did it the other way around (i.e. Freeview wakeup and Freeview epg in humax database all day) the Saorview recordings remain blank synopsis as epg is Freeview and the epgpatch cant use it.

On another note regarding the EPG and multiple regions (or countries). It seems that when you select guide and change channel the EPG is updated from the current broadcast signal. The humax presumably goes through all the channels and over writes them with the new data. If you are on the same region (i.e. Freeview) then you dont notice the difference but switching to Saorview , wipes the epg database and rewrites it with the new channel information. The humax has a 1gb partition specifically for the epg. If it wasn't separate this process would risk overwriting areas where programs were stored.

When I view one country's EPG in the guide and then unplug the cable the picture in the small window freezes and the EPG remains populated.

Changing channel to another Saorview channel then causes the picture in the small window to go blank.

Changing to BBC1 on Freeview with the cable still unplugged still gives a blank signal with the Saorview EPG.

I then plug back in the cable and get live picture in the small window of BBC1, but the Saorview EPG remains.

It is only when I then change channel that the Freeview EPG populates.

So my conclusion is that on channel changes the box updates its epg, clearing the previous cache and rewriting all channels.

Black hole has said in another post
I doubt there is anything that could be done, since it is unlikely the Humax has been designed to accommodate cross-border operation (as is evidenced by its failure to buffer EPGs).
Click to expand...

Anyway an excellent box for Freeview and Saorview.

If you notice why Saorview are in the 800's that is because I manually tune- Tuning in the Freeview frequencies first (starting at 49 or 50 up to mid 60's) and then tune in Saorview (frequency 45).


Edit 19/04/2013: If you use the Humax HDR to record programs for multiple regions (e.g. Freeview/ Saorview) you must (in my non technical opinion) use padded recordings. ( See Padded vs Accurate Recordings). There may be other information on why not to tune to multiple regions. If you use 'AR' you will miss recordings. End of Edit.
 
Congratulations, you seem to have gone a long way to solving (or at least finding work-arounds for) the problems.

Border said:
...If it wasn't separate this process would risk overwriting areas where programs were stored.
Click to expand...

Just for accuracy, this isn't really true. The operating system keeps files separate from each other, otherwise you might expect recordings to overwrite other recordings. If the database file was in the same partition the updates would be written to the same file area (or some unused space if the file needed to expand). Keeping the file in its own partition means it shouldn't suffer from fragmentation.
 
Thanks folks, this is great. There is no seamless saorview and freeview/freesat solution available that I know of without paying either Sky or UPC a subscription. However, for the two million or so of us that can get signal overspill this is not a bad compromise. Looking forward to trying it out (once I fix my broken UK antenna:().
 
There is probably still a bit more to be done in this area via the custom firmware. For example:

Monitor the EPG database and create backup copies of the Saorview data so that there is always a recent copy there for epgpatch to use;
Extract EPG data from a decrypted recording to patch up the synopsis data - could be used if epgpatch misses one.
Look at the possibility of creating merged EPG data - at least for the web interface if not the standard Guide..
 
I had a few thoughts,

af123 said:
Monitor the EPG database and create backup copies of the Saorview data so that there is always a recent copy there for epgpatch to use;
Click to expand...

This information is already in the Schedule when the programs are put into the schedule. Why not backup the EPG data before the recording is made. (Don't worry about unscheduled recordings as when you decide to record a live show then epgpatch will pull the synopsis when it finishes as the live EPG will be the same as the program being recorded.)

If you record a scheduled show and the information is not in the current EPG then why not check the EPG schedule backup file.

Look at the possibility of creating merged EPG data - at least for the web interface if not the standard Guide..
Click to expand...

Lets call Freeview the Primary EPG and Saorview the secondary EPG (In Black Holes case the Primary EPG might be West Region and the Secondary EPG the Wales Region).

I have the channels from Saorview stored in the 800's and Freeview Starting from channel 1.
Starting with webif there is a drop down dialog box in settings with 'Channel Group for EPG' how about changing this to primary EPG and Secondary EPG.
Then could there be a package that could wake up the box at say 5 a.m. on channel 800 (one of the 'secondary' channels ) and update the Secondary EPG and then power down and wake up again at 5.15 a.m. and update the Primary EPG on Channel 1.

Combine both these EPGS for webif. Perhaps with an optional button 'update primary/secondary EPG now' for use during the day if necessary.

I could see problems with this on the box. Who knows how long it takes for the EPG to repopulate/update after channel change. (10 secs 20 secs 3 minutes ?). If this was developed the process would have to wait 10,20 secs or 3 minutes after a channel change before merging that days EPGS and then putting them in place of the current database. Perhaps it is something that might come in Future Humax firmware updates (alas unlikely I fear) .

Could something like this be used to merge two regions of Freeview ?
 
For information, the Freeview EPG cycles every 15 mins or so.

It would certainly be interesting if multi-region could be solved like that, but I am struggling to think of all the implications.
 
Black Hole said:
For information, the Freeview EPG cycles every 15 mins or so.
Click to expand...

Black Hole do you mean it takes one channel 15 Minutes to get the full 7 day EPG. What about todays EPG. Is that cycled more frequently. And are the cycles at different stages on each MUX. (that would make sense).

For example switching Regions I clear the EPG (I presume) I then change over to another region (BBC 3 for Example). I find to-days EPG populates for most channels pretty quick. Skip forward 7 days, after 45 seconds most channels on that MUX are populated, others remain blank.

Go back to current time on Guide (changing channel when viewing guide in 7 days doesn't change chanel) and change channel to another MUX. That mux 7 day guide then updates instantly.

Would it take 15 minutes to up date the full MUX if viewing one channel. Perhaps each Mux broadcasts to-days EPG very frequently, the 7 day EPG for channels on its mux frequently and then other channels spaced out over the 15 minutes but less frequently. Switching channels or moving up and down the Guide seems to speeds up the update of the EPG. Does that make sense?
 
You must log in or register to reply here.
Back
Top