epg data source

[darkflare]

Hi guys,

I like the web epg however its written in a language I just do not understand! Im more comfortable with php, html etc.

With this in mind I want to create a little interface to the humax which displays epg information in a filtered format of my own choosing. Is the epg info stored in a database? and how are items written to the schedule for recording?

Thanks

Alec

 

[darkflare]

I have located the epg database and some tools for reading the epg.

the database is located at /mnt/hd1/epg.dat it looks like its sqlite which is good.

there is also a cli interface over telnet called epg. more here: http://wiki.hummy.tv/wiki/Custom_Firmware_Package_Notes#EPG_.28Command_Line.29

 

[Black Hole]

We know it's sqlite, which is why there are developer tools in the custom packages.

 

[af123]

/mnt/hd1/epg.dat isn't SQLite, it's a dump of the raw broadcast stream. The epg utility is able to parse it and it also creates an SQLite formatted EPG database on the way.

 

[darkflare]

We know it's sqlite, which is why there are developer tools in the custom packages.

You might know but as an external developer with no documentation to consult and no answer to my question I decided to reply so that others might know what I thought I had discovered. (Which turned out to be incorrect! ha!)

/mnt/hd1/epg.dat isn't SQLite, it's a dump of the raw broadcast stream. The epg utility is able to parse it and it also creates an SQLite formatted EPG database on the way.

Thats interesting, I think I meant the epg.db thank you for clearing that up! Does the current web interface call the EPG utility in the background to generate the database file or is it on a timer of some description? If so how often does it get refreshed?

Also great work af123 on the epg utility! It's amazing that it parses the raw stream and produces the sql database, thank you!

Whilst digging through the code I found /var/lib/humaxtv/rsv.db . It looks like this is the recording schedule database. In order to schedule a recording do you add a row to the database and then reset the box so that it reads the db again or is the rsv.db a readable copy of some more complex system that handles the reservations?

Thank you for taking the time to answer these questions, I appreciate it. I apologise for not being able to understand the code in mongoose but JIM is not something ive ever worked with!

Alec

 

[af123]

Does the current web interface call the EPG utility in the background to generate the database file or is it on a timer of some description? If so how often does it get refreshed?

A daemonised copy of the epg utility is started at boot time and is responsible for keeping the SQLite version of the EPG database up-to-date. It checks every 15 minutes to see if the raw datafile has been updated and if so it generates a new SQLite version - the epgd.log file shows you this activity.

The epg class (/mod/var/mongoose/lib/epg.class) has methods in it which use the SQLite database if it's there but fall back to calling the epg utility directly otherwise.

Whilst digging through the code I found /var/lib/humaxtv/rsv.db . It looks like this is the recording schedule database. In order to schedule a recording do you add a row to the database and then reset the box so that it reads the db again or is the rsv.db a readable copy of some more complex system that handles the reservations?

That is the recording schedule database that is maintained directly by the Humax software. If you add a row to it then it will just be overwritten by the Humax software unless you force an abrupt(unclean) restart immediately. The rsvsync package works around this by only modifying the database while the Humax software is not running.

 

[gebbly]

Hi all, I was thinking of a little pet project involving the EPG as well and found this discussion quite interesting. AF123 could you elaborate a little on the rsvsync please?

Will any 3rd party entries in the recording schedule database be over written by the humax when it next attempts to place an entry itself or is there some other mechanism which triggers the overwrite of the new entry?

I assume the only time the "Humax software is not running" is when the unit is in standby, hence that is when rsvsync is able to make the changes? If that is the case where are 3rd party recording schedules stored whilst waiting for rsvsync to apply them to rsv.db?

I havent quite got my head around how I can get a recording scheduled correctly yet.

thanks for any help

 

[Ezra Pound]

I assume the only time the "Humax software is not running" is when the unit is in standby, hence that is when rsvsync is able to make the changes?

The changes would not be made in standby as the main CPU isn't running, They would be made at boot-up in a process that occurs before the humaxtv process starts

 

[af123]

Will any 3rd party entries in the recording schedule database be over written by the humax when it next attempts to place an entry itself or is there some other mechanism which triggers the overwrite of the new entry?

The Humax software holds the schedule in memory and periodically writes it to the database file, replacing whatever is in there. I haven't spent any time looking at how often it does it but if you make a change to the database and wait 5 minutes it will probably be gone. It also writes it to disk during shutdown.

I assume the only time the "Humax software is not running" is when the unit is in standby, hence that is when rsvsync is able to make the changes? If that is the case where are 3rd party recording schedules stored whilst waiting for rsvsync to apply them to rsv.db?

rsvsync makes the changes during boot (from full standby) before the Humax software starts. It's hooked into the xinit framework that the CFW adds, see /etc/init.d/S80xinit. The pending reservations are held in a table called pending in rsvp.db which is created and maintained by the web interface classes. It's generally a lot easier to use the existing Jim library classes and methods.

Here's a very simple script that uses the existing libraries to schedule a series recording for the first "Silent Witness" event it finds in the EPG:

#!/mod/bin/jimsh
 
# Initialise the environment
source /mod/var/mongoose/lib/setup
 
# Load classes
require epg.class rsv.class system.class
 
# Find all matching events. This is the simplest use of the
# {epg fetch} method.
set events [epg fetch search -extra "Silent Witness"]
 
if {[llength $events] < 1} {
        puts "No events found."
        exit
}
 
puts "Found [llength $events] event(s)."
 
# Get first event from the returned list
set event [lindex $events 0]
 
puts "Using event - [$event get name]"
 
# Create reservation object of type 2 (series recording)
set r [rsv construct $event 2]
 
# Insert reservation (into rsvp.db pending table)
$r insert
 
# Flag that a system restart is required (puts banner at top of web pages)
system restartpending

HTH (I haven't tried running it as I'm not at home but it should be fairly close.)

 

[darkflare]

I had a look at some of the reservation code and realised that it would be quite an undertaking to port it out of Jim. I had an idea which is this:

I'll download the epg.db from the device.
filter and find the shows I want(using a process of positive and negative filters ie remove jerermy kyle addin movies)
Add the shows to the schedule using the schedule url provided by the web epg (GET /cgi-bin/epg/schedule.jim?service=22272&event=14227&type=2). (this method is quite fragile as if the webif api changes it will break!)

 

[af123]

It was quite an undertaking to /write/ it in Jim : )
Even if the Web API changes, it should just be a change to the URL (and I don't have any plans to change it).
Btw, does the automatic scheduling in the RS portal not do most of what you want?

 

[gebbly]

Previously the RS portal was almost perfect for my use. The only hiccup was that I was teaching myself some network setup and configuration and had created a home network which was so secure that the only normal (I'm sure a persistant hacker could find a way) way of accessing my pvr remotely was via a vpn. So I didnt really want to register any of my machines on any internet service (although I'm sure the portal is a secure and legitimate service).

However the bigger and main reason I am researching this is I recently spent some time picking up some linux admin skills to branch out a bit and am now looking to start doing some proper linux development. I needed a pet project to teach myself and have come up with the following as both a solution to my hiccup and a fun learning project.

I plan a service similar to the excellent RS Portal but using local (to my network) servers rather than remote ones. I hope to have a linux box sitting on my internal network running a little request management daemon (which means learning how to program linux daemons) which sits listening for comms from remote clients (so I shall practice with message queues, socket programming etc). The remote client will send a request to record a program to the request management daemon. If the pvr is awake the request management daemon will make an update to the rsvp.db (probably making use of an ssh session and the JIM libraries perhaps as they seem pretty developer friendly). If the PVR is off the request management daemon will store the request in a local sql database (teaching myself sql usage), when the PVR next boots I will have a little script that runs at boot to access my little database and make the additions to either rsvp.db if I can get it to run before rsvsync (teaching me how to use scripts triggered at boot up and the boot sequence) or perhaps after rsvsync but before the humax software.

Its a bit of a long project but I should be able to pick up some new skills along the way. So, erm. There may be a few more questions from me in the near future as I stumble over things

Thanks very much for all the help so far. Oh and if anyone has any ideas or suggestions of better ways of doing things or ways to enhance the learning aspects of my project please let me know. All suggestions greatfuly accepted.

 

[af123]

Sounds like fun, pitching Jim into that mix shouldn't be too troublesome then : )

For other readers, the RS portal service is a completely outbound service so it is your PVR that makes outward connections to the portal web site on port 80. It's a basic form of RPC (Remote Procedure Call) so, for example, the PVR connects and says "Do you have any new commands for me to process?" and the remote server responds. Adding RS does not allow anything extra to connect in to your box.

 

[gebbly]

af123, do you have any links to guides on Jim or any info at all. I havent come across it in the past and unsurprisingly putting jim into google returns all sorts.

 

[af123]

Try 'jim tcl' - http://jim.tcl.tk/

Jim is a lightweight TCL implementation with some extensions such as object orientation.

 

[Black Hole]

if anyone has any ideas or suggestions of better ways of doing things or ways to enhance the learning aspects of my project please let me know. All suggestions greatfuly accepted.

How about ripping the whole RS implementation onto your own local server and pointing the Hummy at it?

 

[gebbly]

That would certainly give me something far more professional than my early attempts will be but I was reluctant to do that for 2 reasons. Firstly I wouldnt really want to plagiarise someone elses hard work and secondly I think I would learn less that way. I tend to find I learn more by trying to do something myself and learning from mistakes. I have somewhat derailed darkflare's initial post.

I havent seen a section of the forums for little projects, is there such a place? If so I might start a thread there for people to pick my ideas apart and make any contributions.

 

[af123]

How about ripping the whole RS implementation onto your own local server and pointing the Hummy at it?

RS, at least the server side part of it, isn't open source. The client side is mostly written in Jim so available for inspection.