• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.
  • The forum software that supports hummy.tv has been upgraded to XenForo 2.0!

    This is a major upgrade which changes the look and feel of the forum somewhat but brings a host of improvements too. Please bear with us as we continue to tweak things and report any issues or suggestions in Site/Forum Issues.

Is there a simple way of reading the full EPG as data to my custom Win App?

#1
As my first post, I would like to say bloody well done to all involved in this project and a big thank you!

I would like to write (re-write actually) a Windows .net app that displays a filtered EPG using my own custom methods and would like my app to read the full EPG from my HDR FOX T2 in the simplest way without having to resort to any (of my) custom code/script on the box - I'm not qualified for that!

I could make a few http requests with varying stt to Xepg.jim (or hit it with channel ids) and then scrape the pages, but in an ideal world I would like to http request and get back say a complete XML or database file.

I thought I'd ask for advice before starting out the hard way.

Many thanks for any suggestions and please keep in mind I'm not qualified to code for the Hummy.

Alan.
 

af123

Administrator
Staff member
#2
I don't think there's anything there at the moment but it would be straightforward to write a new .jim file that returned an XML/JSON version of the database - all of the hooks are there. I can write one for you if you like - just need some info on what XML structure you'd like.
 

af123

Administrator
Staff member
#3
Something like this should give you what you need - haven't tested it yet since my box has unhelpfully powered itself down during the day, but you can see it's quite simple. If it was called db.jim and placed under /mod/webif/cgi-bin/epg/db.jim on the disk then it could easily be accessed directly over HTTP. Might be necessary to do something with 8-bit characters as they occasionally appear in programme names.

Code:
#!/mod/bin/jimsh
 
source /mod/webif/lib/setup
require epg.class
 
puts "Content-Type: text/xml"
puts ""
 
puts "<events>"
foreach rec [epg dbfetch dump] {
        $rec get_channel_info
        puts "  <event>"
        foreach var [$rec vars] {
                puts "    <$var>[$rec get $var]</$var>"
        }
        puts "  </event>"
}
puts "</events>"
 
epg cleanup
 
OP
OP
A

Alan1961

New Member
#4
I appreciate you guys have already done great things, get hit by many requests, probably enough to do and possibly have day jobs!

That is very kind, - I'll try to do the leg work on how to place it on the box and try it. Neither the in the Box FTP or Samba appear to let me
browse around at will, so I'll need a little time to figure it out. I'm not a bad programmer - but only Windows .net for a long while.

Thanks again
Alan.

PS - Just for background - I find positive keyword alerting on EPGs works well, but not always quite well enough.
EPGs are so large these days it takes alot to manually plough through them.
Hopefullly my app will allow the easy build up of a set of rules to knock things off the EPG I know I'm never going to be interested in.
eg - Kill all "Eastenders"
Kill all "* Jeremy Kyle *"

Over time as the rules build up the EPG gets sparser and easier to read - its just another (slightly inverted) way of looking at things which works for me.
There is also something quite satisfying about killing some programs from the listing!
 

af123

Administrator
Staff member
#5
That is very kind, - I'll try to do the leg work on how to place it on the box and try it. Neither the in the Box FTP or Samba appear to let me
browse around at will
For FTP, make sure you've disabled the built-in Humax FTP server and installed the betaftp package, then ensure you're logging in as the root user with the box PIN as password. That will place you under /media but allow you to navigate around the filesystem. Any other username will work but will lock you into /media. Hope that helps.
I can add it to the main web interface package so that it's installed and maintained as part of that but we might as well get it approaching useful first.

There is also something quite satisfying about killing some programs from the listing!
I can certainly appreciate that : - )
 
OP
OP
A

Alan1961

New Member
#6
Sorry, after some investigation it appears I'm falling at the first hurdle... and trying not to bug you!
It would appear that when I transfer files up they do not (understandably) have execute permission, it also seems to me
that the FTP client server does not support the chmod command - any advice would be much appreciated to at least get me off the ground!

Alan
 

xyz321

Well-Known Member
#8
Something like this should give you what you need - haven't tested it yet since my box has unhelpfully powered itself down during the day, but you can see it's quite simple.
It displays the header then the jim process progressively eats up more and more memory until the box crashes.
 

af123

Administrator
Staff member
#10
Most modern FTP servers support CHMOD, as do most FTP clients these days. Betaftp doesn't seem to though (I could add it at some point)

@xyz321, thanks, I'll try again : )
 

af123

Administrator
Staff member
#11
This one works (a bit slow but see how it does for you):

Code:
#!/mod/bin/jimsh
 
source /mod/webif/lib/setup
require epg.class
 
puts "Content-Type: text/xml"
puts ""
 
set query "
            select *,
            usLcn as channel_num,
            substr(szSvcName, 2) as channel_name,
            aucDefaultAuthority as channel_crid,
            hSvc as channel_hsvc
            from epg.epg e join TBL_SVC c
            on e.service_id = c.usSvcId
            order by channel_num, start
"
 
set chunk 1000
set offset 0
 
puts "<events>"
 
set recs 1
while {[llength $recs] > 0} {
        set recs [$::channeldb query "$query limit $chunk offset $offset"]
        incr offset $chunk
        foreach r $recs {
                set rec [epg new $r]
                puts "  <event>"
                foreach var [$rec vars] {
                        puts "    <$var>[$rec get $var]</$var>"
                }
                puts "  </event>"
                unset rec
        }
}
puts "</events>"
 
epg cleanup
You could play with the chunk value to see how high you can go - there are around 17K entries in a typical 7 day guide.
 

af123

Administrator
Staff member
#12
Scratch that - it's never going to be fast enough (around 17 minutes!)

If tab separated output is good enough for you then this should be good enough and extremely fast. You still need to call the server script something.jim so that it is executed by the web server.

Code:
#!/bin/sh
 
echo "Content-Type: text/plain"
echo ""
 
/mod/bin/epg -p dump
Faster than I expected:

Code:
air(45) curl -o db.tsv http://humax/cgi-bin/epg/db.jim
  % Total    % Received % Xferd  Average Speed  Time    Time    Time  Current
                                Dload  Upload  Total  Spent    Left  Speed
100 3623k    0 3623k    0    0  1157k      0 --:--:--  0:00:03 --:--:-- 1161k
air(46) ls -l db.tsv
-rw-r--r--  1 af  staff  3.5M  9 Oct 20:07 db.tsv
air(47) head -2 db.tsv
14720    62277    1349740800    10800    0    All 80s All Night    Ben Jones plays the best of the 80s.        3    Show/Game show               
14720    62278    1349751600    7200    0    Nicola Bonn    Nicola Bonn getting your weekday mornings off to a flying start        3    Show/Game show
 
#14
Just to wedge my oar in -

I investigated this a while ago, thinking of making an android client at some point, anyway here's some of the data I collected:

On the box there are 3 sql lite databases generated by af123's fantastic epg.exe


epg database /mnt/hd1/epg.db
channel database /var/lib/humaxtv/channel.db
reservations db? /var/lib/humaxtv/rsv.db

you cant write to the reservations database however af123 has said he has no plans to change the way schedule.jim works and so you can use the following to write a reservation:

GET /cgi-bin/epg/schedule.jim?service=22272&event=14227&type=2
type 1 = show
type 2 = series


I hope this helps, and I sincerely look forward to seeing your windows client as I am VERY interested in an elimation style epg browsing, see my plans/ramblings here http://hummy.tv/forum/threads/rs-portal-search-idea.1612/

Good luck Alan!!
 
OP
OP
A

Alan1961

New Member
#15
Thanks darkflare,

I read your "ramblings" and was much amused that we both wish to banish Jermey Kyle from the EPG!

Af123s approach to getting the EPG is very effective and very quick ... but to do a proper job I really should be reading the channel names from somewhere instead of hard coding the channel id's in the app - so I'll look at some way of getting those at some point. Not being qualified for the box end of things I am wary about reading files directly due to potential locking concerns or interfering with apps on the box.

I've got a basic but messy test WPF .Net client up and doing very useful work already, however I'm only using a time ordered Datagrid listing at the moment and not a proper EPG style display - I'm not actually sure I need one. But my current level of banning (I don't watch much TV) means that a single days listing of all channels just about fits on one screen and easy to scan. I've no intention to do any functions other than display, but do have a button to pull up the lovely Web interface for things like reservations.

Anyone will be welcome to have my embarassing code ... and if interested post a note back here in a week or two to see where I am at.

Alan.
 
#16
af123, I am trying to emulate the db.jim dump script however I have a really silly issue when trying to curl the url:
Error 500: Internal Server Error
CGI program sent malformed HTTP headers: [Mongoose web server v.3.0 started on port(s) 80,443s with web root [/mod/var/mongoose/html,/media=/media,/cgi-bin=/mod/var/mongoose/cgi-bin,/images=/opt/share/images/blue,/plugin=/mod/var/mongoose/plugin]

Can you share some light on what this means please?
 
#18
I see why you say that however I can't seem to add any extra files to the cgi-bin folder to be accessed. For example if I duplicate browse.jim and call it custom2.jim I get the same error. Its baffling and very annoying!
 

af123

Administrator
Staff member
#20
Af123s approach to getting the EPG is very effective and very quick ... but to do a proper job I really should be reading the channel names from somewhere instead of hard coding the channel id's in the app - so I'll look at some way of getting those at some point.
I need to push out an update to the webif package tonight so how about I include a db.jim script that can be called as follows?

http://xxx/cgi-bin/db.jim
http://xxx/cgi-bin/db.jim?data=epg
(get EPG data in TSV)

http://xxx/cgi-bin/db.jim?data=channels
(get Channel data in TSV)

+ any others you need?

This is what I was thinking of, nothing too fancy:

Code:
#!/bin/sh

PATH=$PATH:/mod/bin export PATH

echo "Content-Type: text/plain"
echo ""

[ -n "$QUERY_STRING" ] && data="${QUERY_STRING##*=}" || data=epg

case $data in
        channels)
                sqlite3 -header -batch -separator '     ' \
                    /var/lib/humaxtv/channel.db '
                        select * from TBL_SVC;
                '
                ;;

        *) /mod/bin/epg -p dump ;;
esac

exit 0