1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is there a simple way of reading the full EPG as data to my custom Win App?

Discussion in 'HD/HDR-FOX T2 Customised Firmware' started by Alan1961, Oct 9, 2012.

  1. Alan1961

    Alan1961 New Member

    As my first post, I would like to say bloody well done to all involved in this project and a big thank you!

    I would like to write (re-write actually) a Windows .net app that displays a filtered EPG using my own custom methods and would like my app to read the full EPG from my HDR FOX T2 in the simplest way without having to resort to any (of my) custom code/script on the box - I'm not qualified for that!

    I could make a few http requests with varying stt to Xepg.jim (or hit it with channel ids) and then scrape the pages, but in an ideal world I would like to http request and get back say a complete XML or database file.

    I thought I'd ask for advice before starting out the hard way.

    Many thanks for any suggestions and please keep in mind I'm not qualified to code for the Hummy.

    Alan.
     
  2. af123

    af123 Administrator Staff Member

    I don't think there's anything there at the moment but it would be straightforward to write a new .jim file that returned an XML/JSON version of the database - all of the hooks are there. I can write one for you if you like - just need some info on what XML structure you'd like.
     
  3. af123

    af123 Administrator Staff Member

    Something like this should give you what you need - haven't tested it yet since my box has unhelpfully powered itself down during the day, but you can see it's quite simple. If it was called db.jim and placed under /mod/webif/cgi-bin/epg/db.jim on the disk then it could easily be accessed directly over HTTP. Might be necessary to do something with 8-bit characters as they occasionally appear in programme names.

    Code:
    #!/mod/bin/jimsh
     
    source /mod/webif/lib/setup
    require epg.class
     
    puts "Content-Type: text/xml"
    puts ""
     
    puts "<events>"
    foreach rec [epg dbfetch dump] {
            $rec get_channel_info
            puts "  <event>"
            foreach var [$rec vars] {
                    puts "    <$var>[$rec get $var]</$var>"
            }
            puts "  </event>"
    }
    puts "</events>"
     
    epg cleanup
    
     
  4. Alan1961

    Alan1961 New Member

    I appreciate you guys have already done great things, get hit by many requests, probably enough to do and possibly have day jobs!

    That is very kind, - I'll try to do the leg work on how to place it on the box and try it. Neither the in the Box FTP or Samba appear to let me
    browse around at will, so I'll need a little time to figure it out. I'm not a bad programmer - but only Windows .net for a long while.

    Thanks again
    Alan.

    PS - Just for background - I find positive keyword alerting on EPGs works well, but not always quite well enough.
    EPGs are so large these days it takes alot to manually plough through them.
    Hopefullly my app will allow the easy build up of a set of rules to knock things off the EPG I know I'm never going to be interested in.
    eg - Kill all "Eastenders"
    Kill all "* Jeremy Kyle *"

    Over time as the rules build up the EPG gets sparser and easier to read - its just another (slightly inverted) way of looking at things which works for me.
    There is also something quite satisfying about killing some programs from the listing!
     
  5. af123

    af123 Administrator Staff Member

    For FTP, make sure you've disabled the built-in Humax FTP server and installed the betaftp package, then ensure you're logging in as the root user with the box PIN as password. That will place you under /media but allow you to navigate around the filesystem. Any other username will work but will lock you into /media. Hope that helps.
    I can add it to the main web interface package so that it's installed and maintained as part of that but we might as well get it approaching useful first.

    I can certainly appreciate that : - )
     
  6. Alan1961

    Alan1961 New Member

    Sorry, after some investigation it appears I'm falling at the first hurdle... and trying not to bug you!
    It would appear that when I transfer files up they do not (understandably) have execute permission, it also seems to me
    that the FTP client server does not support the chmod command - any advice would be much appreciated to at least get me off the ground!

    Alan
     
  7. Ezra Pound

    Ezra Pound Well-Known Member

    You will need Telnet access to cary out the things you want to do e.g. chmod etc. PuTTy is a good choice Link HERE
     
  8. xyz321

    xyz321 Well-Known Member

    It displays the header then the jim process progressively eats up more and more memory until the box crashes.
     
  9. Black Hole

    Black Hole Felonius Gru

    Correct - FTP (File Transfer Protocol) is for transferring files. As Ezra said, use Telnet (which provides a remote command console).
     
  10. af123

    af123 Administrator Staff Member

    Most modern FTP servers support CHMOD, as do most FTP clients these days. Betaftp doesn't seem to though (I could add it at some point)

    @xyz321, thanks, I'll try again : )
     
  11. af123

    af123 Administrator Staff Member

    This one works (a bit slow but see how it does for you):

    Code:
    #!/mod/bin/jimsh
     
    source /mod/webif/lib/setup
    require epg.class
     
    puts "Content-Type: text/xml"
    puts ""
     
    set query "
                select *,
                usLcn as channel_num,
                substr(szSvcName, 2) as channel_name,
                aucDefaultAuthority as channel_crid,
                hSvc as channel_hsvc
                from epg.epg e join TBL_SVC c
                on e.service_id = c.usSvcId
                order by channel_num, start
    "
     
    set chunk 1000
    set offset 0
     
    puts "<events>"
     
    set recs 1
    while {[llength $recs] > 0} {
            set recs [$::channeldb query "$query limit $chunk offset $offset"]
            incr offset $chunk
            foreach r $recs {
                    set rec [epg new $r]
                    puts "  <event>"
                    foreach var [$rec vars] {
                            puts "    <$var>[$rec get $var]</$var>"
                    }
                    puts "  </event>"
                    unset rec
            }
    }
    puts "</events>"
     
    epg cleanup
    
    You could play with the chunk value to see how high you can go - there are around 17K entries in a typical 7 day guide.
     
  12. af123

    af123 Administrator Staff Member

    Scratch that - it's never going to be fast enough (around 17 minutes!)

    If tab separated output is good enough for you then this should be good enough and extremely fast. You still need to call the server script something.jim so that it is executed by the web server.

    Code:
    #!/bin/sh
     
    echo "Content-Type: text/plain"
    echo ""
     
    /mod/bin/epg -p dump
    
    Faster than I expected:

    Code:
    air(45) curl -o db.tsv http://humax/cgi-bin/epg/db.jim
      % Total    % Received % Xferd  Average Speed  Time    Time    Time  Current
                                    Dload  Upload  Total  Spent    Left  Speed
    100 3623k    0 3623k    0    0  1157k      0 --:--:--  0:00:03 --:--:-- 1161k
    air(46) ls -l db.tsv
    -rw-r--r--  1 af  staff  3.5M  9 Oct 20:07 db.tsv
    air(47) head -2 db.tsv
    14720    62277    1349740800    10800    0    All 80s All Night    Ben Jones plays the best of the 80s.        3    Show/Game show               
    14720    62278    1349751600    7200    0    Nicola Bonn    Nicola Bonn getting your weekday mornings off to a flying start        3    Show/Game show             
    
     
  13. Alan1961

    Alan1961 New Member

    Many, many thanks guys... and very fast af123!
    I should be able to load this OK ... and a quick test into a spreadsheet looks good.

    Alan.
     
  14. darkflare

    darkflare Member

    Just to wedge my oar in -

    I investigated this a while ago, thinking of making an android client at some point, anyway here's some of the data I collected:

    On the box there are 3 sql lite databases generated by af123's fantastic epg.exe


    epg database /mnt/hd1/epg.db
    channel database /var/lib/humaxtv/channel.db
    reservations db? /var/lib/humaxtv/rsv.db

    you cant write to the reservations database however af123 has said he has no plans to change the way schedule.jim works and so you can use the following to write a reservation:

    GET /cgi-bin/epg/schedule.jim?service=22272&event=14227&type=2
    type 1 = show
    type 2 = series


    I hope this helps, and I sincerely look forward to seeing your windows client as I am VERY interested in an elimation style epg browsing, see my plans/ramblings here http://hummy.tv/forum/threads/rs-portal-search-idea.1612/

    Good luck Alan!!
     
  15. Alan1961

    Alan1961 New Member

    Thanks darkflare,

    I read your "ramblings" and was much amused that we both wish to banish Jermey Kyle from the EPG!

    Af123s approach to getting the EPG is very effective and very quick ... but to do a proper job I really should be reading the channel names from somewhere instead of hard coding the channel id's in the app - so I'll look at some way of getting those at some point. Not being qualified for the box end of things I am wary about reading files directly due to potential locking concerns or interfering with apps on the box.

    I've got a basic but messy test WPF .Net client up and doing very useful work already, however I'm only using a time ordered Datagrid listing at the moment and not a proper EPG style display - I'm not actually sure I need one. But my current level of banning (I don't watch much TV) means that a single days listing of all channels just about fits on one screen and easy to scan. I've no intention to do any functions other than display, but do have a button to pull up the lovely Web interface for things like reservations.

    Anyone will be welcome to have my embarassing code ... and if interested post a note back here in a week or two to see where I am at.

    Alan.
     
  16. darkflare

    darkflare Member

    af123, I am trying to emulate the db.jim dump script however I have a really silly issue when trying to curl the url:
    Error 500: Internal Server Error
    CGI program sent malformed HTTP headers: [Mongoose web server v.3.0 started on port(s) 80,443s with web root [/mod/var/mongoose/html,/media=/media,/cgi-bin=/mod/var/mongoose/cgi-bin,/images=/opt/share/images/blue,/plugin=/mod/var/mongoose/plugin]

    Can you share some light on what this means please?
     
  17. xyz321

    xyz321 Well-Known Member

    Your script may have missing or invalid HTTP content type headers. e.g.

    Content-Type: text/xml
     
  18. darkflare

    darkflare Member

    I see why you say that however I can't seem to add any extra files to the cgi-bin folder to be accessed. For example if I duplicate browse.jim and call it custom2.jim I get the same error. Its baffling and very annoying!
     
  19. darkflare

    darkflare Member

    Good news, fixed the issue, in this case I needed to chmod a+x the custom2.jim!
     
  20. af123

    af123 Administrator Staff Member

    I need to push out an update to the webif package tonight so how about I include a db.jim script that can be called as follows?

    http://xxx/cgi-bin/db.jim
    http://xxx/cgi-bin/db.jim?data=epg
    (get EPG data in TSV)

    http://xxx/cgi-bin/db.jim?data=channels
    (get Channel data in TSV)

    + any others you need?

    This is what I was thinking of, nothing too fancy:

    Code:
    #!/bin/sh
    
    PATH=$PATH:/mod/bin export PATH
    
    echo "Content-Type: text/plain"
    echo ""
    
    [ -n "$QUERY_STRING" ] && data="${QUERY_STRING##*=}" || data=epg
    
    case $data in
            channels)
                    sqlite3 -header -batch -separator '     ' \
                        /var/lib/humaxtv/channel.db '
                            select * from TBL_SVC;
                    '
                    ;;
    
            *) /mod/bin/epg -p dump ;;
    esac
    
    exit 0