Split transmissions not correctly handled by Humax scheduler

[/df]

As expected the modified scheduled recording for "Layer Cake" was magically updated to a single 55 minute recording.

How about setting a massive ulPostOffset to cover the intermission and second part, obviously at the cost of losing Accurate Recording (if that's actually Accurate on "Sony Movies", not supported by the missing chunks at the end of last week's Pulp Fiction Pt1)? Today we are being offered Brazen, a 55 minute + 50 minute split show with a 5 minute intermission.

What happens if we set 1 minute padding before and 2 minutes plus the intermission and part 2 padding after? Apparently aulEventToRecordInfo doesn't know about padding so doesn't have to be adjusted.

# sqlite3  /var/lib/humaxtv/rsv.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> .mode line
sqlite> select * from TBL_RESERVATION where ulslot = 1;
              ulslot = 1
            ersvtype = 3
                hsvc = 65555
             nsttime = 1586289600
            szsttime = 00000000000000
           nduration = 3300
             erepeat = 0
             usevtid = 6718
           szevtname = Braven
         ulPreOffset = 0
        ulPostOffset = 0
         ulProgramId = 0
          ulSeriesId = 0
            ucVolume = 0
         ucInputMode = 0
             usChNum = 0
           ucRecKind = 2
          ucCRIDType = 49
              szCRID = WWW.MOVIEMIX.CO.UK/X4680336
        szFPBRecPath = 
  szRecordedProgCrid = 
     szEventToRecord = 1WWW.MOVIEMIX.CO.UK/X4680336#01|1WWW.MOVIEMIX.CO.UK/X4680336#01|
aulEventToRecordInfo = 
           bRecomRsv = 0
 usLastRecordedEvtId = 0
              eReady = 30
sqlite> update TBL_RESERVATION set ulPreOffset=60, ulPostOffset=3420, ucRecKind=1, szEventToRecord='1WWW.MOVIEMIX.CO.UK/X4680336|' where ulslot = 1;
sqlite>

 

[MymsMan]

I think it is a mistake to be attempting to combine the parts into one recording, it is much simpler and less error prone to join two programmes together than to locate and crop out the intervening programme.
It would be better to focus on how to prevent the recording of unwanted repeats instead.

If you just remove the repeats from aulEventToRecordInfo I think the humax will replace them at the next opportunity but I observed from the problems caused by the Queens broadcast to recordings on ITV that the humax does not update scheduled recordings where the event_id no longer exists in the epg.
So it would be worth experimenting to see if changing the eventids in the aul array prevented recording the repeat (though you would get a failed to track hmt), If that worked you could then try changing the duration in the aul to reduce the size of the potential recording conflict

 

[/df]

I think it is a mistake to be attempting to combine the parts into one recording, it is much simpler and less error prone to join two programmes together than to locate and crop out the intervening programme.

One reason for combining is because the (ab)users of split transmissions are typically poor at identifying the end of the first part of the content, so losing the cliffhanger action at that point when the split is respected. But I'm still interested to see what happens tonight. However an advantage of post-processing is that there's already a hook available for actions on completion of a recording.

It would be better to focus on how to prevent the recording of unwanted repeats instead.

If you just remove the repeats from aulEventToRecordInfo I think the humax will replace them at the next opportunity but I observed from the problems caused by the Queens broadcast to recordings on ITV that the humax does not update scheduled recordings where the event_id no longer exists in the epg.

ITV changed the published CRIDs, no? So they completely broke the spec. Or was it the event_ids? The same thing: Part A 8.4.2.1 "From the time of first presentation in the event schedule the event_id shall be associated with a single event, even if that event is rescheduled within the currently transmitted schedule", though I suppose this allows them to use new event_ids:

... if a programme is significantly rescheduled the event_id is likely to change whereas the CRID shall remain consistent. Therefore, the resolution of a programme CRID to an EIT event should be deferred as long as possible, up to and including resolution through EIT present.

If an event with a changed event_id is still part of a series that is in the receiver‘s list of series to record then a receiver should still be expected to record the changed event. This could occur if the broadcast episode is changed. This will appear to a receiver as if a new event has been added to the series. Care should be taken in this scenario as the new event may appear to clash will the original event in the receiver[']s record list (which will have been dropped from EIT).

Was the problem (eg Belgravia series recording) that Humax didn't adequately defer resolving the CRID?

So it would be worth experimenting to see if changing the eventids in the aul array prevented recording the repeat (though you would get a failed to track hmt), If that worked you could then try changing the duration in the aul to reduce the size of the potential recording conflict

I've only seen aul in TBL_RESERVATION with a format like struct { uint32 hsvc, uint32 start, uint32 end, uint32 event_id }, where the event_id is for the first part of the split. Does it sometimes become an array of these, as suggested in the D-Book quote above "a film divided into two parts by a news programme would be three events", or does the aul just get updated with the event data of the next part once the first part has started (or finished?) recording? nugget schedule.epg shows an event for the first part and a second with unpopulated fields corresponding to those in the aul.

 

[MymsMan]

ITV changed the published CRIDs, no? So they completely broke the spec. Or was it the event_ids? The same thing: Part A 8.4.2.1 "From the time of first presentation in the event schedule the event_id shall be associated with a single event, even if that event is rescheduled within the currently transmitted schedule", though I suppose this allows them to use new event_ids:

They changed the eventids and not the crids so the programmes appeared to be not there when looking via eventid but 'already scheduled' when looking via crid which was my schedule checker failed to resolve the issue and I suspect was also why the humax didn't sort it out itself

I've only seen aul in TBL_RESERVATION with a format like struct { uint32 hsvc, uint32 start, uint32 end, uint32 event_id }, where the event_id is for the first part of the split. Does it sometimes become an array of these, as suggested in the D-Book

Yes, If you look at the entry for a series you will see an aul array entry for each of the scheduled episodes currently in the epg with the corresponding crids in the EventToRecord list
Similarly for a split recording with repeat there will be four sets of aul data and four identical crids

If working in JIM code the rsv class has an aul method to retrieve and convert out of binary, and a buildaul method to convert a list back to binary which could make it easier to see what you are dealing with

 

[/df]

They changed the eventids and not the crids so the programmes appeared to be not there when looking via eventid but 'already scheduled' when looking via crid which was my schedule checker failed to resolve the issue and I suspect was also why the humax didn't sort it out itself.

Ah, so the Humax s/w didn't "adequately defer resolving the CRID". Apparently they should have had a check on a new event to see if its p-CRID matched any existing event and if so transfer any schedule for the existing event to the new one. However in this case the schedule change was minimal and ought to have been accommodated using the original events.

... for a split recording with repeat there will be four sets of aul data and four identical crids ...

Maybe that gets set in memory or in the DB later? I see the multiple events, at least parts 1 and 2, in szEventToRecord but not in the DB, which had aulEventToRecordInfo = X'13000100 c0db8c5e a4e88c5e 3e1a0000' (my spacing) for Brazen, corresponding to part 1.

Yes, it needs the EPG to update. I borked the Brazen test by not rebooting, but after scheduling the Layer Cake repeat overnight I see aulEventToRecordInfo = X'13000100 104e8e5e f45a8e5e 611a0000 13000100 205c8e5e 886c8e5e 631a0000' this morning. It looks like the lifecycle is for szEventToRecord to be set on scheduling and later aulEventToRecordInfo gets updated.

Obviously the only way I'd know any of this structure is by reference to the rsv class!

 

[/df]

...How about setting a massive ulPostOffset to cover the intermission and second part, obviously at the cost of losing Accurate Recording ...
What happens if we set 1 minute padding before and 2 minutes plus the intermission and part 2 padding after? Apparently aulEventToRecordInfo doesn't know about padding so doesn't have to be adjusted.

But as MM pointed out it does.
Once I set up a test correctly that didn't crash in the middle of the recording, I was able to show that setting

• ulPreOffset for pre-padding,
• ulPostOffset as the total duration of the intermission and second part plus post-padding,
• ucRecKind=1,
• szEventToRecord to the first part with no IMI, and
• aulEventToRecordInfo to the corresponding binary blob.

made a reservation that recorded the entire programme including intermission. And presumably whose repeat parts would not have caused a conflict.

I hoped that a little Jim script would enable me to update the test reservation but despite running with no errors this had no effect:

#!/mod/bin/jimsh

source /mod/webif/lib/setup
require rsv.class system.class

# let's mess with the internals
rsv method set {ivName val} {
        set $ivName $val
}

set slot 1
set duration 7500
set duration1 3300

set table "TBL_RESERVATION"

set event [rsv slot $table $slot]

$event set ucRecKind 1
$event set_pad  60 [expr $duration - $duration1 + 120] 
$event set szEventToRecord [string cat [lindex [split [$event get szEventToRecord] "|"] 0] "|"]
$event setaul [binary format iiii [$event get hsvc] [$event start] [$event end] [$event get usevtid]]
$event insert
# shouldn't this be unnecessary/unwanted with RTS after [rsv commit] in the previous call?
system restartpending

 

[af123]

I hoped that a little Jim script would enable me to update the test reservation but despite running with no errors this had no effect:

Take a look in rsvsync.log. Hopefully that will provide some clues.
Alternatively, call insert with the pending flag and take a look in the rsvp.db file to see what has been put there.

 

[Black Hole]

Isn't this a case where you have to use nugget or something to circumvent the database lock while booted?

 

[MymsMan]

I hoped that a little Jim script would enable me to update the test reservation but despite running with no errors this had no effect:

I tried your exec and it does change the padding figures including the nugget timers but the aul still shows the original episodes.
I think you did successfully update the aul but the humax has refreshed the field with the missing recording as I suggested in post #22

humax /mnt/hd2/mod/git/schedchk/webif/plugin/schedchk # ./fmtrsv.jim  -s 11 -fmt
ulslot               = 11
ersvtype             = 3
hsvc                 = 65554
nsttime              = 1586685900     12/04/2020 11:05
szsttime             = 00000000000000
nduration            = 3300           00:55
erepeat              = 0
usevtid              = 6860
szevtname            = i7Labyrinth   (Labyrinth)
ulPreOffset          = 0
ulPostOffset         = 0
ulProgramId          = 0
ulSeriesId           = 0
ucVolume             = 0
ucInputMode          = 0
usChNum              = 0
ucRecKind            = 2
ucCRIDType           = 49
szCRID               = WWW.MOVIEMIX.CO.UK/X4811444
szFPBRecPath         =
szRecordedProgCrid   =
szEventToRecord      = 1WWW.MOVIEMIX.CO.UK/X4811444#01|1WWW.MOVIEMIX.CO.UK/X4811444#01|1WWW.MOVIEMIX.CO.UK/X4811444#01|1WWW.MOVIEMIX.CO.UK/X4811444#01|
aulEventToRecordInfo = {65554 1586685900 1586689200 6860} {65554 1586689500 1586693700 6862} {65554 1586871000 1586874300 6912} {65554 1586874600 1586878500 6914}
bRecomRsv            = 0
usLastRecordedEvtId  = 0
eReady               = 30
szSvcName            = Sony Movies
usLcn                = 32
  Svcid  Start                         End                             Evtid  CRID
  65554  1586685900 12/04/2020 11:05   1586689200 12/04/2020 12:00      6860  1WWW.MOVIEMIX.CO.UK/X4811444#01
  65554  1586689500 12/04/2020 12:05   1586693700 12/04/2020 13:15      6862  1WWW.MOVIEMIX.CO.UK/X4811444#01
  65554  1586871000 14/04/2020 14:30   1586874300 14/04/2020 15:25      6912  1WWW.MOVIEMIX.CO.UK/X4811444#01
  65554  1586874600 14/04/2020 15:30   1586878500 14/04/2020 16:35      6914  1WWW.MOVIEMIX.CO.UK/X4811444#01
========================================================
humax /mnt/hd2/mod/git/schedchk/webif/plugin/schedchk # ./dftest
{65554 1586685900 1586689200 6860} {65554 1586689500 1586693700 6862} {65554 1586871000 1586874300 6912} {65554 1586874600 1586878500 6914}

{65554 1586685900 1586689200 6860}
humax /mnt/hd2/mod/git/schedchk/webif/plugin/schedchk # ./fmtrsv.jim  -s 11 -fmt
ulslot               = 11
ersvtype             = 3
hsvc                 = 65554
nsttime              = 1586685900     12/04/2020 11:05
szsttime             = 00000000000000
nduration            = 3300           00:55
erepeat              = 0
usevtid              = 6860
szevtname            = i7Labyrinth   (Labyrinth)
ulPreOffset          = 60
ulPostOffset         = 4320
ulProgramId          = 0
ulSeriesId           = 0
ucVolume             = 0
ucInputMode          = 0
usChNum              = 0
ucRecKind            = 2
ucCRIDType           = 49
szCRID               = WWW.MOVIEMIX.CO.UK/X4811444
szFPBRecPath         =
szRecordedProgCrid   =
szEventToRecord      = 1WWW.MOVIEMIX.CO.UK/X4811444#01|1WWW.MOVIEMIX.CO.UK/X4811444#01|1WWW.MOVIEMIX.CO.UK/X4811444#01|1WWW.MOVIEMIX.CO.UK/X4811444#01|
aulEventToRecordInfo = {65554 1586685900 1586689200 6860} {65554 1586689500 1586693700 6862} {65554 1586871000 1586874300 6912} {65554 1586874600 1586878500 6914}
bRecomRsv            = 0
usLastRecordedEvtId  = 0
eReady               = 30
szSvcName            = Sony Movies
usLcn                = 32
  Svcid  Start                         End                             Evtid  CRID
  65554  1586685900 12/04/2020 11:05   1586689200 12/04/2020 12:00      6860  1WWW.MOVIEMIX.CO.UK/X4811444#01
  65554  1586689500 12/04/2020 12:05   1586693700 12/04/2020 13:15      6862  1WWW.MOVIEMIX.CO.UK/X4811444#01
  65554  1586871000 14/04/2020 14:30   1586874300 14/04/2020 15:25      6912  1WWW.MOVIEMIX.CO.UK/X4811444#01
  65554  1586874600 14/04/2020 15:30   1586878500 14/04/2020 16:35      6914  1WWW.MOVIEMIX.CO.UK/X4811444#01
========================================================
humax /mnt/hd2/mod/git/schedchk/webif/plugin/schedchk # nugget schedule.timers 11
Slot           Stat           Type Time       Left
----           ----           ---- ----       ----
  11: 0x1512d88  0/ 0 (idle  )  3   1586685810 215612 Sun Apr 12 11:03:30 2020

1011    Thu Apr  9 23:08:02 2020: Loading schedule information to HumaxTV binary.
1010    Thu Apr  9 23:08:02 2020: Final schedule entries: 60
1009    Thu Apr  9 23:08:02 2020: Slots: 11
1008    Thu Apr  9 23:08:02 2020: Padding slot 11 = 60/4320
1007    Thu Apr  9 23:08:02 2020: Found slot 11 for 3/65554/1586685900/6860
1006    Thu Apr  9 23:08:02 2020: Opening /var/lib/humaxtv/rsvp.db
1005    Thu Apr  9 23:08:02 2020: Schedule saved.
1004    Thu Apr  9 23:08:02 2020: Nugget is available.
1003    Thu Apr  9 23:08:02 2020: Real-time mode.
1002    Thu Apr  9 23:08:02 2020: rsvsync starting, version 1.1.12.

 

[MymsMan]

Eureka,

Isn't this a case where you have to use nugget or something to circumvent the database lock while booted?

The use of nugget is built into the rsv class methods using the action field.
The set_pad call sets action to 3, and causes the pad fields in the database to be updated but ignores other changes.
To update the aul and events to record fields you need to set action to 5 (skip)
To update both pad and aul you would need to call insert twice.
I have now successfully updated the entry for my test recording with dummy entries for the repeat, time will tell if it works (or I will look for a more imminent test case)

szEventToRecord      = 1WWW.MOVIEMIX.CO.UK/X4811444#01|1WWW.MOVIEMIX.CO.UK/X4811444#01|1WWW.MOVIEMIX.CO.UK/X4811444#02|1WWW.MOVIEMIX.CO.UK/X4811444#02|
aulEventToRecordInfo = {65554 1586685900 1586689200 6860} {65554 1586689500 1586693700 6862} {65554 1586871000 1586871000 1234} {65554 1586874600 1586874600 1234}
bRecomRsv            = 0
usLastRecordedEvtId  = 0
eReady               = 30
szSvcName            = Sony Movies
usLcn                = 32
  Svcid  Start                         End                             Evtid  CRID
  65554  1586685900 12/04/2020 11:05   1586689200 12/04/2020 12:00      6860  1WWW.MOVIEMIX.CO.UK/X4811444#01
  65554  1586689500 12/04/2020 12:05   1586693700 12/04/2020 13:15      6862  1WWW.MOVIEMIX.CO.UK/X4811444#01
  65554  1586871000 14/04/2020 14:30   1586871000 14/04/2020 14:30      1234  1WWW.MOVIEMIX.CO.UK/X4811444#02
  65554  1586874600 14/04/2020 15:30   1586874600 14/04/2020 15:30      1234  1WWW.MOVIEMIX.CO.UK/X4811444#02

 

[/df]

Eureka,
...
To update both pad and aul you would need to call insert twice.
...

Is there any reason why there shouldn't be a "just do it" action that causes insert to update all fields in TBL_RESERVATION (or pending) from the corresponding fields in the rsv instance?

 

[MymsMan]

Is there any reason why there shouldn't be a "just do it" action that causes insert to update all fields in TBL_RESERVATION (or pending) from the corresponding fields in the rsv instance?

Possibly for the same reason that the 'set' method wasn't implemented in the rsv class - to keep bumbling amateurs like us from doing too much damage

 

[af123]

The actions are:

0 - add​

1 - delete​

2 - unpad​

3 - pad​

4 - folder​

5 - skip​

6 - refresh​

 

[MymsMan]

I have now successfully updated the entry for my test recording with dummy entries for the repeat, time will tell if it works

Alas, after recording today's showing of Labyrinth the schedule has now updated to show a recording due for Tuesday's showing

 

[/df]

Some observations on current FreeView broadcaster practice:

• ITV, PBS America, Sony and a few other bottom feeders are using p-CRIDs with different IMIs (say #01, #02) to identify the same show on the main channel and its +1, which is allowed according to D-Book Pt A 8.7.2.2;
• C4, Discovery (eg Quest) and Sony when it's CBS use the same p-CRID for the show and its +1 airing, but of course while the content is the same the channel ident differs;
• Sky (eg Pick) uses totally different p-CRIDs for a show and its +1 airing.

Presumably this list corresponds to three different content management/play-out software configurations, perhaps from different suppliers?

 

[MymsMan]

This thread in the other place discusses the problem that, whereas split transmissions are identified by a Programme CRID followed by an Instance Metadata Identifier (IMI), such as www.itv.com/1001281561#1, and owing to a brain-dead specification the IMI is always the same for all parts and repeats of the split show, but repeats must be detected by starting more than 3 hours from the end of the previous set of split transmissions, the Humax software (HD/R-Fox T2 and some others) treats all occurrences of the P-CRID+IMI as parts of the same show regardless of the 3 hour rule. This results in false conflicts and possibly unwanted recordings.

Assuming that the on-box UI is a lost cause in this regard, could such situations be handled in WebIF or related functions so that (a) split repeats don't show as conflicts (b) if the scheduled split show has been recorded any repeats are marked as already recorded?

Or does this happen already?

Separately, are there ever 3-part splits, as rsv.class appears to assume not?

I had the reverse problem yesterday - I tried to record Dads Army (film) on ITV and it recorded the first half but not the second half and there don't appear to be any repeats currently in the schedule, once again the Queen has caused ITV to mess up the recording schedule!

 

[emdubya77]

I had the reverse problem yesterday - I tried to record Dads Army (film) on ITV and it recorded the first half but not the second half and there don't appear to be any repeats currently in the schedule, once again the Queen has caused ITV to mess up the recording schedule!

Same for me - glad to hear it was not my error.

 

[MymsMan]

I have just published a new package schedchk to the Advanced section of the package catalogue

One of the checks the new package makes is for split recordings with repeats that the humax, incorrectly, treats as additional parts of the same programme
When schedchk finds a split recording with parts more than three hours apart it chooses the first occurrence (that doesn't conflict with other recordings), removes the unwanted repeats and changes the IMI (the #nn part of the crid) for the wanted recording to prevent the humax rescheduling the repeats.

See https://wiki.hummy.tv/wiki/Schedchk for discussion of the new package.