1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Accurate Recording (AR), Padding, and Lost Series Links

Discussion in 'HDR-FOX T2 Freeview Recorder' started by Black Hole, Nov 24, 2011.

  1. Black Hole

    Black Hole May contain traces of nut

    This is a get-up-to-speed for newcomers, being as the question came up again recently. It summarises the state of the knowledge, well known to HD/HDR-FOX T2 veterans. I should say at this point that I don't know whether this applies to the HD-FOX, but I presume that it does.

    Ignoring manual timer programming (that's like going back to programming a VCR!), the HDR-FOX has two recording timer modes, one we call AR (Accurate Recording) and the other Auto-Padding. These are configured through the settings menus - if you set up any pre-padding or post-padding at all, then you have cancelled AR (AR is only on if padding is entirely off).

    So what are AR and padding? Padding is a time allowance added before the beginning and after the end of a programme's advertised start and end times, to allow for slippage. When you book a recording by selecting a programme in the EPG, the details from the EPG are added to the recording schedule. If the transmission times are revised in the mean time, the schedule is updated to track it. When it comes to recording the programme, the HDR-FOX fires up and starts to record at the EPG time minus the setting you have selected for the start padding, then records until the EPG end time plus the end padding you have selected. Thus you can add, say, 1 minute before to allow for a programme starting a few seconds early, and 5 minutes after to allow for it to be slightly late running.

    Without padding, the HDR-FOX tries to do something more intelligent called Accurate Recording. In addition to the EPG programme guide, the broadcasters transmit markers to say when each programme actually starts and finishes. With AR, the HDR-FOX fires up 15 minutes before the expected programme start time and looks for the start marker. When the marker arrives the recording starts, and ends when the marker for the next programme arrives. This allows for big (but not infinite) shifts in a programme time, say a live event is running late, if you are recording the live event the recording will be extended as long as required, or if you want to record a subsequent programme the recording will be delayed as long as required.

    This sounds great, but there are drawbacks. For one, if the programme is so late running that the start marker does not arrive within the programme's nominal time window, the HDR-FOX will give up and say "unable to track". Not much is lost here: using padding you would have recorded the late-running programme instead of the one you wanted anyway. The biggest drawback is that the markers are not necessarily transmitted reliably, some broadcasters are worse than others. If you miss a programme that was transmitted on time but the markers were missing, that's galling.

    Some people swear by AR, others won't use it. I suspect it depends on your viewing habits - stick to the BBC and you're probably OK. I record some things on C5 and Quest, which do not have a good reputation for marker reliability, so I use padding. I find that -1+5 gives me the complete recording in the vast majority of cases, only letting me down for something like The Last Night of the Proms which (predictably) over-ran. One inconvenience of padding occurs when you record consecutive programmes (on the same channel, eg Autumnwatch and Autumnwatch Unsprung). With AR you might reasonably expect the two recordings to switch at the right point, but with padding (which gets deleted between consecutive programmes) you might well end up with the end of the first programme being cut off and added to the beginning of the next (or vice versa).

    Unfortunately, whichever you decide to use it applies to all your recordings. Suck it and see. If all else fails there's iPlayer! However, when you have to explain to The Boss why the favourite soap didn't get recorded while she had grudgingly gone with you to see the latest block-buster action movie release, it's much easier to understand that the programme was running late than that the broadcasters cocked up the VCR control signals.

    Update: New facilities in the Custom Software now allow for mix-and-match selection of AR/padding on a programme-by-programme (WebIF schedule editor) or channel-by-channel (multimode package) basis, and as such I have been experimenting with AR specifically on StDef BBC channels. In summary, I have to say that my experience is not good, losing a significant proportion of recordings to "failed to track" errors (even just a few is not good enough), even flagship BBC1 prime-time programmes. It is reported that the HiDef channels are better, I cannot comment on that. It has become clear however that my difficulties are extreme and most people do not have the same problems.

    Further Update: My AR experience improved substantially around May 2012 when my HDR-FOX reset itself, and I have been satisfied with it ever since. Prior to that an experiment with simultaneous recordings made from the same transmitter showed that my HDR was behaving differently from another HDR. Some users still prefer padding, particularly when recording radio channels where even the BBC do not seem to get it right (it has been reported that AR is not supported for radio, so padding is needed for safety).

    Which brings me on to the subject of...

    Missed Series Recordings (lost series links)

    You're going on holiday for a couple of weeks, and 'er indoors wants to catch up on her soaps when you get back. That was your main justification for spending household budget on a fancy FreeviewHD PVR after all! So you go through the EPG and mark the soaps for series-linked recording, then come back only to find... after the first week away there are no more recordings at all. You can imagine you won't be getting your oats for a while after that!

    So what happened? This is the one real known uncorrected bug* in the HDR-FOX. If you use padding and if the box is left in stand-by for unattended recording, it never updates the EPG and therefore never re-programmes series recordings that are over the "horizon" - ie not present in the EPG at the last moment the HDR-FOX was actually turned on (not just recording). The EPG is only valid for 7 days ahead, so that's all the recording it's going to do.

    In the event that you use AR (not padding), the 15 minutes early start the Humax uses to detect and track the AR signal seems to be sufficient to also update the EPG. This removes the problem except in one extreme situation: if there is more than a week between consecutive recordings (perhaps you are set to record a fortnightly programme and that programme alone), again the subsequent programme will not be in the EPG at the time so no recording will be added to the schedule.

    Update: it has been reported that firmware revision 1.03.12 has corrected the auto-padding bug, and now auto-padding recordings behave the same as AR recordings in that they also serve to update the EPG.

    There is another problem though: occasionally programmes are re-scheduled, and if the Humax only wakes up just before transmission is due it will not be able to respond to the schedule change (which may have been present in the EPG data for several days). Thus it would be as well to refresh the EPG on a regular basis.

    We have work-arounds for this, so all is not lost as long as you are aware of the situation and take preventative action. Either:

    1) Never leave it unattended for more than a week. It gets lonely.

    2) Never let it sleep***.

    3) If neither of the above suit you, do what the rest of us do (even if you use AR): either set a manually scheduled reminder** event for 20 minutes daily, or set a timed on/off for 20 minutes daily. They achieve the same effect, except the scheduled reminder survives a firmware update but not a retune, whereas the timed on/off survives a retune but not a firmware update. The collective wisdom used to be that it is best to avoid the 4.30am automatic update slot, but it is now being investigated whether a daily scheduled reminder event (not timed on/off) for 20 mins at 4.20am would actually be an anti-OTA update strategy that does not require the custom firmware. Why would you want to do that? See HERE (click).

    How to set a timed on/off: Menu >> Settings >> Preferences >> Time >> Power On Timer / Power Off Timer

    How to set a manually scheduled reminder: Guide >> Schedule (yellow) >> New Reservation then set Channel to whatever you want it to power up on next boot, Date = tomorrow, Start Time = 04:20, End Time = 04:40, Repeat = Daily, Mode = Reminder.

    * At the time of writing. The bug which limits DLNA streaming (client) to 4GB was eliminated in 1.02.27 and later.

    ** A reminder event wakes the Humax up properly and allows it to update the EPG. A timed recording (without AR) does not. Update: the auto-padding EPG update bug appears to have been eliminated in 1.03.12.

    *** If the unit is left powered on, the EPG becomes stale unless there is activity, including channel switching. This can be circumvented using two daily reminders to toggle the channel, eg 4:00-4:20 and 4:20-4:40am.

    More Failed Recordings

    Another potential cause of failed recordings is EPG confusion. If you live within the broadcast area of more than one transmitter, you could be tuned to stations from more than one region. Unfortunately you do not then know which region's EPG and AR data is being used to set up and track recordings - it could be different from the region which is actually transmitting the programme, even (possibly) looking for different codes.

    The solution is to avoid being tuned to more than one transmitter. For more information see Things Every HD/HDR-FOX Owner Should Know (click).

    Failed Series Recording - No Attempt Made

    Usually the various failure mechanisms reported above leave some kind of evidence in the form of a "broken" recording in the media list. Sometimes you know you had a series set for recording, you know there was a programme in the series, but there is no sign of it in the media list and when you look in the schedule [Guide >> Schedule (yellow)] the reservation is present but inactive.

    This highlights another problem of divesting control to the broadcasters. Series linked recordings depend on codes sent in the EPG data called Programme CRID and Series CRID. The next programme in a series will only be automatically scheduled for recording as long as it has the same SRID and a different PRID. What sometimes happens, even on the BBC, is the SRID gets changed part way through a series. It is as well to check your schedule periodically, and ensure the expected reservations are present. (For an explanation of terms, see the Glossary.)

    PRID and SRID codes can be inspected by turning them on in a hidden settings menu - they will then be displayed in the information panels for EPG programme details and recordings. See Things Every... (click) section 9.

    The solution is to not rely on series linked recordings at all and schedule each recording yourself, but most people are not going to want to do that (and just accept the occasional failure). The custom firmware offers another solution in the form of the Remote Scheduler, which provides the means to set up recordings while away from home. RS is also capable of running a continuous search for programmes of a specific name and scheduling them for recording (as individual programmes, therefore not dependent on SRID) automatically.

    Recording Priorities

    The HDR-FOX T2 records a maximum of two programmes simultaneously (regardless of how those programmes are distributed across multiplexes or services - if they happen to be on the same multiplex, occupying only one tuner, the other tuner is available for unrestricted live viewing). Attempting to schedule a third overlapping programme for recording will produce a warning message, but what if an AR shift or padding allowances produce a three-programme conflict at recording time?

    AR users have reported their experience of how this gets resolved, and I have performed experiments with padding. Here's what happens if a recording needs to be dropped because there is a three-way conflict:

    Padding Priorities

    1. Actual EPG programme time is recorded in preference to any padding.
    2. End padding time is recorded in preference to start padding time.
    3. Padding between consecutive programmes on the same service (channel) are always dropped, regardless of there being no conflict.​

    AR Priorities

    Where a conflict occurs, the end of the existing recording is completed before the new recording starts, thus sacrificing the beginning of the next scheduled recording - see below.​

    Recordings Reported as Failed "Unable to Track"

    In the event that a recording has missed its start point (possibly by only a few seconds), it will be reported as broken on playback "unable to track". It will still play though. If there is no recording (other than a failed marker), the programme start marker did not occur while the Humax was watching for it (from 15 minutes prior to the nominal EPG time to 30 minutes or more after).

    There are other broken recording reports, eg loss of signal or loss of power, but always attempt to play the recording because frequently these are only glitches at the start and it is (to all intents and purposes) in fact intact.

    Recordings Reported as Failed "Recordings (less than 30secs) may not be stored"

    This occurs if the recording duration was very short - but is also a symptom if the "sidecar" files are corrupt (particularly the .nts). Each recording on the HD/HDR-FOX actually comprises a set of several (typically four - .ts, .hmt, .nts, .thm) files, the actual recording being the .ts file and the others being referred to as "sidecar" files which are used to present the thumbnail on the media browser (.thm), control playback (.nts), and hold the properties, synopsis, and general info (.hmt).

    The .nts is created from the broadcast to index the .ts, and recording from the new BBC3HD and BBC4HD services seems to result in a corrupt .nts at times.

    A work-around is to remove the .nts and .hmt files as long as the recording has been decrypted first. This can be accomplished via FTP (the FTP server needs to be turned on: Menu >> Settings >> System >> Internet Setting >> FTP Server = On, unless the Custom Firmware betaftp package is installed). Alternatively the recording can be copied to USB, and the files deleted by plugging the USB drive into a computer. If the recording is unplayable due to a corrupt .nts/.hmt, removing them will make the recording play (but without skip, fast-forward, etc).

    Last updated 2017/06/01
    Last edited: Jun 1, 2017
  2. af123

    af123 Administrator Staff Member

    Does anyone happen to know how long the box will watch for that initial signal before giving up? I know it starts 15 minutes before the scheduled start time, but wonder if it waits until 15 minutes after or until the scheduled end of the recording? If nobody knows I'll test it at some point over the weekend.
  3. MartinLiddle

    MartinLiddle Super Moderator Staff Member

    I certainly don't know the answer and think that would be a helpful experiment.
  4. Black Hole

    Black Hole May contain traces of nut

    How can you test it? It would involve injecting a known stimulus instead of the broadcast ones, so unless you have a means to do that you could be trying a long time before a programme comes along where the start mark deviates sufficiently from the EPG start time.
  5. af123

    af123 Administrator Staff Member

    I'll manually manipulate the scheduling database. I just need something set to record that won't be signalled by the broadcaster.
  6. jack616

    jack616 Member

    I thought I'd throw this out in case it is an issue and something could be done about it.

    For some time I've seen boxes with messages like "program not broadcast - would you like to view it anyway" or words to that effect.

    I had the chance yesterday to see a box that did this and I discovered there was a clash - not between recordings - but between the padding and a recording.
    Of course if you scrap the padding you can miss the program as explained above.

    But I dont think there was any clash notified when either recording was set - suggesting the padding time isn't included when testing for clashes.

    I dont know if thats the case or not - I just thought I'd throw it out for others to look for or comment on if its been seen before.
    It all seems tied up with the discussion in this thread.
  7. grahamlthompson

    grahamlthompson Well-Known Member

    Correct clashes created by padding aren't identified when you set the reservations. They only get picked up as the second clashing recorded is due to start. It's easy to check just make two impossible recordings with autopadding set.
  8. Black Hole

    Black Hole May contain traces of nut

    In theory I have conflicting recordings this evening - one tuner is tied up on Radio 2 (Friday Night is Music Night), the other one has to do Fifth Gear then Wild Britain and then Autumnwatch, Unsprung, and Bond. I'm sure it worked last week (but I would have accessed them from the network so wouldn't see any warning messages). I'll pay more attention this week (padding).
  9. Black Hole

    Black Hole May contain traces of nut

    FNIMN recorded its full duration inc padding, everything else had the padding lopped off as the tuner skipped from one recording to the next. No errors reported, I suspect there might have been if my schedule didn't include commercial stations with advert breaks to act as padding.

    I'm off to see if the HDR-FOX can stream a file to the HD-FOX/Qumi while it is still being recorded!
  10. Black Hole

    Black Hole May contain traces of nut

    Answer: 'course not Malcolm. It isn't available to stream until the file has been indexed in the database.
  11. Ezra Pound

    Ezra Pound Well-Known Member

    I have read on other threads that AR runs from 15 Mins before EPG start until EPG END stop time ( or more likely until End Stop time minus 15 Mins). I have a FreeView spec. somewhere I'll try and dig it out, I think it's in this doc. ETSI TS 102 323 V1.4.1
  12. rafletcher

    rafletcher Member

    As far as series recording is concerned when you're away, can you not just set a "weekly" recurring recording?
  13. Black Hole

    Black Hole May contain traces of nut

    You mean on manual control? You could but that presumes the programme will be transmitted at the same time every week, and it's much harder work than bringing up the EPG and selecting "record whole series". The 20-minute wake-up makes sure it works as it was intended to work - easy enough, fit and forget.
  14. rafletcher

    rafletcher Member

    Yep - after all that's what I've been doing for theast 5 yearsith the toppy :) although I agree a series link is a much better thing as it compensates for the vagaries of scheduling.
  15. af123

    af123 Administrator Staff Member

    I set a dummy one hour recording on my HDR for 3am today. It had event ID 1234 and a dummy CRID so it wasn't going to match anything the broadcaster sent (had some trouble faking the entry because, until I also tweaked the aulEventToRecordInfo field in the database, it kept resetting all of the recording parameters! Anyway, sure enough, today I have the recording in the media list with the lightning mark across it and trying to play it gives the expected "Recording failed: unable to track programme". Actually the only file that's there on the disk is the .hmt.

    The box was up and running from 02:45 to 03:30

    I'll schedule another shorter test for tomorrow morning, something like 12 minutes long, and see how it goes.
  16. af123

    af123 Administrator Staff Member

    My 12 minute test recording at 3am this morning resulted in the box being on from 02:45 to 03:16.. The .hmt file on disk , indicating the failed recording, is dated 03:16 too.

    I'll try another couple of different length recordings overnight tonight.
  17. af123

    af123 Administrator Staff Member

    Some more results; looks like I may need to try some more recording lengths to determine what is happening.

    | Recording Length |  Start Time | Stop Time |
    | 00:05            | -15m        | +16m     |
    | 00:12            | -15m        | +16m     |
    | 00:20            | -15m        | +16m     |
    | 01:00            | -15m        | +30m     |
  18. Black Hole

    Black Hole May contain traces of nut

    Revisiting this topic has answered a curiosity I raised in the other topic - my experience (1 hour programmes) was also that it gave up half an hour after the start.
  19. Ezra Pound

    Ezra Pound Well-Known Member

    The ETSI SPEC detailed in #11 refers to two modes or groups of identifer types of 00 or 01 / 10, For type 00 the broadcaster supplies an early_start_window = 0 to 7 Mins. and a late_end_window = 0 to 62Mins (doesn't seem to fit with thre Humax's 15 Min. 'early start' . For types 01 /10 the 'receiver implementation' generates a "monitoring window" based on now/next EPG times, It looks like it is left up to the PVR Maker

    NOTE: One of the issues for a receiver implementation in providing support for this mode is the strategy for how
    and when to look for the presence of the relevant event_identifier in the broadcast stream. An obvious
    candidate is to use the scheduled start_time and duration to define a "monitoring window". However, it
    should not be assumed that a service provider is always able to update an item of content's scheduled start
    time and duration before changes to the actual start time and duration occur. In extreme cases the
    programme may start before its scheduled start time or after it is scheduled to have finished. Despite this,
    the broadcast of either event_id and/or TVA_id may still enable accurate recording
  20. Black Hole

    Black Hole May contain traces of nut

    Thanks for the info. I note the inappropriate use of the word "assumed" in a technical document ("presumed" would be correct in this usage).