Force DLNA Index, Auto Decrypt, Auto Shrink

[af123]

A feature I would like though (and I might have a go at developing it) after having missed the dreaded popup "TSR will stop and the recording will start from the current broadcasting point. Continue?" several times - if there was a webif button to dump the missed TSR contents into a standard recording.

Nicesplice can deal with that - just needs a way to drive it from the interface.

 

[Black Hole]

Actually I won't bother posting my observations in the other thread, for they closely match those of MontysEvilTwin above!

I don't see how you make that out - this has been much more successful than I was. A better solution is to post a link to this topic from that one (which I will do imminently).

 

[Black Hole]

Sorry this is going back a bit - I have been reviewing the discussion.

Does it matter if they are deleted after a few days - by then the actual recordings should have been indexed. The helper files are just to provide a valid index to the file before it would otherwise be available.

If I have understood correctly, the point is to create a permanent proxy file set that remains permanently indexed. Then, whenever an auto-decryption is required, the decryption process updates the proxy files with symbolic links to the actual file set, which results in the new .ts being streamable (therefore decryptable) regardless of it not having been indexed in its own right.

It seems to me that placing these proxy files in their own [... folder is more appropriate than in [Deleted Items], partly because it would remove the dependency on undelete, but also because it would be a kludge.

I need to read a bit more to see if this is already covered: it seems to me that only the .ts needs a symbolic link. The sidecar files could be dummies, as long as they have the correct properties for the actual .ts. Maybe there should be several proxy sets - one for HiDef, another for StDef, and any other variations that are necessary.

 

[Black Hole]

I believe the decryption method uses "wget"! wget allows the bandwidth it uses to be limited to a specific rate (--limit-rate=RATE). Therefore the rate could be set to be "definiately slower" than the rate at which a recording will be written. That might be useful for some of the above? One would probably want a delayed start to such processing (for example: 5 minutes).

And/or the wget could also be spawned as a separate process and "suspended" if the output file is getting too large compared to the input file; then "resumed" once things look okay again (there might be some caveats; and of course it is more work to implement that as a strategy).

wget could be rate-limited to real time for any decryptions started before the end of the live recording - if this becomes the norm, decryption will start within a few minutes of the recording starting. Something which may need to be handled is what happens if the HDR shuts down immediately the recording ends, with an incomplete decryption.

 

[Oatcake]

It seems to me that placing these proxy files in their own [... folder is more appropriate than in [Deleted Items], partly because it would remove the dependency on undelete, but also because it would be a kludge.

A kludge that avoids extra directories appearing in the Media list. Especially a directory that most people will never go into. Perhaps a tiered approach would be best, having a [cf] directory, and then under this we could have [cf]/Deleted Items ; [cf]/dlna helpers ; [cf]/whatever

I need to read a bit more to see if this is already covered: it seems to me that only the .ts needs a symbolic link.

Probably true, but I created all three files ts, nts, hmt because the Tcl routine that provides the DLNA reference doesn't seem to work if all three aren't present. Something that should probably be fixed in the Tcl libraries, but not urgent while a workaround exists.

 

[Oatcake]

Let me elaborate on some of the potential problems I can foresee with TSR viewing, and why I think this might be a lot of work to turn it into a robust, slick TV viewing experience.

Problem 1.

It will be very tempting to watch near-live TV. Especially if you can hear your neighbour shouting "goal" through the wall a few seconds before you see it on screen. If you are watching very close to live the media player could request a block that hasn't yet been written in the TSR, or worse has only been half written. If this happens then at best playback WILL break up. At worst the player may crash and you'll have to find your place again by skipping forward from the start.


Problem 2.

Pausing the playback while you take a phone call. This would be OK unless the circular buffer has kicked in...

      0 ====-----------------------===== 2hrs     <- TSR
            ^                      ^
        rec point            watching point

In the ascii picture above the "=" symbol indicates "to be seen" parts of the buffer.

A big problem now occurs if the humax starts to record two programmes on separate LCN which forces a channel change. (Or maybe one of the kids changes channel in the lounge). Now the humax resets the TSR and starts to write at 0. A nice "hole" is now appearing (and growing) in the area of catchup buffer you have yet to watch.

One way to overcome this: Within (2hrs?) "toggle" the channel away, and back, to reset the write point in the TSR and prevent circular buffering. You'd also have to restart your media player just after you've done this. This would mean you can't pause live TV (or at least you must be able to catch up within 2hrs). So not a very slick solution - but it should work OK.


Possible solution to problems

To overcome these problems, I think the following architecture would be required:

  -------------      ----------------       ============================     -------
    Humax.tv          Humax media            Relaying media server,           media
    writes to   --->  server performs  -->   written by us, to overcome   --> player
    TSR               decryption             buffering problems                    
  -------------      ----------------       ============================     -------
                  ^
             This flow made
             possible by the
             symbolic links
              trick

Such a "relaying media server" would have to be written by us. It would have to somehow sense the point in the TSR where the humax is currently writing. To overcome problem 1, the relay would automatically delay sending responses to the media player until data is available in the TSR. To overcome problem 2, the relay could start its own timeshift buffer when it detects a problem could occur. Overall lots of development work and hurdles to overcome/ test here.

Ironically to implement this workaround would probably be more work than designing a streaming system from scratch, say running on a raspberry pi with a TV capture card. But I know that this isn't the whole point. Sometimes its the challenge that motivates!

 

[af123]

A kludge that avoids extra directories appearing in the Media list. Especially a directory that most people will never go into. Perhaps a tiered approach would be best, having a [cf] directory, and then under this we could have [cf]/Deleted Items ; [cf]/dlna helpers ; [cf]/whatever

It looks like there might be an even easier way to achieve this.
I've managed to manually index a recording which means that I can put the files into a usually hidden directory (I'm using /media/My Video/.dlna/) and it should work straight away without any need to force indexing.
It looks like it would also be possible to create an 'index recording' routine too but for now I'll concentrate on just enabling quick encryption (using the code that Oatcake has sent me a the base).

humax# pwd
/mod/video/.dlna
humax# ls -l
total 12K
lrwxrwxrwx 1 root root 66 Apr 10 21:47 helper.hmt -> ../[Deleted Items]/webif_autodecrypt/Hummingbird_20150405_2103.hmt
lrwxrwxrwx 1 root root 66 Apr 10 21:47 helper.nts -> ../[Deleted Items]/webif_autodecrypt/Hummingbird_20150405_2103.nts
lrwxrwxrwx 1 root root 65 Apr 10 21:47 helper.ts -> ../[Deleted Items]/webif_autodecrypt/Hummingbird_20150405_2103.ts
humax#
humax# wget http://127.0.0.1:9000/web/media/4062.TS
--2015-04-10 21:47:26--  http://127.0.0.1:9000/web/media/4062.TS
Connecting to 127.0.0.1:9000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1270259712 (1.2G) [video/ts]
Saving to: `4062.TS'

100%[====================================>] 1,270,259,712 6.61M/s  in 2m 56s

 

[af123]

Not a bad result!

humax# hmt /media/My\ Video/NCIS/New_\ NCIS_20150410_2159. | grep end:
Recording end:1428702846 (Fri Apr 10 22:54:06 2015)

... auto.log

10/04/2015 21:54:15 -  DECRYPT: /media/My Video/NCIS/New_ NCIS_20150410_2159
10/04/2015 21:54:15 -  DLNA: http://172.29.0.252:9000/web/media/4067.TS
10/04/2015 21:57:36 -  Removing/binning old copy.
10/04/2015 21:57:37 - Done... 1.08 GiB in 201.646 seconds - 5.46 MiB/s

Started decrypting about 10 seconds after the recording finished.

 

[af123]

If anyone wants to try this test version the you can install it from the command line with:

humax# opkg install http://hpkg.tv/hdrfoxt2/test/webif_1.2.2-4_mipsel.opk

Thanks to Oatcake, this version is able to decrypt files even before they are indexed.
It also triggers an automatic processing run on the right folder within a few seconds of a recording completing.

 

[MymsMan]

Thanks to Oatcake, this version is able to decrypt files even before they are indexed.
It also triggers an automatic processing run on the right folder within a few seconds of a recording completing.

The decryption without indexing is working very nicely and means that decryption and ad detection is running a lot sooner than before.

I am having mixed results with the immediate triggering of Auto processing. I see it happening for one off recordings that go directly into My Video - the decryption starts immediately after the recording ends but for series recordings I only see Flatten process starting at the next scheduled scan and Decryption starting ten minutes later (without DLNA indexing).

Note: I deliberately do not have recursive auto-Decrypt options since detectads fails if the recording moves in the interval between Decryption and the Detectads periodic task.

 

[af123]

Thanks for the feedback - it's good to get confirmation of the decryption without indexing working for somebody else. What you're seeing with the immediate triggering might be expected with the way it currently works but I'll need to check as I actually wrote that bit some months ago now.
I've found and resolved a contention issue in my local copy but this looks like something else.

 

[Oatcake]

I've tried the new code also. Decryption without DLNA indexing works brilliantly. Just a thought - now that you can force an entry into the DLNA database why not bypass the symbolic link idea and just force the actual video into the database?

I haven't seen the immediate triggering of Auto processing work yet. During testing of the decryption I used a combination of manual recordings using the REC/ STOP button and a few single timer recordings of 15 min programmes. I didn't turn the Humax off during the time I was testing, so maybe this had an impact (does immediate triggering depend on a reboot?).

 

[Drutt]

A feature I would like though (and I might have a go at developing it) after having missed the dreaded popup "TSR will stop and the recording will start from the current broadcasting point. Continue?" several times - if there was a webif button to dump the missed TSR contents into a standard recording.

Nicesplice can deal with that - just needs a way to drive it from the interface.

Details are about here if needed. Current, or previous TSR buffers (what is left of them!) can be copied out and turned into usable recordings. The snapshot can be triggered using the remote and the "magic folders" interface, or the cmdline.

 

[MontysEvilTwin]

If anyone wants to try this test version the you can install it from the command line with:

humax# opkg install http://hpkg.tv/hdrfoxt2/test/webif_1.2.2-4_mipsel.opk

Thanks to Oatcake, this version is able to decrypt files even before they are indexed.
It also triggers an automatic processing run on the right folder within a few seconds of a recording completing.

I've been running the test version of webif for a couple of days. As a tidying exercise, I reset the dlna database. It seems to have worked but now in webif>diagnostics, when I click on the DLNA server button I just get a blank page. I have tried reinstalling Web-If but this fails and I can't roll back to 1.2.2-3.

 

[af123]

Anything in webif-error.log?

 

[MymsMan]

Anything in webif-error.log?

4    at file "/mod/webif/html/dlna/dlna.jim", line 19
3    /mod/webif/html/dlna/dlna.jim:19: Error: can't read "dmsfile": no such variable
2    at file "/mod/webif/html/dlna/dlna.jim", line 19
1    /mod/webif/html/dlna/dlna.jim:19: Error: can't read "dmsfile": no such variable

 

[MontysEvilTwin]

Anything in webif-error.log?

The same two lines as MymsMan: glad it wasn't just me!

 

[af123]

Will fix, thanks. : )

 

[MymsMan]

I believe the decryption method uses "wget"! wget allows the bandwidth it uses to be limited to a specific rate (--limit-rate=RATE). Therefore the rate could be set to be "definiately slower" than the rate at which a recording will be written. That might be useful for some of the above? One would probably want a delayed start to such processing (for example: 5 minutes).

And/or the wget could also be spawned as a separate process and "suspended" if the output file is getting too large compared to the input file; then "resumed" once things look okay again (there might be some caveats; and of course it is more work to implement that as a strategy).

I have been playing around with wget (and curl which also has a limit-rate option) reading the current recording (not Time shift buffer) and both only return the amount of data that was available at the time the command is started. e.g. if you start wget 5 minutes into the recording you will only get the first five minutes of data even if you throttle the speed down so that the retrieval takes much more than 5 minutes.

Unless I have missed something in the command options (very possible/likely) to allow the retrieval of the entire stream it would be necessary to issue a series of wget commands with --continue or --start-pos options to retrieve the entire program whilst recording is in progress.

 

[MontysEvilTwin]

@MymsMan. I presume the sidecar files are needed to decrypt on the fly: the hmt file definitely is. What if you use a symbolic link to point to dummy sidecar files? If these dummy files were configured so that the recording appears to be full length from the start your method might work.