[suggestion] Pooling Recordings

Black Hole

May contain traces of nut
This is to start a discussion how this might be done, moved into the CF section following on from an enquiry from an "independent" developer taking about this kind of thing but trying to use bespoke PC software and using unmodified HDR-FOXes on different home networks. A solution extending the existing facilities of the CF and RS has to be a better way to go than starting from scratch, and implementing it completely within the HDR-FOX+CF+RS environment means it is not necessary to have a central PC running to manage it all.

The original topic is here: https://hummy.tv/forum/threads/developing-an-app-for-fox-t2.7759/

I have also previously posted about the possibility of using our Humii as a recording hive, I'll post a link to that when I find it.

The request is to make recordings that exist on a group of friends HDR-FOXes available to each other. Clearly there are going to be bandwidth considerations, particularly the uplink speed, so local buffering will be necessary, and the subscribers will need to watch their data caps. That aside...
 
OP
Black Hole

Black Hole

May contain traces of nut
What would it take to create the equivalent of a VPN between a "cooperating set" of non-co-networked HDR-FOXes? Could it be done using RS to track the IP addresses of (registered) participating units via their Unique IDs, and hand them out on request from a particular unit in that set?

Once a unit has the current Internet IP address for the network of another participating unit, how is it possible to ping that unit directly when it has another IP address on the local network (this is a bit of networking I don't fully understand - presumably something to do with port forwarding, so maybe the router has to be specifically configured to translate a port number on the outward facing IP address into an IP address on the local network)?

Assuming this step can be done (and I imagine it can), it is therefore possible to send DLNA index queries to the remote unit. (Alternatively SMB or NFS could be used to query the file system directly, but that would assume pre-decryption whereas DLNA will work regardless of decryption - and I recommend not getting into HiDef!). A local representation of the remote content can be built up... but can that be added to the local DLNA database for easy access via the menus? Maybe, instead of adding to the existing index, an in-the-middle approach could be used to create a separate index on the local HDR-FOX, or on RS.

I come back to thinking that the need for deferred playback is preventing all this working within the standard on-screen menus. It might be possible to use the TV Portal (we know we can pause it to complete the download), but that ties up the box for the duration of the download. It would be better to trigger a download which completes in the background without affecting the normal operation - and that implies using WebIF.

Okay, so we want to download a recording from elsewhere to be available to play at a later time (when the download is sufficiently complete to play to the end without breaking). Using DLNA at the server end overcomes any requirement to have the content pre-decrypted. If wget can be pointed at the content, via come kind of IP sharing mechanism, it can be made to get on with the download in the background at whatever data rate is available. The resulting content can be in a special folder within the normal on-screen Media list, and the display name of the recording can be updated on the fly to show how many percent of the total has been downloaded so far (changing to just the name of the programme when the download is complete - so you would see in the on-screen media list "(11%) The Gadget Show", for example).

So all that is left is to have a WebIF extension for handling all this - registering the co-operating set with RS, listing the content available on remote units, and initiating the download of a particular recording (which needs to be able to resume even if the remote unit lost contact). That sounds a piece of piss compared with EPG pooling and the TV Diary package.

This has got to be a better way to go than something completely bespoke. And, if you'll forgive me for saying, could be available to use in a matter of weeks or even a few days if the relevant experts get behind it, whereas an independent (little spare time) project will take months or years to complete.
 

EEPhil

Number 28
This has got to be a better way to go than something completely bespoke. And, if you'll forgive me for saying, could be available to use in a matter of weeks or even a few days if the relevant experts get behind it, whereas an independent (little spare time) project will take months or years to complete.
A fair point. It depends on whether there is any demand for such a service.
At least if you're going down the CF route, I won't be able to offer any advice :(.
 

Bartyblaster

New Member
I like the sound of this. Also, if the "wget" works against a Humax without CF, then some members/friends could be willing to share their recordings, without having to install CF. A simple website out on the net could track friends/members and their IP addresses, to enable the peer-to-peer request to be made. Port forwarding of the DNLA ports would need to be configured on the router (not sure how happy people would be to do this). Also, I'm sure I recollect that extending DNLA outside of your home network is problematic, perhaps due to ISP constraints. Does anyone know? This might lead to the need to set up a VPN in any case.
 
OP
Black Hole

Black Hole

May contain traces of nut
That would be throwing the baby out with the bathwater.

Can I take it you have not actually tried installing the CF and having a good look around? Installation is easy and very low risk (I might even say zero risk, but you can never say never). You really need to see what's available before you reject it as a suitable approach, there are enormous benefits.

Quick Guide to Custom Firmware (click)

It's fine if you want to do this as a project for personal skills development reasons, but be prepared for a long hard road. If you want something that works and works soon, go CF.

At least if you're going down the CF route, I won't be able to offer any advice :(.
There would still need to be a fair amount of poking around to do to get the various protocols sorted out, so I don't see why not.
 
Last edited:
OP
Black Hole

Black Hole

May contain traces of nut
A simple website out on the net could track friends/members and their IP addresses, to enable the peer-to-peer request to be made.
That's where RS (the on-line server that makes Remote Scheduling possible) comes in. It already receives incoming status updates from registered units, and is operated and maintained as part of the CF ecosystem.

Port forwarding of the DNLA ports would need to be configured on the router (not sure how happy people would be to do this).
Services such as TeamViewer are able to get around this, I can remote-control friends' PCs when they get into trouble, and I have not needed to resort to setting up routers specifically (theirs or mine).

Also, I'm sure I recollect that extending DNLA outside of your home network is problematic, perhaps due to ISP constraints. Does anyone know? This might lead to the need to set up a VPN in any case.
I don't see why it should be, as per the answer above.
 

EEPhil

Number 28
There would still need to be a fair amount of poking around to do to get the various protocols sorted out, so I don't see why not.
I know nothing about CF, as I can't use it on a 2000T. I have set a watch on this thread and will comment if anything obvious pops up.
Also, I'm sure I recollect that extending DNLA outside of your home network is problematic, perhaps due to ISP constraints. Does anyone know? This might lead to the need to set up a VPN in any case.
I hadn't though of this before - so this might throw a spanner in the works. DLNA devices deal with a multicast address (239.255.255.250 port 1900). As far as I can see, this is local to your network. It would not work on the open internet. It might be possible to use the multicast address in a VPN, although I'm having a problem locating such a solution on google.

[Edit] Having thought about it. It may not be such a problem if any shared recordings have another way of passing data about themselves to other Humaxes (Humii?). On the local network, the multicast address is used for device discovery. If you know the device exists and the contents already, you might not need the local multicast. If you know the IP address of the Humax or router of a friend and it is possible to route a request through the router to port 9000 of the Humax, and you know the file you are looking for it should be possible to stream (wget) the file. On my small local network I can wget http://192.168.1.10:9000/web/940.ts . I expect that if your router's IP address is 212.58.246.91 (www.bbc.co.uk) and somehow you can get port 1234 to link to your Humax on port 9000, anyone could access http://212.58.246.91:1234/web/940.ts and get this file. The trick is to let people know this file exists and what it contains without using the multicast address.
(The hyperlinks are not meant to be hyperlinks, I tried to unlink them but the BB software has put them back.)
 
Last edited:

prpr

Well-Known Member
If you know the IP address of the Humax or router of a friend and it is possible to route a request through the router to port 9000 of the Humax, and you know the file you are looking for it should be possible to stream (wget) the file.
Anyone who opens this stuff up to the internet is stupid.
As I said yesterday, how are the addresses going to be managed (given the the majority of people will have a dynamic IP), how are you going to handle authentication and encryption and how are you going to deal with the lack of bandwidth?
You'd need some sort of VPN and the T2 hasn't got anything like the CPU poke needed, which means external equipment is required, to solve the first ones. You need at least 40/10 fibre to solve the last one (that shitty 40/2 that various ISPs are peddling is barely more use than ADSL).

It's a duck. And it ain't quacking...
 

renzz

Member
What about using Syncthing (https://syncthing.net/) to do the donkey-work of actually synchronising the files? It's basically an open source peer-to-peer app using Bittorrent tech to get remote systems syncing data. Essentially, you create a folder on your box and add it to Syncthing which generates a secure private key. Give that to whoever you want to sync with, they add it to their Synchthing and point at an empty folder - and that's it. Whatever either of them adds to, or removes from, the folder is synchronised between the two of them (or more that two). No need to worry about network ports or IP addresses - it all happens automatically. You can make folders read-only so whoever you share with can't delete your stuff if you want.

I've not personally used Syncthing but I am using Resillio BTSync, which was the original incarnation of this until it went closed source (and Syncthing was then written by people who wanted to keep everything open). I use it to sync stuff from a remote server and it works very well. In fact it really is set it and forget it and rarely do I need to check if it's working or not.

I guess af123 would need to see if it can be added as a package to the Hummy before we can see if it does what we want.
 

EEPhil

Number 28
Anyone who opens this stuff up to the internet is stupid.
Don't shoot me, I'm just suggesting how you could do it, if you wanted to go the DLNA route.
Personally, I agree with most of your comments.
I'll leave it to others (those who are interested in using such a service - I'm not) to decide if the whole thing is too risky (without VPN I think it is), or too slow (probably).
 

Mike2

Forum Supporter
... and the issue of the legality (or not) of serving copyrighted material over the internet hasn't been mentioned yet.
It's illegal. End of project. Period.

On the other hand, I once allowed someone near Glasgow to stream a recording of a bio of RVW that they had forgotten to record. That was a one off. I just stuck the file outside my firewall, and hoped nobody else noticed.
 
Last edited:

EEPhil

Number 28
I thought of mentioning the legalities of the project, but thought I'd be criticised for bringing it up. After all, the legality of the CF itself is questionable - circumventing the encryption on HiDef to allow copying, hacking the firmware...:)
 

rpb424

Active Member
Any criticism would be entirely unjustified. It immediately struck me as the elephant in the room before even reading any of the ensuing technical debate.

Whilst I fully accept that the cause would be in a spirit of benevolence and 'not for profit', I doubt the broadcasters would see it like that.
 
OP
Black Hole

Black Hole

May contain traces of nut
Doing it may breach copyright (that's all, not a Crown offence), but providing the facility doesn't.
 
Top