WIP: Dealing with large files for qtube & youtube-dl

[bottletop]

That's because the initial memory display for Edit1 was probably after a successful YouTube-dl. It wasn't just after a reboot (like the others). Nevertheless, the results for that run was similar and consistent with previous results.

 

[Black Hole]

So your 5.65GB tennis download got fixed up successfully?

 

[bottletop]

Yes!

 

[Black Hole]

This was an edit I've only now noticed:

So far I've found the easiest option is to use --fixup warn or --fixup never for youtube-dl.
Alternatively temporarily disable content sharing - Settings/System/Internet Setting/Content Share after queuing the download. Your HDR may slow down to a miserable speed and the HDMI audio and video will disappear near the end of the fixup process. But, it will chug along to completion. You'll need to reboot the HDR to resume normal functions.

How did you discover the content share thing, what made you think of it as a possibility? How much RAM is it freeing up? Why is a reboot needed afterwards?

 

[bottletop]

This was an edit I've only now noticed:

How did you discover the content share thing, what made you think of it as a possibility? How much RAM is it freeing up? Why is a reboot needed afterwards?

Content share - because I wasn't convinced it was only a swapfile / ffmpeg interaction issue. So I looked at my tests and noticed it look longer each time I ran similar tests when increasing the swapfile size. So that implied it was a swapfile thrashing issue, as mentioned by /df. But I gave it more thought - what about file contention? I looked at DLNA/Content Sharing. So, to get it through ffmpeg fixup of youtube-dl, the user can turn off DLNA/Content Sharing or save to /media/drive1 (if virtual-disk2 is installed).
There is the option of not using the fixup stage of youtube-dl - quicker run time but I think the resultant mp4 will not have cue/review facility. The skip forward/backward will be there though.
A reboot is needed because the HDR gets overtaxed during the last half of the fixup process. This leads to loss of TV video and sound (both live and recordings). WebIF, telnet and ssh access will slow down drastically but not actually stop, so you can check the queue progress etc.

 

[Black Hole]

But I gave it more thought - what about file contention?

I don't understand what file contention there could be. I don't think the DLNA server/indexer would be interested in what files FFMPEG is producing, neither is there anything to trigger it to look.

 

[bottletop]

I don't understand what file contention there could be.

That's fair enough. We have a difference of opinion. But bear in mind I've based mine on the results I've got from the tests I've done.

I don't think the DLNA server/indexer would be interested in what files FFMPEG is producing, neither is there anything to trigger it to look.

I think you're wrong on both counts. The DLNA server/indexer does serve MP4 files on the HDR (like those produced from iPlayer via YouTube-DL) - if they're saved in the media directory My Video (or sub directories). While I can't find the interval for the DLNA server/indexer I suspect the trigger for the service is obviously turning the Content Share on.

 

[prpr]

I've had exactly the same effects as bottletop when trying to download stuff. it just kills the normal operation. The last one I tried needed two reboots to complete.
Never thought about DLNA, but it seems like an obvious thing, now you mention it. I will try the next download with Content Share turned off.
Of course, it would be better if it downloaded to somewhere outside "My Video" and then moved it after the process was complete.

 

[bottletop]

I've had exactly the same effects as bottletop when trying to download stuff. it just kills the normal operation. The last one I tried needed two reboots to complete.
Never thought about DLNA, but it seems like an obvious thing, now you mention it. I will try the next download with Content Share turned off.
Of course, it would be better if it downloaded to somewhere outside "My Video" and then moved it after the process was complete.

Yes, someone who actually understands! To get it to work, very large swap file, no DLNA on the file ffmpeg is writing to, lots of patience, etc.

Content share - because I wasn't convinced it was only a swapfile / ffmpeg interaction issue. So I looked at my tests and noticed it look longer each time I ran similar tests when increasing the swapfile size. So that implied it was a swapfile thrashing issue, as mentioned by /df. But I gave it more thought - what about file contention? I looked at DLNA/Content Sharing. So, to get it through ffmpeg fixup of youtube-dl, the user can turn off DLNA/Content Sharing or save to /media/drive1 (if virtual-disk2 is installed).
There is the option of not using the fixup stage of youtube-dl - quicker run time but I think the resultant mp4 will not have cue/review facility. The skip forward/backward will be there though.
A reboot is needed because the HDR gets overtaxed during the last half of the fixup process. This leads to loss of TV video and sound (both live and recordings). WebIF, telnet and ssh access will slow down drastically but not actually stop, so you can check the queue progress etc.

The bold bit will be of interest, because it's not indexed by DLNA.
But the HDR will slow to snails pace and lose AV etc.

 

[Black Hole]

Why do either of you think DLNA is so problematic? What does it have to do to index a file other than record its presence?

I could understand that the entire sweep for index changes competes for system resources, but to my mind that will happen even with the downloaded file elsewhere.

 

[MymsMan]

If DLNA processing is causing a problem how about downloading to a directory outside the media directories (e.g. /mod/youtube) and then moving it the actual target directory after the download and fixup is complete

 

[prpr]

Why do either of you think DLNA is so problematic? What does it have to do to index a file other than record its presence?

Who knows what wackiness goes on in the Humax code.

how about downloading to a directory outside the media directories (e.g. /mod/youtube) and then moving it the actual target directory after the download and fixup is complete

Did you read #28 ?

 

[bottletop]

If DLNA processing is causing a problem how about downloading to a directory outside the media directories (e.g. /mod/youtube) and then moving it the actual target directory after the download and fixup is complete

Did you read #236 #28 ?

That was why I suggested an alternative (to switching Contect Share Off) in #233 #25
If necessary install virtual-disk2 and output to
"/media/drive1/mydir/" (store file out of DLNA indexing reach, easy access via Humax UI eg Media-Video/Storage(blue)/USB)

Then optionally move it to target directory (My Video & sub directories) if you wish to serve it via Humax DLNA.
I have tested the above and can confirm it works but don't forget the fixup processing will slug the HDR.

So far I've found the easiest option is to bypass fixup processing using --fixup warn or --fixup never for Youtube-DL. The result will be an MP4 without cue/review functions.
If you require fixup (so that cue/review on MP4 will work) then try the following - at the expense of extra 1-2 Hours processing and HDR crawling to glacier speed, losing audio & video output to HDMI and eventually requiring reboot(s).

I used these options in /mod/etc/youtube-dl.conf​

Better resolutions are available, but I don't know how well the HDR will cope.​

-f "best[height<=?576][fps<=?60]" (to keep file size down)​

-o "/media/drive1/dir1/%(title)s.%(ext)s" (store file out of DLNA indexing reach, easy access via Humax UI eg Media-Video/Storage(blue)/USB)​

Worked on​

Eastbourne - uninterrupted day 4​

Requires 400MB swapfile​

Result 5.65GB mp4​

​

Eastbourne - uninterrupted day 5​

Requires 625MB swapfile​

Result 9.78GB mp4​

 

[Black Hole]

Who knows what wackiness goes on in the Humax code.

I suppose we'll have to put it down to lucky accident that this effect was discovered, because it makes no sense to me.

Perhaps we need something similar to Maintenance Mode, with settop not running but the HDD still online and WebIF operational. That will free up maximum resources, similar to running fixdisk – especially if unnecessary WebIF background processes are also suspended. I guess youtube-dl can be run on the command line in Maintenance Mode without the qtube front-end, but still needs access to the HDD.

 

[Black Hole]

(No longer relevant – thread split out)

It strikes me this whole discussion ought to be moved into the youtube-dl thread.

Post 209 onwards, post 212 will need patching up, and delete this post.

 

[Black Hole]

-o "/media/drive1/dir1/%(title)s.%(ext)s" (store file out of DLNA indexing reach, easy access via Humax UI eg Media-Video/Storage(blue)/USB)

"/media/drive1" only necessarily applies if the user doesn't have any external storage plugged in. Is there a path string which will target virtual USB unambiguously?

I've summarised and patched this information into my guide to qtube.

 

[bottletop]

I suppose we'll have to put it down to lucky accident that this effect was discovered, because it makes no sense to me.

It was a little more than blind luck. But hey.

Perhaps we need something similar to Maintenance Mode, with settop not running but the HDD still online and WebIF operational. That will free up maximum resources, similar to running fixdisk – especially if unnecessary WebIF background processes are also suspended. I guess youtube-dl can be run on the command line in Maintenance Mode without the qtube front-end, but still needs access to the HDD.

That's not a bad idea. But as this is a niche issue, I'm not sure it'll be worth it. Anyone wishing to download these large files and require fixup are probably better off using a pc or laptop.

It strikes me this whole discussion ought to be moved into the youtube-dl thread.

Post 209 onwards, post 212 will need patching up, and delete this post.

That's also not a bad idea. But there is the small issue of - my report on #209 #1 reboot loop between qtube , Youtube-DL and as it happens a bug in swapper (at that time) that led me to notice the bug in swapper and the fact the reboot loop is still there (because I couldn't stop the qtube queue). It's not all blind luck!

As an alternative - maybe append it to
https://hummy.tv/forum/threads/what-is-an-malformed-aac-bitstream.11136/ rather than qtube thread?
Or or maybe on a new thread of it's own with links to the other two?

 

[Black Hole]

But as this is a niche issue, I'm not sure it'll be worth it.

Who knows what other uses a "Non-PVR Mode" could be put to? Or maybe "PC Mode".

But there is the small issue of...

There may be a bug in the interaction between qtube and youtube-dl.

...so you could have made your initial report in either thread and the discussion would have continued the same. 20/20 hindsight.

As an alternative - maybe append it to
https://hummy.tv/forum/threads/what-is-an-malformed-aac-bitstream.11136/ rather than qtube thread?

Fine by me.

Or or maybe on a new thread of it's own with links to the other two?

I don't think that would be a problem either. Click "report" and describe what you want the on-duty mod to do (your call). If splitting to a new thread, supply a thread title (but as yours will become post 1, you'll have title-editing privilege).

It's not all blind luck!

Did I say "blind" luck? It takes skill to spot these things. But I still don't know how you came up with the interaction with DLNA.

 

[MymsMan]

Did you read #236 #28?

missed it

 

[Black Hole]

That's not a bad idea. But as this is a niche issue, I'm not sure it'll be worth it. Anyone wishing to download these large files and require fixup are probably better off using a pc or laptop.

If you are willing, try enabling a swap file and running your youtube-dl command (with fix-up) in the shell in Maintenance Mode.