• The forum software that supports hummy.tv has been upgraded to XenForo 2.3!

    Please bear with us as we continue to tweak things, and feel free to post any questions, issues or suggestions in the upgrade thread.

[youtube-dl] Download files from youtube.com or other video platforms

/df said:
It's this problem.

yt-dl has extraction and downloading phases. Subtitles are fetched like metadata in the extraction phase, so using the Python network stack with its 2011-ish SSL; ITV, he no like. The media files are fetched in the downloading phase, where --prefer-ffmpeg or --external-downloader apply, so allowing the TLS-1.2-challenged Python 2.7.1/OpenSSL 0.0.9 (IIRC) stack to be bypassed.

Let's bypass the old stack for extraction too, by spawning the latest wget that, like ffmpeg, has been linked with OpenSSL 1.1.1. Make sure you have that version installed.

Acquire fakehttp.py from here and put it in the yt-dl extractor directory with extractor/itv.py. Edit the latter by adding the import line from the linked post before line 22 (class ITVIE(InfoExtractor):). Profit?
Click to expand...
I am not sure where to place 'fakehttp.py'. Thanks for your help.
 
In the yt-dl extractor directory where itv.py is: should be s/t like /mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/ nowadays.
 
/df said:
Let's bypass the old stack for extraction too, by spawning the latest wget that, like ffmpeg, has been linked with OpenSSL 1.1.1.
Click to expand...
Is this something that should be built in to our package then? Is it OK to do without risk of blowing something else up?
 
prpr said:
Is this something that should be built in to our package then? Is it OK to do without risk of blowing something else up?
Click to expand...
It has to be enabled and tested for each extractor, and you have to trust that I've correctly emulated what the _request_webpage() method does (I don't). One issue is fixed in this PR: otherwise an attempt to check for a valid URL can hang yt-dl.

It would be better to build a good Python 2.7.18, but that might be a forlorn hope.
 
/df said:
In the yt-dl extractor directory where itv.py is: should be s/t like /mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/ nowadays.
Click to expand...
I made the suggested changes and ITV hub is now working, downloading both videos and subtitles.
 
  • Like
Reactions: /df
/df said:
Although no-one has yet complained about it here, downloads from YouTube have begun to be throttled to ~50kB/s. This comment links to a drop-in fix for the YT extractor.
Click to expand...
@/df
I'm now getting an error with this:
Code: 
WARNING: [youtube] Couldn't parse unidentified YouTube video throttling parameter descrambling data
Near: "e.split(""))},
-162xxxx556,-296xxxx45,b,363xxxx19,701xxxx00,1xx2"
WARNING: [youtube] Invalid data type encountered during YouTube video throttling parameter descrambling transformation chain, aborting
Couldn't descramble YouTube throttling URL parameter: data transfer will be throttled
WARNING: [youtube] Couldn't process youtube video URL, please check for updates to this script
The only instance of e.split("") I can find is in youtube.py but appears to be commented out, so I'm perplexed.

I'm trying to download a specific pair of formats e.g. using -f 140,299 --merge-output-format mp4 and it's running at glacial speed.
Without these options, it runs at normal speed (with your patch), but I get sub-standard video.
Any ideas appreciated.
 
Apparently this works.

So what happens is that YT sends the code to respond to the "n" parameter challenge in the 2MB of JS that plays your videos, but in a daftly obfuscated way. The "n" parameter is made into a list and the list is transformed by a sequence of operations, of which there is a large handful, Finally the list is joined up to make the "n" response.

In player 8d287e4d YT introduced a new operation (but its code was already used in 2 existing operations: the e.split() bit), and a new calling sequence fn(tab, s, fn2()). But now we know how to respond to those.
 
Possibly. On the other hand there is a theory that the challenge response mechanism is so daft that it must have been a joke response to a YT management demand.

Anyhow, there is an unbreakable work-around, which is to run the challenge response in an improved toy JS interpreter, as yt-dlp (a version that we can't use directly because it wants Python 3.6+) does.
 
Last edited:
/df said:
Apparently this works.
Click to expand...
Can I make the following observations? (I am using your links on this site to have a working youtube-dl on an old version of Windows with Python 3.5 installed.)
This does work - after a misunderstanding and false start on my behalf (I had to patch the youtube.py into the contents of the zip file pointed to in the next post) - and improves the speed no end, for now. Also, as I haven't signed up to github I'll make the comment here - I see no evidence in my latest downloads of the problem of dummy corrupted files that one of the contributors reported.
 
/df said:
Apparently this works.
Click to expand...
Well, it's gone from this (over 16 hours and 3 failures with "ERROR: unable to download video data: HTTP Error 403: Forbidden" necessitating restarts):
Code: 
[download]  87.7% of 3.22GiB at 51.56KiB/s ETA 02:13:48
ERROR: Interrupted by user
to this:
Code: 
[download] Resuming download at byte 3031194309
[download] Destination: /mnt/hd2/My Video/xxx.mp4
[download]  90.2% of 3.22GiB at  1.08MiB/s ETA 05:00
so I think you've nailed it. Huge thanks, and I hope you're getting enough sleep!
 
EEPhil said:
Can I make the following observations? (I am using your links on this site to have a working youtube-dl on an old version of Windows with Python 3.5 installed.)
This does work - after a misunderstanding and false start on my behalf (I had to patch the youtube.py into the contents of the zip file pointed to in the next post) - and improves the speed no end, for now. Also, as I haven't signed up to github I'll make the comment here - I see no evidence in my latest downloads of the problem of dummy corrupted files that one of the contributors reported.
Click to expand...
No-one else has either, apparently; thanks for the info.

This advice regarding Windows installations that I provided on GH may be helpful.

When updating yt-dl by dropping the modified youtube.py into a directory with administrator/root permissions, it'll be necessary to force compile the new version, eg by running youtube-dl --version as admin. This applies equally to Windows and Unix-like systems, though not to the CF package since we are root there.
 
/df said:
When updating yt-dl by dropping the modified youtube.py into a directory with administrator/root permissions, it'll be necessary to force compile the new version,
Click to expand...
Thanks for that. Unfortunately compilation appears not to be an option for me. Something in my Python setup is clearly wrong. But it runs ok in script form - so I’m happy.
 
Maybe it's running Python with -B or the PYTHONDONTWRITEBYTECODE environment variable is set.

Your compiled files should appear, assuming CPython with otherwise default settings, as __pycache__\file.cpython-35.pyc
 
Ah, different definition of compile. Those .pyc files are produced. It's the compile to standalone .exe file I can't do.
 
That's fine. IIRC making the standalone exe version is called 'building' in the yt-dl repo. It exists because Windows systems typically don't have Python already. If you have Python you can just use Pythonic installation tools like pip, and your installations will look much more like those on Unix-like systems that do come with Python by default (or for which yt-dl demands Python as a pre-req).
 
Thanks for all the slightly off-topic info. Although I’m a reasonably competent programmer, I’m new to Python and the nomenclature. To be honest, I’m just grateful to get a working downloader for YouTube. Learning Python can go on the back burner.:D
 
I have 3 HDR-FOX T2s in some sort of use. Two of them have the youtube-dl package installed. The third I only added the package on Wednesday, or Thursday, lunch time. The install was a bit odd as even after a reboot the Queue Video Download icon did not appear in the WebIf. Checking for updates it then updated from something like 2021.04.26 to 2021.06.06b, despite the attempted install only occurring a few minutes before.
The package icon still hasn't appeared and looking at /mnt/hd2/mod/webif/plugin/ there isn't a directory for qtube.
I have also tried to uninstall, reboot, the re-install, but the icon and the directory remain absent.

Could this be a general issue for new users of the package?

Any ideas of what is causing this? For now I'm leaving my 2 other HDRs with 2021.04.26 just in case it's a general issue and causes a similar issue with upgrading previously working versions.
 
You must log in or register to reply here.
Back
Top