[youtube-dl] Download files from youtube.com or other video platforms
- Thread starter prpr
- Start date
MontysEvilTwin
Well-Known Member
I made sure I had the latest version of youtube-dl (2021.12.17b) and then set up a download on youtube. The download worked but was throttled: I got the error message
Is there a fix for this available?
Code:
WARNING: [youtube] Couldn't extract YouTube video throttling parameter descrambling function nameYes, in the upstream master branch (https://github.com/ytdl-org/youtube-dl/commit/7009bb9f3182449ae8cc05cc28b768b63030a485), but not yet in the release or in the CF package repo.
Since 2021 YT has been sending a chunk of obfuscated JS that the web player runs on the query parameter
The cumulative result of this affects several files, so in this case it isn't possible just to patch in the
Since 2021 YT has been sending a chunk of obfuscated JS that the web player runs on the query parameter
n in each download URL; downloading is throttled from the URL unless the new n value is substituted. Every so often YT changes the syntax of the JS in a way that confuses yt-dl's toy JS interpreter.The cumulative result of this affects several files, so in this case it isn't possible just to patch in the
extractor/youtube.py file.
everthewatcher
Forum Supporter
OP
prpr
Well-Known Member
OK, that looks good...
...ah, that's not so good. So there's a a fix available, but then you're saying it can't be used?
So what does need doing? Just picking up the head of the master branch and packaging that? Or something else?
The player JS is ~2MB but the challenge JS found in it is cached (
~/.cache/youtube-dl/youtube-nsig on my Linux desktop), so that the player JS is only downloaded when YT changes to a new player. Although the n-response calculation is probably challenging for the HD/R Python environment, it's only done once per n value per yt-dl invocation; typically all the formats have the same n value.
OP
prpr
Well-Known Member
The JS player in the YT bloat-page is ~ 2MB. The YT extractor caches the data needed for the two stages of processing to turn the raw links into playable stream URLs so as to avoid loading the player and extracting the JS fragments each time:
1. Decrypt the
2. Respond to the
The cache directory may not always be defined. I had this:
From the Manual:
(it has)
As
1. Decrypt the
s query parameter to make a valid URL - cached in youtube-sigfuncs2. Respond to the
n query parameter challenge - cached in youtube-nsig.The cache directory may not always be defined. I had this:
Code:
WARNING: Writing cache to u'/.cache/youtube-dl/youtube-nsig/1f77e565.json' failed: ...
OSError: [Errno 30] Read-only file system: '/.cache'
Code:
--cache-dir DIR
Location in the filesystem where youtube-dl can store some down‐
loaded information permanently. By default
$XDG_CACHE_HOME/youtube-dl or ~/.cache/youtube-dl . At the mo‐
ment, only YouTube player files (for videos with obfuscated sig‐
natures) are cached, but that may change.As
XDG_CACHE_HOME isn't defined, we're using ~/.cache where ~ is supposed to mean $HOME. In the post above it has been expanded to /mod as expected. In a QTube run, HOME seems to be unset. Maybe it should be set in the script that QTube calls, like this?
Diff:
#!/bin/sh
+[ -n "$HOME" -a -d "$HOME" ] || HOME=/mod
python /mod/lib/python2.7/dist-packages/youtube-dl "$@"
Last edited:
I assumed that
The QTube script is run by the Jim
There is also a
HOME is normally set to /mod but either not set or set to / for the user under which queued tasks run. But perhaps it's just that /etc/profile is either not run or run as non-interactive, which means that HOME isn't set:
Code:
...
# If running interactively, then:
if [ -n "$PS1" ]; then
...
export HOME=/mod
fi
...exec command which accepts an environment list from $::env (a {name value ...} list). Then a non-interactive shell is run from the #!/bin/sh in the script. It could have a -i (interactive) option but that does other things too. Actually the clunky solution seems quite reasonable, and I did get new cached items.There is also a
--cache-dir ... option of yt-dl that could be used, but I think it should be reserved for overriding the default behaviour./sbin/modinit
OP
prpr
Well-Known Member
Thanks. I thought I'd searched that but must have slipped through without. Tired I guess.
You could test it by modifying /mod/.env to include it and seeing if it picks it up.
Yes, seems so.
Update released.
Black Hole
May contain traces of nut
I was trying to pull down some music from YouTube yesterday (non-queued via Qtube), even -F got stuck and the CF timed it out. Let me try again:
At 16:07 a pop-up with "192.168.1.14 Says:" in it (and nothing else) appeared. The spinners were still going and I wondered about waiting some more, but switching away from that page and back refreshed the screen and the spinners were no longer there (so I guess that means it had aborted anyway).
I'll give it another go via the queue, but it's academic now because I wanted it for a presentation last night. I got around the problem by playing the video on the iPad and running a screen capture.
Code:
15:57:55 - --------------------------------------------------------------
15:57:55 - Starting immediate download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -F
15:59:03 - [youtube] pBv1-AI0Klg: Downloading webpage
15:59:05 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
15:59:33 - [youtube] pBv1-AI0Klg: Downloading MPD manifestAt 16:07 a pop-up with "192.168.1.14 Says:" in it (and nothing else) appeared. The spinners were still going and I wondered about waiting some more, but switching away from that page and back refreshed the screen and the spinners were no longer there (so I guess that means it had aborted anyway).
I'll give it another go via the queue, but it's academic now because I wanted it for a presentation last night. I got around the problem by playing the video on the iPad and running a screen capture.
Black Hole
May contain traces of nut
Hmph. It worked on the queue!
My attempts yesterday:
...and the failed attempt today:
But then success when queued rather than immediate:
Note it says "Resuming download at byte 3072" - so yesterday's download attempt must have started before aborting. Is this some kind of browser timeout rather than anything else?
Note also the immediate actions did not get the timestamp prepended by the date. That feels like an inconsistency to me.
My attempts yesterday:
Code:
14:10:56 - --------------------------------------------------------------
14:10:56 - Starting immediate download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -F
14:12:21 - [youtube] pBv1-AI0Klg: Downloading webpage
14:12:23 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
14:12:51 - [youtube] pBv1-AI0Klg: Downloading MPD manifest
14:26:27 - --------------------------------------------------------------
14:26:27 - Starting immediate download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -x --audio-format mp3
14:27:59 - [youtube] pBv1-AI0Klg: Downloading webpage
14:28:01 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
14:28:29 - [youtube] pBv1-AI0Klg: Downloading MPD manifest...and the failed attempt today:
Code:
15:57:55 - --------------------------------------------------------------
15:57:55 - Starting immediate download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -F
15:59:03 - [youtube] pBv1-AI0Klg: Downloading webpage
15:59:05 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
15:59:33 - [youtube] pBv1-AI0Klg: Downloading MPD manifestBut then success when queued rather than immediate:
Code:
30/10/2022 16:15:02 - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
30/10/2022 16:15:02 - Starting queued download URL https://www.youtube.com/watch?v=pBv1-AI0Klg Options -x --audio-format mp3 QID 8591
30/10/2022 16:16:10 - [youtube] pBv1-AI0Klg: Downloading webpage
30/10/2022 16:16:12 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
30/10/2022 16:16:39 - [youtube] pBv1-AI0Klg: Downloading MPD manifest
30/10/2022 16:23:37 - [download] Resuming download at byte 3072
30/10/2022 16:23:38 - [download] Destination: /mnt/hd2/My Video/Dusk_Cecil_Armstrong_Gibbs_1889-1960.mp4
30/10/2022 16:23:38 - [download] 0.1% of 7.74MiB at 98.46KiB/s ETA 01:24
30/10/2022 16:23:38 - [download] 0.1% of 7.74MiB at 191.87KiB/s ETA 00:42
30/10/2022 16:23:38 - [download] 0.1% of 7.74MiB at 116.01KiB/s ETA 01:09
...
30/10/2022 16:25:53 - [download] 99.5% of 7.74MiB at 58.26KiB/s ETA 00:00
30/10/2022 16:25:53 - [download] 100.0% of 7.74MiB at 58.50KiB/s ETA 00:00
30/10/2022 16:25:54 - [download] 100% of 7.74MiB in 02:16
30/10/2022 16:25:55 - [ffmpeg] Destination: /mnt/hd2/My Video/Dusk_Cecil_Armstrong_Gibbs_1889-1960.mp3
30/10/2022 16:36:45 - Deleting original file /mnt/hd2/My Video/Dusk_Cecil_Armstrong_Gibbs_1889-1960.mp4 (pass -k to keep)
30/10/2022 16:36:46 - Completed queued download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -x --audio-format mp3Note it says "Resuming download at byte 3072" - so yesterday's download attempt must have started before aborting. Is this some kind of browser timeout rather than anything else?
Note also the immediate actions did not get the timestamp prepended by the date. That feels like an inconsistency to me.