[youtube-dl] Download files from youtube.com or other video platforms

rodp · Mar 14, 2022

good news... it's also working for me too now. I wonder if it was an update / install issue of the package, or something else. Will probably never know. Anyhow, thanks for your help!

Rodp

MontysEvilTwin · Sep 26, 2022

I made sure I had the latest version of youtube-dl (2021.12.17b) and then set up a download on youtube. The download worked but was throttled: I got the error message

Code:

WARNING: [youtube] Couldn't extract YouTube video throttling parameter descrambling function name

Is there a fix for this available?

/df · Sep 26, 2022

Yes, in the upstream master branch (https://github.com/ytdl-org/youtube-dl/commit/7009bb9f3182449ae8cc05cc28b768b63030a485), but not yet in the release or in the CF package repo.

Since 2021 YT has been sending a chunk of obfuscated JS that the web player runs on the query parameter n in each download URL; downloading is throttled from the URL unless the new n value is substituted. Every so often YT changes the syntax of the JS in a way that confuses yt-dl's toy JS interpreter.

The cumulative result of this affects several files, so in this case it isn't possible just to patch in the extractor/youtube.py file.

everthewatcher · Sep 26, 2022

/df said:
Since 2021 YT has been sending a chunk of obfuscated JS that the web player runs on the query parameter n in each download URL; downloading is throttled from the URL unless the new n value is substituted. Every so often YT changes the syntax of the JS in a way that confuses yt-dl's toy JS interpreter.

Youtube-DL fork YT-DLP has had a workaround since YT introduced this. It's what I use on PCs under TarTube as a result.

/df · Sep 26, 2022

yt-dlp and yt-dl use basically the same code to unscramble the n challenge, although yt-dlp can also use an external JS interpreter. The unscrambling code has changed several times since the last yt-dl release.

prpr · Sep 26, 2022

MontysEvilTwin said:
Is there a fix for this available?

/df said:
Yes, in the upstream master branch

OK, that looks good...

/df said:
The cumulative result of this affects several files, so in this case it isn't possible just to patch in the extractor/youtube.py file.

...ah, that's not so good. So there's a a fix available, but then you're saying it can't be used?
So what does need doing? Just picking up the head of the master branch and packaging that? Or something else?

/df · Sep 26, 2022

Yes, https://github.com/ytdl-org/youtube-dl/archive/refs/heads/master.tar.gz, eg.

prpr · Sep 27, 2022

Have generated a new package, but it seems to sit at the "Downloading player" line for several minutes (7-ish), using up most of one CPU.
After that it seems to be unthrottled.

/df · Sep 27, 2022

Some .pyc left over somewhere? YoutubeSearchURLIE is definitely there (~l.3250).

prpr · Sep 27, 2022

/df said:
Some .pyc left over somewhere?

Yes, screwed up the packaging by keeping old versions of youtube.py, bbc.py and itv.py
Then made an even bigger mess by deleting my makefile. Fortunately I recovered that, otherwise...

/df · Sep 27, 2022

The player JS is ~2MB but the challenge JS found in it is cached (~/.cache/youtube-dl/youtube-nsig on my Linux desktop), so that the player JS is only downloaded when YT changes to a new player. Although the n-response calculation is probably challenging for the HD/R Python environment, it's only done once per n value per yt-dl invocation; typically all the formats have the same n value.

prpr · Sep 28, 2022

Updated version 2022.09.03 available in the Beta section of the repository.

prpr · Sep 28, 2022

/df said:
the player JS is only downloaded when YT changes to a new player. Although the n-response calculation is probably challenging for the HD/R Python environment, it's only done once per n value per yt-dl invocation; typically all the formats have the same n value.

Is the file the same for everybody? Currently I see md5sum:
605091b0bde06cfb49ebb62127f5460f /mod/.cache/youtube-dl/youtube-nsig/bd1343fa.json

Does that mean the file could be generated once and distributed as part of the package?

What is all the cr@p in .../youtube-sigfuncs/ ?

/df · Oct 14, 2022

The JS player in the YT bloat-page is ~ 2MB. The YT extractor caches the data needed for the two stages of processing to turn the raw links into playable stream URLs so as to avoid loading the player and extracting the JS fragments each time:
1. Decrypt the s query parameter to make a valid URL - cached in youtube-sigfuncs
2. Respond to the n query parameter challenge - cached in youtube-nsig.

The cache directory may not always be defined. I had this:

Code:

WARNING: Writing cache to u'/.cache/youtube-dl/youtube-nsig/1f77e565.json' failed: ...
OSError: [Errno 30] Read-only file system: '/.cache'

From the Manual:

Code:

       --cache-dir DIR
              Location in the filesystem where youtube-dl can store some down‐
              loaded      information      permanently.       By       default
              $XDG_CACHE_HOME/youtube-dl  or ~/.cache/youtube-dl .  At the mo‐
              ment, only YouTube player files (for videos with obfuscated sig‐
              natures) are cached, but that may change.

(it has)

As XDG_CACHE_HOME isn't defined, we're using ~/.cache where ~ is supposed to mean $HOME. In the post above it has been expanded to /mod as expected. In a QTube run, HOME seems to be unset. Maybe it should be set in the script that QTube calls, like this?

Diff:

 #!/bin/sh
+[ -n "$HOME" -a -d "$HOME" ] || HOME=/mod          
 python /mod/lib/python2.7/dist-packages/youtube-dl "$@"

prpr · Oct 15, 2022

/df said:
In a QTube run, HOME seems to be unset. Maybe it should be set in the script that QTube calls

Feels a bit clunky. Presumably lack of HOME ought to be dealt with elsewhere, more generally. How's it supposed to work?
Brief research seem to lead to /etc/profile and one component that loads is /mod/.env , so maybe in the latter? But what creates that?

/df · Oct 15, 2022

I assumed that HOME is normally set to /mod but either not set or set to / for the user under which queued tasks run. But perhaps it's just that /etc/profile is either not run or run as non-interactive, which means that HOME isn't set:

Code:

...
# If running interactively, then:
if [ -n "$PS1" ]; then
        ...
        export HOME=/mod
fi
...

The QTube script is run by the Jim exec command which accepts an environment list from $::env (a {name value ...} list). Then a non-interactive shell is run from the #!/bin/sh in the script. It could have a -i (interactive) option but that does other things too. Actually the clunky solution seems quite reasonable, and I did get new cached items.

There is also a --cache-dir ... option of yt-dl that could be used, but I think it should be reserved for overriding the default behaviour.

xyz321 · Oct 15, 2022

prpr said:
Brief research seem to lead to /etc/profile and one component that loads is /mod/.env , so maybe in the latter? But what creates that?

/sbin/modinit

prpr · Oct 16, 2022

xyz321 said:
/sbin/modinit

Thanks. I thought I'd searched that but must have slipped through without. Tired I guess.

/df said:
But perhaps it's just that /etc/profile is either not run or run as non-interactive, which means that HOME isn't set

You could test it by modifying /mod/.env to include it and seeing if it picks it up.

Actually the clunky solution seems quite reasonable

Yes, seems so.
Update released.

Black Hole · Oct 30, 2022

I was trying to pull down some music from YouTube yesterday (non-queued via Qtube), even -F got stuck and the CF timed it out. Let me try again:

Code:

15:57:55 - --------------------------------------------------------------
15:57:55 - Starting immediate download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -F
15:59:03 - [youtube] pBv1-AI0Klg: Downloading webpage
15:59:05 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
15:59:33 - [youtube] pBv1-AI0Klg: Downloading MPD manifest

At 16:07 a pop-up with "192.168.1.14 Says:" in it (and nothing else) appeared. The spinners were still going and I wondered about waiting some more, but switching away from that page and back refreshed the screen and the spinners were no longer there (so I guess that means it had aborted anyway).

I'll give it another go via the queue, but it's academic now because I wanted it for a presentation last night. I got around the problem by playing the video on the iPad and running a screen capture.

Black Hole · Oct 30, 2022

Hmph. It worked on the queue!

My attempts yesterday:

Code:

14:10:56 - --------------------------------------------------------------
14:10:56 - Starting immediate download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -F
14:12:21 - [youtube] pBv1-AI0Klg: Downloading webpage
14:12:23 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
14:12:51 - [youtube] pBv1-AI0Klg: Downloading MPD manifest
14:26:27 - --------------------------------------------------------------
14:26:27 - Starting immediate download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -x --audio-format mp3
14:27:59 - [youtube] pBv1-AI0Klg: Downloading webpage
14:28:01 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
14:28:29 - [youtube] pBv1-AI0Klg: Downloading MPD manifest

...and the failed attempt today:

Code:

15:57:55 - --------------------------------------------------------------
15:57:55 - Starting immediate download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -F
15:59:03 - [youtube] pBv1-AI0Klg: Downloading webpage
15:59:05 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
15:59:33 - [youtube] pBv1-AI0Klg: Downloading MPD manifest

But then success when queued rather than immediate:

Code:

30/10/2022 16:15:02 - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
30/10/2022 16:15:02 - Starting queued download URL https://www.youtube.com/watch?v=pBv1-AI0Klg Options -x --audio-format mp3 QID 8591
30/10/2022 16:16:10 - [youtube] pBv1-AI0Klg: Downloading webpage
30/10/2022 16:16:12 - [youtube] pBv1-AI0Klg: Downloading player 19fc75cf
30/10/2022 16:16:39 - [youtube] pBv1-AI0Klg: Downloading MPD manifest
30/10/2022 16:23:37 - [download] Resuming download at byte 3072
30/10/2022 16:23:38 - [download] Destination: /mnt/hd2/My Video/Dusk_Cecil_Armstrong_Gibbs_1889-1960.mp4
30/10/2022 16:23:38 - [download]   0.1% of 7.74MiB at 98.46KiB/s ETA 01:24
30/10/2022 16:23:38 - [download]   0.1% of 7.74MiB at 191.87KiB/s ETA 00:42
30/10/2022 16:23:38 - [download]   0.1% of 7.74MiB at 116.01KiB/s ETA 01:09
...
30/10/2022 16:25:53 - [download]  99.5% of 7.74MiB at 58.26KiB/s ETA 00:00
30/10/2022 16:25:53 - [download] 100.0% of 7.74MiB at 58.50KiB/s ETA 00:00
30/10/2022 16:25:54 - [download] 100% of 7.74MiB in 02:16
30/10/2022 16:25:55 - [ffmpeg] Destination: /mnt/hd2/My Video/Dusk_Cecil_Armstrong_Gibbs_1889-1960.mp3
30/10/2022 16:36:45 - Deleting original file /mnt/hd2/My Video/Dusk_Cecil_Armstrong_Gibbs_1889-1960.mp4 (pass -k to keep)
30/10/2022 16:36:46 - Completed queued download of https://www.youtube.com/watch?v=pBv1-AI0Klg Options -x --audio-format mp3

Note it says "Resuming download at byte 3072" - so yesterday's download attempt must have started before aborting. Is this some kind of browser timeout rather than anything else?

Note also the immediate actions did not get the timestamp prepended by the date. That feels like an inconsistency to me.

[youtube-dl] Download files from youtube.com or other video platforms

Member

Well-Known Member

Well-Known Member

Forum Supporter

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

May contain traces of nut

May contain traces of nut