[youtube-dl] Download files from youtube.com or other video platforms

[Ian Manning]

Have you set "show advanced packages"?

Aaaahhh...thank you!

 

[Black Hole]

You're panicking, and not considering whatever's wrong might be just finger trouble / operator error.

 

[/df]

Are there any up-to-date instructions for how to install the youtube-dl package? It's not listed in the "Available" list in the web IF, and the instructions from Github give this error:

humax# wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl
wget: not an http or ftp url: https://yt-dl.org/downloads/latest/youtube-dl

This should work if the wget package is installed (ie using /mod/bin/wget). The Busybox wget (/bin/wget) in the firmware image doesn't handle HTTPS. Because of the way the download site is configured, http://yt-dl.org/downloads/latest/youtube-dl redirects to the same secure link and fails in the same way.

 

[Ian Manning]

This should work if the wget package is installed (ie using /mod/bin/wget). The Busybox wget (/bin/wget) in the firmware image doesn't handle HTTPS. Because of the way the download site is configured, http://yt-dl.org/downloads/latest/youtube-dl redirects to the same secure link and fails in the same way.

Thanks - I managed to find it in the "Available" list of the web IF after following Black Hole's advice and turning on the "Advanced" option

 

[prpr]

This should work if the wget package is installed

But it won't work properly even if it could be downloaded because some of the files are patched in the repository package.

 

[thule]

When I set the media_set to iptv-uhd in youtube-dl, I get UHD content listed, BUT the files are either "audio only" or "video only". Anyone have any idea on why this is happening? Is the common practice to tell youtube-dl to merge audio+video?

 

[Black Hole]

It is not unusual for video and audio steams to be provided separately, otherwise the number of files required to support multiple video standards and multiple audio standards explodes.

 

[/df]

yt-dl by default merges (using ffmpeg) the best video-only stream with the best audio-only stream, or gets the best combined AV stream if either of those is not available. This can be overridden with -f .../--format ...: review Format Selection in the manual.

Incidentally, @prpr, breaking YT changes mean that a new patch release of the package here would be good, and would also test that I haven't broken anything for the ancient Python that it has to use.

 

[prpr]

Incidentally, @prpr, breaking YT changes mean that a new patch release of the package here would be good, and would also test that I haven't broken anything for the ancient Python that it has to use.

It's been on my mind, as I do read the update logs, even if I don't understand most of it.

 

[prpr]

I see you did another update yesterday, but testing the other day I found it was trying to use a different nsig every single time, even when re-downloading what I'd just downloaded and deleted. This meant roughly 11 minutes of tedium before it even tried downloading any media, which then went at the speed expected.

 

[/df]

The player JSON from which we run the code to descramble the nsig is supposed to be cached. Specific descrambled nsigs are cached during any yt-dl run. The player cache can be specified with this option which is accurately documented:

--cache-dir DIR
              Location in the filesystem where youtube-dl can store some down‐
              loaded      information      permanently.       By       default
              $XDG_CACHE_HOME/youtube-dl  or ~/.cache/youtube-dl . ...

So, since ~ is RO, export XDG_CACHE_HOME=/mod/var/cache ?

This seems familiar: wasn't this discussed before? Yes, post #552, so perhaps that isn't the fix that's needed.

 

[prpr]

It's not player cache like before. Here's the output from the latest build a couple of days ago:

humax# youtube -v https://www.youtube.com/watch?v=fBKScHaUo2I
[debug] System config: [u'--restrict-filenames', u'--prefer-ffmpeg', u'-f', u'best[height<=?1080][fps<=?60]', u'-o', u'/mnt/hd2
/My Video/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--external-downloader', u'wget', u'--hls-prefer-native', u'--cache-dir', u'/mod/.cache/youtube-dl
', u'-v', u'https://www.youtube.com/watch?v=fBKScHaUo2I']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2023.07.05
[debug] Python 2.7.1 (CPython 7405b0-smp 32bit) - Linux-2.6.18-7.1-7405b0-smp-with-libc0 - OpenSSL 1.0.0a 1 Jun 2010 - libc 0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[youtube] fBKScHaUo2I: Downloading webpage
[youtube] fBKScHaUo2I: Downloading player 1dda5629
[debug] [youtube] Decrypted nsig zROm0vk_ydYgrqth => bpxFOUJz11WpOQ
[debug] [youtube] Decrypted nsig 4sEOfLHP--BeSaCl => lEBErbvWHoGzEw
...

New in /mod/.cache/youtube-dl/youtube-nsig/
-rw------- 1 root root 5636 Fri 7 Jul 2023 17:40:44 1dda5629.json

Deleting and repeating:

...
[youtube] fBKScHaUo2I: Downloading webpage
[debug] [youtube] Decrypted nsig f25UQ6FZ0gf92pT6 => zjMAI9TXpmVWTw
[debug] [youtube] Decrypted nsig _XY4FVhtLoWv5Ahz => 7syLntHcV-40sw
...

Nothing new appeared in /mod/.cache/youtube-dl/youtube-nsig/

Roughly 9 minutes to get to this point both times.

 

[/df]

Deleting the downloaded content ?

Then maybe this player is sufficiently more complex that the descrambling takes that long. Overnight (?) queued download is the answer.

There is a possible work-around. The YT extraction tactics used in yt-dl and its fancier fork yt-dlp are described here. yt-dl could identify as an Android app and bypass the descrambling, but fewer formats are available. Some core changes would be necessary to be able to control the client selection as well. Overall, it's not at the top of the list.

 

[Black Hole]

yt-dl could identify as an Android app

Better than identifying as a cat I suppose, but still sounds terribly woke.

 

[Black Hole]

I'm not having much luck with yt-dl lately, are we due an update?

30/09/2023 14:37:02 - Starting queued download URL https://youtu.be/k6Vu7SRMlFk Options -x --audio-format mp3 QID 13287
30/09/2023 14:38:10 - [youtube] k6Vu7SRMlFk: Downloading webpage
30/09/2023 14:38:12 - [youtube] k6Vu7SRMlFk: Downloading API JSON
30/09/2023 14:38:14 - Caught error: WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
30/09/2023 14:38:14 - -code 1 -level 0 -errorinfo {{} /mod/webif/plugin/qtube/queue.hook 29} -errorcode {CHILDSTATUS 18130 1}

03/10/2023 10:14:02 - Starting queued download URL https://www.bbc.co.uk/programmes/m001qw8x Options -x --audio-format mp3 QID 13292
03/10/2023 10:14:34 - [bbc.co.uk] m001qw8x: Downloading video page
03/10/2023 10:14:35 - [bbc.co.uk] m001qw8x: Downloading playlist JSON
03/10/2023 10:14:42 - Caught error: ERROR: Unable to download JSON metadata: '' (caused by BadStatusLine("''",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
03/10/2023 10:14:42 - -code 1 -level 0 -errorinfo {{} /mod/webif/plugin/qtube/queue.hook 29} -errorcode {CHILDSTATUS 22220 1}

 

[EEPhil]

I don't use a Humax or CF for YouTube. FWIW an Android tablet identifies the yt-dlp it uses as updating on 24/9/23, so an update is probably required. (This version downloads from yt without any problems. )

 

[prpr]

'm not having much luck with yt-dl lately, are we due an update?

I can't get either of those to work even with the latest version:

humax# youtube https://youtu.be/k6Vu7SRMlFk --verbose -x --audio-format mp3
[debug] System config: [u'--restrict-filenames', u'--prefer-ffmpeg', u'-f', u'best[height<=?1080][fps<=?60]', u'-o', u'/mnt/hd2/My Video/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--external-downloader', u'wget', u'--hls-prefer-native', u'--cache-dir', u'/mod/.cache/youtube-dl', u'https://youtu.be/k6Vu7SRMlFk', u'--verbose', u'-x', u'--audio-format', u'mp3']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2023.09.24
[debug] Python 2.7.1 (CPython 7405b0-smp 32bit) - Linux-2.6.18-7.1-7405b0-smp-with-libc0 - OpenSSL 1.0.0a 1 Jun 2010 - libc 0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[youtube] k6Vu7SRMlFk: Downloading webpage
[youtube] k6Vu7SRMlFk: Downloading API JSON
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/YoutubeDL.py", line 859, in wrapper
    return func(self, *args, **kwargs)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/YoutubeDL.py", line 955, in __extract_info
    ie_result = ie.extract(url)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/youtube.py", line 2128, in _real_extract
    self._sort_formats(formats)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/common.py", line 1444, in _sort_formats
    raise ExtractorError('No video formats found')
ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

humax# youtube https://www.bbc.co.uk/programmes/m001qw8x --verbose -x --audio-format mp3
[debug] System config: [u'--restrict-filenames', u'--prefer-ffmpeg', u'-f', u'best[height<=?1080][fps<=?60]', u'-o', u'/mnt/hd2/My Video/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--external-downloader', u'wget', u'--hls-prefer-native', u'--cache-dir', u'/mod/.cache/youtube-dl', u'https://www.bbc.co.uk/programmes/m001qw8x', u'--verbose', u'-x', u'--audio-format', u'mp3']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2023.09.24
[debug] Python 2.7.1 (CPython 7405b0-smp 32bit) - Linux-2.6.18-7.1-7405b0-smp-with-libc0 - OpenSSL 1.0.0a 1 Jun 2010 - libc 0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[bbc.co.uk] m001qw8x: Downloading video page
[bbc.co.uk] m001qw8x: Downloading playlist JSON
[bbc.co.uk] m001qw8x: Downloading legacy playlist XML
ERROR: Unable to download XML: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/YoutubeDL.py", line 2461, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/mod/lib/python2.7/urllib2.py", line 398, in open
    response = meth(req, response)
  File "/mod/lib/python2.7/urllib2.py", line 511, in http_response
    'http', request, response, code, msg, hdrs)
  File "/mod/lib/python2.7/urllib2.py", line 436, in error
    return self._call_chain(*args)
  File "/mod/lib/python2.7/urllib2.py", line 370, in _call_chain
    result = func(*args)
  File "/mod/lib/python2.7/urllib2.py", line 519, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Needs our resident expert @/df to work some magic, but he seems to have been AWOL for quite a while .

 

[/df]

For YT the problem is https://github.com/ytdl-org/youtube-dl/issues/32499. There will be a new nightly release when I push the fix for YT. Or there is a patch. A workaround using browser cookies is too hard in this case.

I'll have to look at the BBC problem, but I do have a new BBC extractor somewhere way down the stack.

 

[/df]

For Uncharted, use either https://www.bbc.co.uk/programmes/m001qw8x/episodes/player for all available episodes or (eg) https://www.bbc.co.uk/sounds/play/m001r1s4 for episode 10. https://www.bbc.co.uk/programmes/m001qw8x should probably be treated like one of these two depending on the --no-playlist option. And that still isn't in the new version.

 

[Black Hole]

Oooh, I didn't know we could use a sounds link! Thanks.