[youtube-dl] Download files from youtube.com or other video platforms

EEPhil · Oct 18, 2023

Black Hole said:
Er, no. DVB replaces analogue using the same spectrum. It's one or the other.

Not when it first started and UHF channels 21 to 69 were available. But now, barely enough space for the DVB channels, so no going back. (Not that there is any possibility. More likely to lose even more spectrum to mobile devices)

Black Hole · Oct 18, 2023

EEPhil said:
Not when it first started and UHF channels 21 to 69 were available.

Not many of those would have been actually usable – channel allocations were to avoid interfering with geographically adjacent transmitters (which analogue was particularly susceptible to).

dodders · Nov 17, 2023

Hi, sorry for the probably duplicate post but is there an instruction page anywhere on how to get the certificate and modify youtube-dl to download uhd iPlayer?
Thanks

Black Hole · Nov 17, 2023

dodders said:
Hi, sorry for the probably duplicate post but is there an instruction page anywhere on how to get the certificate and modify youtube-dl to download uhd iPlayer?
Thanks

Considering the size of the files involved, I think it would be daft to try doing that on a HDR-FOX. The processor is slow, there is limited RAM, limited disk space, and limited throughput. Also, you can be sure the copyright holders are very keen to enforce DRM.

You might like to look into yt-dlp (a fork of youtube-dl) which has a user community on Discord: https://github.com/yt-dlp/yt-dlp. Our user community is for the HDR-FOX, which would be incapable of playing UHD files anyway.

/df · Nov 19, 2023

Why not search this thread?

The same technique would apply whether tweaking the CF youtube-dl package, or the youtube-dl master code or nightly release, or yt-dlp (whose BBC extractor is basically the same). But in the first case you might be downloading material that can't be played on the same platform.

Black Hole · Nov 19, 2023

/df said:
Why not search this thread?

Because this...

dodders said:
sorry for the probably duplicate post

...implies the OP can't be bothered to look for himself and wants spoon-feeding, within minutes of signing up and having contributed nothing. I suspect he's only here because Internet searches turn up very little about youtube-dl.

prpr · Nov 20, 2023

It's typical of the scatter-gun approach we've seen before. Register, post one non-sensical message, then never be seen again.
This thread needs deleting from post #623 onwards and the user in question removing.

/df · Jan 4, 2024

Since the ITVX extractor is probably not working now or in the future, we can get shows that are available N of the border from STV instead. Astonishingly, ITV and STV had different streaming sites even before ITVX, and the STV site is more amenable, while ITVX has joined the opaque ranks of All4, My5, etc.

To make STV work, it has to have the FakeHTTP hack (to make the extractor use wget for web access, since no-one has managed to rebuild the CF python with a new OpenSSL) applied. I've actually applied the hack to extractor/common.py to find out if any extractors don't like the hack, but a more conservative approach would be to repeat the hack applied to extractor/itv.py. Because of dependencies, the hack also has to be applied to extractor/adobepass.py.

The per-extractor hack:

Code:

-from .common import InfoExtractor
+from .fakehttp import FakeHTTP as InfoExtractor

The extractor/common.py hack (append to existing file):

Code:

_InfoExtractor = InfoExtractor

from .fakehttp import FakeHTTP as InfoExtractor

This may look as if it would fail because of cyclic dependencies (InfoExtractor is imported from extractor/common.py in extractor/fakehttp.py`) but the Python import logic sees that the name InfoExtractor has already been defined and associated with extractor/common.py and uses that (the "unhacked" class), so skipping the nested import. Or so it appears. I would not suggest using this pattern in code written from scratch.

Finally, the script /mod/bin/youtube-dl is using --hls-prefer-native which will generally fail because of the same SSL version issues. It should be --hls-prefer-ffmpeg for successful M3U8 downloads. Discussion following post #543 suggests that there was some problem with iPlayer that I haven't seen.

Example result:

Code:

# /mod/bin/youtube-dl -vF 'https://player.stv.tv/episode/4jcy/bates-vs-post-office'
[debug] System config: [u'--restrict-filenames', u'--prefer-ffmpeg', u'-f', u'(best/bestvideo+bestaudio)[height<=?1080][fps<=?60][tbr<=?3500]', u'-o', u'/mnt/hd2/My Video/%(title)s.%(ext)s', u'--no-overwrites', u'--config-location', u'/mod/.config/youtube-dl/config']
[debug] User config: [u'-f', u'(best/bestvideo+bestaudio)[height<=?1080][fps<=?60][tbr<=?1900]', u'--hls-prefer-ffmpeg']
[debug] Custom config: []
[debug] Command-line args: [u'--external-downloader', u'wget', u'--hls-prefer-native', u'--cache-dir', u'/mod/.cache/youtube-dl', u'-vF', u'https://player.stv.tv/episode/4jcy/bates-vs-post-office']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2023.10.04
[debug] Python 2.7.1 (CPython 7405b0-smp 32bit) - Linux-2.6.18-7.1-7405b0-smp-with-libc0 - OpenSSL 1.0.0a 1 Jun 2010 - libc 0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[stv:player] 4jcy: Downloading webpage
[debug] Using fake IP 25.2.163.235 (GB) as X-Forwarded-For.
[brightcove:new] 6343657716112: Downloading JSON metadata
[brightcove:new] 6343657716112: Downloading m3u8 information
[brightcove:new] 6343657716112: Downloading m3u8 information
[brightcove:new] 6343657716112: Downloading m3u8 information
[brightcove:new] 6343657716112: Downloading m3u8 information
[brightcove:new] 6343657716112: Downloading MPD manifest
[brightcove:new] 6343657716112: Downloading MPD manifest
[brightcove:new] 6343657716112: Downloading MPD manifest
[brightcove:new] 6343657716112: Downloading MPD manifest
[info] Available formats for 6343657716112:
format code                                  extension  resolution note
hls-audio-0-en__Main_-0                      mp4        audio only [en] 
hls-audio-0-en__Main_-1                      mp4        audio only [en] 
hls-audio-1-en__Main_-0                      mp4        audio only [en] 
hls-audio-1-en__Main_-1                      mp4        audio only [en] 
hls-audio-2-en__Main_-0                      mp4        audio only [en] 
hls-audio-2-en__Main_-1                      mp4        audio only [en] 
dash-850d7415-6e34-42cc-aa51-0b5189264280-0  m4a        audio only [en] DASH audio   63k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-850d7415-6e34-42cc-aa51-0b5189264280-1  m4a        audio only [en] DASH audio   63k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-850d7415-6e34-42cc-aa51-0b5189264280-2  m4a        audio only [en] DASH audio   63k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-850d7415-6e34-42cc-aa51-0b5189264280-3  m4a        audio only [en] DASH audio   63k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-26f4a706-efc7-4135-a4cc-f049af18ece4-0  m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-26f4a706-efc7-4135-a4cc-f049af18ece4-1  m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-26f4a706-efc7-4135-a4cc-f049af18ece4-2  m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-26f4a706-efc7-4135-a4cc-f049af18ece4-3  m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-7bd8c88e-ef76-44ac-aab2-16d05aeed326-0  m4a        audio only [en] DASH audio  127k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-7bd8c88e-ef76-44ac-aab2-16d05aeed326-1  m4a        audio only [en] DASH audio  127k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-7bd8c88e-ef76-44ac-aab2-16d05aeed326-2  m4a        audio only [en] DASH audio  127k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-7bd8c88e-ef76-44ac-aab2-16d05aeed326-3  m4a        audio only [en] DASH audio  127k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-0b5704c9-ffc8-4182-95a7-13503c7c1427-0  mp4        384x216    DASH video  220k , mp4_dash container, avc1.42000d, video only
dash-0b5704c9-ffc8-4182-95a7-13503c7c1427-1  mp4        384x216    DASH video  220k , mp4_dash container, avc1.42000d, video only
dash-0b5704c9-ffc8-4182-95a7-13503c7c1427-2  mp4        384x216    DASH video  220k , mp4_dash container, avc1.42000d, video only
dash-0b5704c9-ffc8-4182-95a7-13503c7c1427-3  mp4        384x216    DASH video  220k , mp4_dash container, avc1.42000d, video only
hls-311-0                                    mp4        384x216     311k , avc1.42000d, video only
hls-311-1                                    mp4        384x216     311k , avc1.42000d, video only
dash-463d68fa-fd99-4c14-8813-01072cb39f05-0  mp4        512x288    DASH video  393k , mp4_dash container, avc1.4d0015, video only
dash-463d68fa-fd99-4c14-8813-01072cb39f05-1  mp4        512x288    DASH video  393k , mp4_dash container, avc1.4d0015, video only
dash-463d68fa-fd99-4c14-8813-01072cb39f05-2  mp4        512x288    DASH video  393k , mp4_dash container, avc1.4d0015, video only
dash-463d68fa-fd99-4c14-8813-01072cb39f05-3  mp4        512x288    DASH video  393k , mp4_dash container, avc1.4d0015, video only
hls-537-0                                    mp4        512x288     537k , avc1.4d0015, video only
hls-537-1                                    mp4        512x288     537k , avc1.4d0015, video only
dash-4d33df32-c5eb-4cb6-ad38-4bbcc17d5647-0  mp4        768x432    DASH video  746k , mp4_dash container, avc1.4d001e, video only
dash-4d33df32-c5eb-4cb6-ad38-4bbcc17d5647-1  mp4        768x432    DASH video  746k , mp4_dash container, avc1.4d001e, video only
dash-4d33df32-c5eb-4cb6-ad38-4bbcc17d5647-2  mp4        768x432    DASH video  746k , mp4_dash container, avc1.4d001e, video only
dash-4d33df32-c5eb-4cb6-ad38-4bbcc17d5647-3  mp4        768x432    DASH video  746k , mp4_dash container, avc1.4d001e, video only
hls-960-0                                    mp4        768x432     960k , avc1.4d001e, video only
hls-960-1                                    mp4        768x432     960k , avc1.4d001e, video only
dash-c694e118-fcd3-41c7-a377-c5dd3255d36a-0  mp4        1024x576   DASH video 1203k , mp4_dash container, avc1.4d001f, video only
dash-c694e118-fcd3-41c7-a377-c5dd3255d36a-1  mp4        1024x576   DASH video 1203k , mp4_dash container, avc1.4d001f, video only
dash-c694e118-fcd3-41c7-a377-c5dd3255d36a-2  mp4        1024x576   DASH video 1203k , mp4_dash container, avc1.4d001f, video only
dash-c694e118-fcd3-41c7-a377-c5dd3255d36a-3  mp4        1024x576   DASH video 1203k , mp4_dash container, avc1.4d001f, video only
hls-1463-0                                   mp4        1024x576   1463k , avc1.4d001f, video only
hls-1463-1                                   mp4        1024x576   1463k , avc1.4d001f, video only
dash-5c295666-d1e0-4813-85f5-a25cdb42a699-0  mp4        1280x720   DASH video 1870k , mp4_dash container, avc1.4d001f, video only
dash-5c295666-d1e0-4813-85f5-a25cdb42a699-1  mp4        1280x720   DASH video 1870k , mp4_dash container, avc1.4d001f, video only
dash-5c295666-d1e0-4813-85f5-a25cdb42a699-2  mp4        1280x720   DASH video 1870k , mp4_dash container, avc1.4d001f, video only
dash-5c295666-d1e0-4813-85f5-a25cdb42a699-3  mp4        1280x720   DASH video 1870k , mp4_dash container, avc1.4d001f, video only
hls-2196-0                                   mp4        1280x720   2196k , avc1.4d001f, video only
hls-2196-1                                   mp4        1280x720   2196k , avc1.4d001f, video only
dash-8ab3194d-c826-4b85-b948-38d562e25915-0  mp4        1920x1080  DASH video 3554k , mp4_dash container, avc1.640028, video only
dash-8ab3194d-c826-4b85-b948-38d562e25915-1  mp4        1920x1080  DASH video 3554k , mp4_dash container, avc1.640028, video only
dash-8ab3194d-c826-4b85-b948-38d562e25915-2  mp4        1920x1080  DASH video 3554k , mp4_dash container, avc1.640028, video only
dash-8ab3194d-c826-4b85-b948-38d562e25915-3  mp4        1920x1080  DASH video 3554k , mp4_dash container, avc1.640028, video only
hls-4049-0                                   mp4        1920x1080  4049k , avc1.640028, video only
hls-4049-1                                   mp4        1920x1080  4049k , avc1.640028, video only
hls-311-2                                    mp4        384x216     311k , avc1.42000d, mp4a.40.2
hls-311-3                                    mp4        384x216     311k , avc1.42000d, mp4a.40.2
hls-537-2                                    mp4        512x288     537k , avc1.4d0015, mp4a.40.2
hls-537-3                                    mp4        512x288     537k , avc1.4d0015, mp4a.40.2
hls-960-2                                    mp4        768x432     960k , avc1.4d001e, mp4a.40.2
hls-960-3                                    mp4        768x432     960k , avc1.4d001e, mp4a.40.2
hls-1463-2                                   mp4        1024x576   1463k , avc1.4d001f, mp4a.40.2
hls-1463-3                                   mp4        1024x576   1463k , avc1.4d001f, mp4a.40.2
http-2123k-720p-0                            mp4        1280x720   2123k , MP4 container, H264, 715.04MiB
http-2123k-720p-1                            mp4        1280x720   2123k , MP4 container, H264, 715.04MiB
hls-2196-2                                   mp4        1280x720   2196k , avc1.4d001f, mp4a.40.2
hls-2196-3                                   mp4        1280x720   2196k , avc1.4d001f, mp4a.40.2
hls-4049-2                                   mp4        1920x1080  4049k , avc1.640028, mp4a.40.2
hls-4049-3                                   mp4        1920x1080  4049k , avc1.640028, mp4a.40.2 (best)
#

Black Hole · Jan 5, 2024

I have suddenly acquired an urgent need for this. The box I was recording the Post Office scandal dramatisation on crashed before the last episode!

EEPhil · Jan 5, 2024

Black Hole said:
I have suddenly acquired an urgent need for this.

It's only TV.

I've certainly found the STV player/yt-dlp useful.

/df · Jan 5, 2024

The urgent need could indeed be handled by using a youtube-dl nightly version (works on elderly Windows, too), or the downstream yt-dlp, on a grown-up machine. Or add the 3-line hack to extractor/common.py (from memory, in /mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/).

I'm looking forward to Season/Series 2, where PO managers stand trial for conspiracy to pervert the course of justice (please) along with any PO lawyers not deemed too incompetent to have done so, compensation funds are reimbursed by Fujitsu, Bates is offered (but probably refuses) a CH or GBE, and the presumed validity of computer-generated evidence is overturned, restoring the position at PACE(1984).

Black Hole · Jan 5, 2024

/df said:
PO managers stand trial for conspiracy to pervert the course of justice (please)

Agreed. They knowingly caused people to be wrongfully imprisoned (and worse), and nothing less than them serving time behind bars will do.

/df said:
Or add the 3-line hack to extractor/common.py (from memory, in /mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/).

I'll give it a go, but if that doesn't work settle for ITVX on my iPad. Will report back...

Black Hole · Jan 5, 2024

Meh.

First I tried without the mod, as a baseline...

Code:

05/01/2024 15:25:02 - Starting queued download URL https://player.stv.tv/episode/4jd1/bates-vs-post-office Options  QID 13565
05/01/2024 15:26:28 - [stv:player] 4jd1: Downloading webpage
05/01/2024 15:26:28 - [stv:player] 4jd1: Downloading JSON metadata
05/01/2024 15:26:29 - Caught error: WARNING: Unable to download webpage: <urlopen error [Errno 1] _ssl.c:499: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>
ERROR: Unable to download JSON metadata: <urlopen error [Errno 1] _ssl.c:499: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure> (caused by URLError(SSLError(1, '_ssl.c:499: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure'),))
05/01/2024 15:26:29 - -code 1 -level 0 -errorinfo {{} /mod/webif/plugin/qtube/queue.hook 29} -errorcode {CHILDSTATUS 11866 1}

...then inserted the extra lines in common.py (having discovered the diagnostics file editor complained the file is too big, installed nano, figured out how to use it...):

Code:

05/01/2024 15:39:02 - Starting queued download URL https://player.stv.tv/episode/4jd1/bates-vs-post-office Options  QID 13565
05/01/2024 15:42:18 - [stv:player] 4jd1: Downloading webpage
05/01/2024 15:42:21 - [brightcove:new] 6343657635112: Downloading JSON metadata
05/01/2024 15:42:22 - [brightcove:new] 6343657635112: Downloading JSON metadata
05/01/2024 15:42:22 - [brightcove:new] 6343657635112: Downloading m3u8 information
05/01/2024 15:42:23 - [brightcove:new] 6343657635112: Downloading m3u8 information
05/01/2024 15:42:24 - [brightcove:new] 6343657635112: Downloading m3u8 information
05/01/2024 15:42:24 - [brightcove:new] 6343657635112: Downloading m3u8 information
05/01/2024 15:42:25 - [brightcove:new] 6343657635112: Downloading MPD manifest
05/01/2024 15:42:30 - [brightcove:new] 6343657635112: Downloading MPD manifest
05/01/2024 15:42:35 - [brightcove:new] 6343657635112: Downloading MPD manifest
05/01/2024 15:42:39 - [brightcove:new] 6343657635112: Downloading MPD manifest
05/01/2024 15:42:57 - [hlsnative] Downloading m3u8 manifest
05/01/2024 15:43:00 - Caught error: ERROR: unable to download video data: <urlopen error [Errno 1] _ssl.c:499: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version>
05/01/2024 15:43:00 - -code 1 -level 0 -errorinfo {{} /mod/webif/plugin/qtube/queue.hook 29} -errorcode {CHILDSTATUS 18908 1}

So, should I be adding --hls-prefer-ffmpeg in the options line, or hasn't it got far enough for that to have an effect?

Black Hole · Jan 5, 2024

Black Hole said:
So, should I be adding --hls-prefer-ffmpeg in the options line, or hasn't it got far enough for that to have an effect?

Well, that definitely changed something, but it seems to be stuck:

Code:

05/01/2024 16:20:04 - Starting queued download URL https://player.stv.tv/episode/4jd1/bates-vs-post-office Options --hls-prefer-ffmpeg QID 13565
05/01/2024 16:22:49 - [stv:player] 4jd1: Downloading webpage
05/01/2024 16:22:51 - [brightcove:new] 6343657635112: Downloading JSON metadata
05/01/2024 16:22:52 - [brightcove:new] 6343657635112: Downloading m3u8 information
05/01/2024 16:22:53 - [brightcove:new] 6343657635112: Downloading m3u8 information
05/01/2024 16:22:53 - [brightcove:new] 6343657635112: Downloading m3u8 information
05/01/2024 16:22:54 - [brightcove:new] 6343657635112: Downloading m3u8 information
05/01/2024 16:22:55 - [brightcove:new] 6343657635112: Downloading MPD manifest
05/01/2024 16:23:00 - [brightcove:new] 6343657635112: Downloading MPD manifest
05/01/2024 16:23:05 - [brightcove:new] 6343657635112: Downloading MPD manifest
05/01/2024 16:23:09 - [brightcove:new] 6343657635112: Downloading MPD manifest
05/01/2024 16:23:25 - [download] Destination: /mnt/hd2/My Video/[MP3]/Mr_Bates_vs_The_Post_Office_-_Series_1_-_Episode_4_of_4.mp4

prpr · Jan 5, 2024

I've packaged these mods. - available now in the Beta repository.
My first attempt hung the box at about the same point as yours, so I had to power cycle it.
The second appears to be working and is currently in progress (another 25 mins. or so).

(I have the .TS recordings off-air if all else fails!)

/df said:
I'm looking forward to Season/Series 2, where PO managers stand trial for conspiracy to pervert the course of justice (please) along with any PO lawyers not deemed too incompetent to have done so, compensation funds are reimbursed by Fujitsu, Bates is offered (but probably refuses) a CH or GBE, and the presumed validity of computer-generated evidence is overturned, restoring the position at PACE(1984).

And that evil witch is hounded relentlessly, forced to re-pay her millions, stripped of her CBE, and sent to prison for the rest of time. See how she likes it.

prpr · Jan 5, 2024

/df said:
I've actually applied the hack to extractor/common.py to find out if any extractors don't like the hack, but a more conservative approach would be to repeat the hack applied to extractor/itv.py. Because of dependencies, the hack also has to be applied to extractor/adobepass.py.

I couldn't get it to work with only itv.py and adobepass.py patched (the same way). At the moment, common.py and the other two are all patched.

Black Hole · Jan 5, 2024

Black Hole said:
it seems to be stuck

Well, it actually completed. I was fooled into thinking there was no progress because I normally see a running tally – has that been turned off?

It's downloaded half a gigabyte in 40 minutes, which isn't bad for me (2Mbps)!

(And then 3 minutes "fixing" the bitstream...)

Black Hole · Jan 6, 2024

Pleasant to discover the STV download has no adverts!

Black Hole · Jan 6, 2024

prpr said:
I've packaged these mods. - available now in the Beta repository.

Does that include --hls-prefer-ffmpeg?

I only see 2023.12.07

prpr · Jan 6, 2024

Black Hole said:
Does that include --hls-prefer-ffmpeg?

Yes.

I only see 2023.12.07

What were you expecting? There haven't been any commits to the repository since.

[youtube-dl] Download files from youtube.com or other video platforms

Number 28

May contain traces of nut

New Member

May contain traces of nut

Well-Known Member

May contain traces of nut

Well-Known Member

Well-Known Member

May contain traces of nut

Number 28

Well-Known Member

May contain traces of nut

May contain traces of nut

May contain traces of nut

Well-Known Member

Well-Known Member

May contain traces of nut

May contain traces of nut

May contain traces of nut

Well-Known Member