[youtube-dl] Download files from youtube.com or other video platforms

Apparently there is a football final featuring two minor London teams this evening, whose free-to-air viewing is via YouTube. The following procedures may allow you to watch the show live using your Humax HD/HDR, once you've found the URL ($url below) from YouTube, since YouTube no longer understands its Humax Portal app.
...
For tonight's final, this command is working for me (HD-Fox):
Code:
youtube-dl -f 96 'https://m.youtube.com/watch?v=mnO3QQY5Jwo' -o - | ffmpeg -hide_banner -i - -vcodec copy -acodec copy -mpegts_m2ts_mode true "/media/drive1/Video/Oporto.ts"
 
For tonight's final, this command is working for me (HD-Fox):
Code:
youtube-dl -f 96 'https://m.youtube.com/watch?v=mnO3QQY5Jwo' -o - | ffmpeg -hide_banner -i - -vcodec copy -acodec copy -mpegts_m2ts_mode true "/media/drive1/Video/Oporto.ts"
I was curious as to what this did and tried it myself on a HDR - it works great!
Just some info - you can substitute 96 with 95 to 91 for decreasing quality to save on bandwidth at the cost of quality.
As the output goes to "/media/drive1/Video/Oporto.ts", I had to navigate to it via storage/USB/VirtualUSB/Oporto.ts
Code:
youtube-dl -v -F  'https://m.youtube.com/watch?v=mnO3QQY5Jwo'
[info] Available formats for mnO3QQY5Jwo:
format code  extension  resolution note
139          m4a        audio only DASH audio   64k , m4a_dash container, mp4a.40.5 (22050Hz)
140          m4a        audio only DASH audio  144k , m4a_dash container, mp4a.40.2 (44100Hz)
278          webm       256x144    DASH video  111k , webm_dash container, vp9, 30fps, video only
242          webm       426x240    DASH video  166k , webm_dash container, vp9, 30fps, video only
160          mp4        256x144    DASH video  212k , mp4_dash container, avc1.42c00b, 15fps, video only
243          webm       640x360    DASH video  292k , webm_dash container, vp9, 30fps, video only
133          mp4        426x240    DASH video  456k , mp4_dash container, avc1.4d4015, 30fps, video only
244          webm       854x480    DASH video  528k , webm_dash container, vp9, 30fps, video only
134          mp4        640x360    DASH video 1008k , mp4_dash container, avc1.4d401e, 30fps, video only
247          webm       1280x720   DASH video 1040k , webm_dash container, vp9, 30fps, video only
135          mp4        854x480    DASH video 1350k , mp4_dash container, avc1.4d401f, 30fps, video only
248          webm       1920x1080  DASH video 1816k , webm_dash container, vp9, 30fps, video only
136          mp4        1280x720   DASH video 2684k , mp4_dash container, avc1.4d401f, 30fps, video only
137          mp4        1920x1080  DASH video 5018k , mp4_dash container, avc1.640028, 30fps, video only
91           mp4        256x144     290k , avc1.42c00b, 15.0fps, mp4a.40.5
92           mp4        426x240     546k , avc1.4d4015, 30.0fps, mp4a.40.5
93           mp4        640x360    1209k , avc1.4d401e, 30.0fps, mp4a.40.2
94           mp4        854x480    1568k , avc1.4d401f, 30.0fps, mp4a.40.2
95           mp4        1280x720   2969k , avc1.4d401f, 30.0fps, mp4a.40.2
96           mp4        1920x1080  5420k , avc1.640028, 30.0fps, mp4a.40.2 (best)

An alternative short clip for non footie fans to try
Code:
youtube-dl -f 22 'https://youtu.be/ZCzTse-8p6o' -o - | ffmpeg -hide_banner -i - -vcodec copy -acodec copy -mpegts_m2ts_mode true "/m
edia/drive1/Obuses1.ts"
This is a great technique as it allows you to watch the stream while it is downloading. Thanks @/df !

edit:
For the HDR a suitable output could be "/media/My Video/Ofilename1.ts" - so that it'll appear in the Media - Video lists.
 
Last edited:
As the output goes to "/media/drive1/Video/Oporto.ts"
/df was using a HD-FOX. For HDR-FOX you should substitute "/media/My\ Video/Oporto.ts" (or some such). I presume it only worked for you because you happened to have a USB drive connected!
 
/df was using a HD-FOX. For HDR-FOX you should substitute "/media/My\ Video/Oporto.ts" (or some such). I presume it only worked for you because you happened to have a USB drive connected!
Ah yes I have virtual-disk2 installed - so no physical USB mass storage device attached.
I tried using "/mnt/hd2/My\ Video/Otest.ts" but that failed so I stuck to the original output path.
 
Something I noticed on my HDR, even though I had kept the package up to date - or so I thought, but
Code:
youtube-dl  -v
[debug] System config: [u'--restrict-filenames', u'--prefer-ffmpeg', u'-f', u'best[height<=?576][fps<=?60]', u'-o', u'/mnt/hd2/My Video/40-Imported_Video/action_team/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2020.09.20
[debug] Python version 2.7.1 (CPython) - Linux-2.6.18-7.1-7405b0-smp-with-libc0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
Usage: youtube-dl [OPTIONS] URL [URL...]
When I navigate to the packages webpage it seems to think it is up to date.
So to resolve the issue I have to remove the 2 packages qtube youtube-dl, then install them again.
Note if you've made any changes /mod/etc/youtube-dl.conf you should back it up before the removal process.[/icode]
 

Attachments

  • Screenshot from 2021-05-30 10-02-49.png
    Screenshot from 2021-05-30 10-02-49.png
    9.8 KB · Views: 9
Last edited:
...Just some info - you can substitute 96 with 95 to 91 for decreasing quality to save on bandwidth at the cost of quality.
...
This is a great technique as it allows you to watch the stream while it is downloading. Thanks @/df !
Pleasure!

I selected 96 on the grounds that (a) we are dealing with a box that's meant to show HD (b) there was more chance of seeing where the moving ball was at a given time rather than a smear. But I expect 720 (95) would have been quite adequate. The format codes had the same interpretations in 2019, but I suppose they could change at YT's whim.

A paranoid fan would have run the command from abduco in case the telnet session failed.

The fans were hooting up and down the streets of W London last night.
 
Although it's not yet a problem with iPlayer, some media sites are being more aggressive about the TLS version that they will accept.

If this is just for the final download, the --external-downloader and --hls-prefer-mpeg options enable yt-dl to punt the download to a program (ffmpeg, wget) that's been built with a new-ish OpenSSL and so manages to negotiate a secure connection.

If a site demands a modern TLS version just for yt-dl to download the video page and/or its metadata, the extraction will fail.

A new build of Python 2.7, or at least its lib/ssl, is needed to solve this issue. Or a massive hack to insert wget into yt-dl's web access routines.
 
Although it's not yet a problem with iPlayer, some media sites are being more aggressive about the TLS version that they will accept.
...
A new build of Python 2.7, or at least its lib/ssl, is needed to solve this issue. Or a massive hack to insert wget into yt-dl's web access routines.
Such a massive hack is possible in a basic way although supporting all the functionality of the yt-dl routines may be trickier/too tricky. Overriding the _request_webpage() method of InfoExtractor intercepts the web access performed during extraction (though not when downloading the media file(s) themselves).
 
In another thread, it was observed that the iPlayer app provided different resolutions from qtube downloads. This occurs because the BBC extractor for yt-dl uses a different Mediator URL for listing the available media; the URL used by the iPlayer app requires a client certificate recognised by the BBC.

I wrote:
...
yt-dl doesn't (yet) know how to offer a client certificate although the SSL/TLS library that it uses does.
It's actually a straightforward change that should be offered in a PR, for what good that does. Here is a modified BBC extractor using the same Mediator service and Media Set as the iPlayer app, and passing the certificate using a new --client-certificate option:
Code:
# youtube-dl -v -F --ignore-config --client-certificate /mod/tmp/hdrfoxt2_20201202.pem  'https://www.bbc.co.uk/iplayer/episode/m000j4wd/darkest-hour'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'--ignore-config', u'--client-certificate', u'/mod/tmp/hdrfoxt2_20201202.pem', u'https://www.bbc.co.uk/iplayer/episode/m000j4wd/darkest-hour']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2021.06.06.1
[debug] Python version 2.7.1 (CPython) - Linux-2.6.18-7.1-7405b0-smp-with-libc0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[bbc.co.uk] m000j4wd: Downloading video page
[bbc.co.uk] Downloading media selection JSON
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[info] Available formats for m000j4wb:
format code           extension  resolution note
mf_akamai-0           mp4        512x288     576k , h264
mf_akamai-1           mp4        512x288     576k , h264
mf_cloudfront-0       mp4        512x288     576k , h264
mf_cloudfront-1       mp4        512x288     576k , h264
mf_akamai-2           mp4        960x540    1732k , h264
mf_akamai-3           mp4        960x540    1732k , h264
mf_cloudfront-2       mp4        960x540    1732k , h264
mf_cloudfront-3       mp4        960x540    1732k , h264
mf_akamai-1800-0      mp4        704x396    1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_akamai-1800-1      mp4        704x396    1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_bidi-1800-0        mp4        704x396    1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_bidi-1800-1        mp4        704x396    1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_cloudfront-1800-0  mp4        704x396    1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_cloudfront-1800-1  mp4        704x396    1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k (best)
#
 
This occurs because the BBC extractor for yt-dl uses a different Mediator URL for listing the available media; the URL used by the iPlayer app requires a client certificate recognised by the BBC.
could this be used to persuade the BBC servers that youtube-dl is a 'smart TV' that has been approved by the BBC to stream (or download) 4K content like Blue Planet?
 
Well, apparently the HDR can haz UHD, Media Set iptv-uhd:
Code:
# youtube-dl -F --ignore-config --client-certificate /mod/tmp/hdrfoxt2_20201202.pem 'https://www.bbc.co.uk/iplayer/episode/p04thmv7/blue-planet-ii-series-1-1-one-ocean'
[bbc.co.uk] p04thmv7: Downloading video page
[bbc.co.uk] p04thmv7: Downloading playlist JSON
[bbc.co.uk] Downloading media selection JSON
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading media selection JSON
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[bbc.co.uk] Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading media selection JSON
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[info] Available formats for p06mqywx:
format code                                                                       extension  resolution note
mf_akamai-audio_eng_1=96000-0                                                     m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_akamai-audio_eng_1=96000-1                                                     m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_bidi-audio_eng_1=96000-0                                                       m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_bidi-audio_eng_1=96000-1                                                       m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_cloudfront-audio_eng_1=96000-0                                                 m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_cloudfront-audio_eng_1=96000-1                                                 m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_akamai_uhd-7cf791a8-8f71-5069-fc09-a2d45515a71f_audio=192000_dur=3840-0        m4a        audio only [eng] DASH audio  190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai_uhd-7cf791a8-8f71-5069-fc09-a2d45515a71f_audio=192000_dur=3840-1        m4a        audio only [eng] DASH audio  190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_cloudfront_uhd-7cf791a8-8f71-5069-fc09-a2d45515a71f_audio=192000_dur=3840-0    m4a        audio only [eng] DASH audio  190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_cloudfront_uhd-7cf791a8-8f71-5069-fc09-a2d45515a71f_audio=192000_dur=3840-1    m4a        audio only [eng] DASH audio  190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai-video=281000-0                                                          mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_akamai-video=281000-1                                                          mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_bidi-video=281000-0                                                            mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_bidi-video=281000-1                                                            mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_cloudfront-video=281000-0                                                      mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_cloudfront-video=281000-1                                                      mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015, 25fps, video only
...
mf_akamai-video=5070000-0                                                         mp4        1280x720   DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_akamai-video=5070000-1                                                         mp4        1280x720   DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_bidi-video=5070000-0                                                           mp4        1280x720   DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_bidi-video=5070000-1                                                           mp4        1280x720   DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_cloudfront-video=5070000-0                                                     mp4        1280x720   DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_cloudfront-video=5070000-1                                                     mp4        1280x720   DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
...
mf_cloudfront_uhd-eb579e03-9dd0-e25f-bf1b-ec77bca4f66e_video=6300000_dur=3840-1   mp4        1920x1080  DASH video 8054k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-7c31b37e-2bec-7988-b87c-2409a6bf48e9_video=8080000_dur=3840-0       mp4        2560x1440  DASH video 10214k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-7c31b37e-2bec-7988-b87c-2409a6bf48e9_video=8080000_dur=3840-1       mp4        2560x1440  DASH video 10214k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-7c31b37e-2bec-7988-b87c-2409a6bf48e9_video=8080000_dur=3840-0   mp4        2560x1440  DASH video 10214k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-7c31b37e-2bec-7988-b87c-2409a6bf48e9_video=8080000_dur=3840-1   mp4        2560x1440  DASH video 10214k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-42a7dea3-7e7f-df7e-3d32-2b79d652ea30_video=12640000_dur=3840-0      mp4        3200x1800  DASH video 16086k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-42a7dea3-7e7f-df7e-3d32-2b79d652ea30_video=12640000_dur=3840-1      mp4        3200x1800  DASH video 16086k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-42a7dea3-7e7f-df7e-3d32-2b79d652ea30_video=12640000_dur=3840-0  mp4        3200x1800  DASH video 16086k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-42a7dea3-7e7f-df7e-3d32-2b79d652ea30_video=12640000_dur=3840-1  mp4        3200x1800  DASH video 16086k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-9dc5851e-d99b-10de-9315-0c5464e635c7_video=18160000_dur=3840-0      mp4        3840x2160  DASH video 21715k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-9dc5851e-d99b-10de-9315-0c5464e635c7_video=18160000_dur=3840-1      mp4        3840x2160  DASH video 21715k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-9dc5851e-d99b-10de-9315-0c5464e635c7_video=18160000_dur=3840-0  mp4        3840x2160  DASH video 21715k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-9dc5851e-d99b-10de-9315-0c5464e635c7_video=18160000_dur=3840-1  mp4        3840x2160  DASH video 21715k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
...
mf_akamai-2                                                                       mp4        960x540    1732k , h264
mf_akamai-3                                                                       mp4        960x540    1732k , h264
mf_cloudfront-2                                                                   mp4        960x540    1732k , h264
mf_cloudfront-3                                                                   mp4        960x540    1732k , h264
mf_akamai-1802-0                                                                  mp4        960x540    1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_akamai-1802-1                                                                  mp4        960x540    1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_bidi-1802-0                                                                    mp4        960x540    1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_bidi-1802-1                                                                    mp4        960x540    1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_cloudfront-1802-0                                                              mp4        960x540    1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_cloudfront-1802-1                                                              mp4        960x540    1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_akamai-4                                                                       mp4        960x540    3000k , h264
mf_akamai-5                                                                       mp4        960x540    3000k , h264
mf_cloudfront-4                                                                   mp4        960x540    3000k , h264
mf_cloudfront-5                                                                   mp4        960x540    3000k , h264
mf_akamai-3117-0                                                                  mp4        960x540    3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_akamai-3117-1                                                                  mp4        960x540    3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_bidi-3117-0                                                                    mp4        960x540    3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_bidi-3117-1                                                                    mp4        960x540    3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_cloudfront-3117-0                                                              mp4        960x540    3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_cloudfront-3117-1                                                              mp4        960x540    3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k (best)
#
We also get 1280x720 50fps playable in the HD/HDR as well, instead of 960x540, supposing that these can actually be downloaded.

Also
...It's actually a straightforward change that should be offered in a PR, for what good that does. ...
see here.
 
Last edited:
... Overriding the _request_webpage() method of InfoExtractor intercepts the web access performed during extraction (though not when downloading the media file(s) themselves).
Create the Python source file youtube_dl/extractor/fakehttp.py; then in an extractor that fails because of SSL/TLS version issues, insert this line before the extractor class definition:
Code:
from .fakehttp import FakeHTTP as InfoExtractor

fakehttp.py (version 2021-10-06)
Code:
# coding: utf-8
from __future__ import unicode_literals

from .common import InfoExtractor
from ..compat import (
    compat_str,
    compat_http_client,
    compat_HTTPError,
    compat_urllib_request,
    compat_urllib_error,
)
from ..utils import (
    base_url,
    error_to_compat_str,
    ExtractorError,
    sanitized_Request,
    std_headers,
    update_Request,
    update_url_query,
    urljoin,
    YoutubeDLRedirectHandler,
)

from StringIO import StringIO
import email

def encode_hdr(string, encoding='iso8859-1', errors='strict'):
    return string if isinstance(string, bytes) else string.encode(encoding, errors)

class UrlHandle(StringIO):
    _code = 0
    _url = ''
    _info = {}

    from types import MethodType

    def __init__(self, text, code, url, info):
        StringIO.__init__(self, text)
        self._info = email.message_from_string(encode_hdr(info, errors='backslashreplace'))
        # interface required by Py2 CookieJar.extract_cookies(); Py3 just uses info()
        self._info.getheaders = self.MethodType(lambda s, n: s.get_all(n) or [], self._info)
        self._code = code
        self._url = url

    def _read_only(self):
        forbidden = PermissionError()
        raise compat_urllib_error.URLError(forbidden)

    def truncate(self, size=float('inf')):
        self._read_only()

    def write(self, str):
        self._read_only()

    def writelines(self, sequence):
        self._read_only()

    @property
    def mode(self):
        return 'r'

    @property
    def headers(self):
        return self._info

    def geturl(self):
        return self._url

    def info(self):
        return self._info

    def getcode(self):
        return self._code


import subprocess
import re
import sys
import zlib
from itertools import takewhile

class FakeHTTP(InfoExtractor):

    def _request_webpage(self, url_or_request, video_id, note=None, errnote=None, fatal=True, data=None, headers={}, query={}, expected_status=None):
        if isinstance(url_or_request, compat_urllib_request.Request):
            url_or_request = update_Request(
                url_or_request, data=data, headers=headers, query=query)
        else:
            if query:
                url_or_request = update_url_query(url_or_request, query)
            url_or_request = sanitized_Request(url_or_request, data, headers)

        # type may not be an attribute before Py3
        def req_type(req):
           return req.type if not callable(getattr(req, 'get_type')) else (req.type or req.get_type())

        if req_type(url_or_request) not in ('http', 'https', ):
            return super(FakeHTTP, self)._request_webpage(url_or_request, video_id, note=None, errnote=errnote, fatal=fatal, expected_status=expected_status)

        if note is None:
            self.report_download_webpage(video_id)
        elif note is not False:
            if video_id is None:
                self.to_screen('%s' % (note,))
            else:
                self.to_screen('%s: %s' % (video_id, note))

        # Some sites check X-Forwarded-For HTTP header in order to figure out
        # the origin of the client behind proxy. This allows bypassing geo
        # restriction by faking this header's value to IP that belongs to some
        # geo unrestricted country. We will do so once we encounter any
        # geo restriction error.
        if self._x_forwarded_for_ip and not url_or_request.has_header('X-Forwarded-For'):
            url_or_request.add_header('X-Forwarded-For', self._x_forwarded_for_ip)

        for h, v in std_headers.items():
             #if h.capitalize() not in url_or_request.headers and h != 'Accept-Encoding':
             if not url_or_request.has_header(h.capitalize()) and h != 'Accept-Encoding':
                 url_or_request.add_header(h, v)

        exceptions = [compat_urllib_error.URLError, compat_http_client.HTTPException] #, socket.error]

        try:
            cmd = ['wget', '-4', '-U ""', '-q', '-S', '-O-']
            method = url_or_request.get_method()
            if method != 'GET':
                cmd.append('--method=%s' % method)
            self._downloader.cookiejar.add_cookie_header(url_or_request)
            for h_k, h_v in url_or_request.header_items():
                cmd.append('--header=%s: %s' % (h_k, h_v))
            #if encoding:
            #    cmd.append('--remote-encoding=%s ' % encoding)
            cmd.append('--connect-timeout=%s ' % self._downloader._socket_timeout)
            if data:
                post1 = (method in ('GET', 'POST'))
                cmd.append('--%s-data=%s' %
                           ('post' if method == 'GET' else 'body', data))
            else:
                post1 = (method == 'POST')
            # if POSTing, manually handle redirect to GET
            if post1:
                cmd.append('--max-redirect=0')
            url = url_or_request.get_full_url()
            cmd.append(url)
            debug_printtraffic = self._downloader.params.get('debug_printtraffic', False)
            if debug_printtraffic:
                self._downloader._write_string(' '.join(cmd))
            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout, stderr = p.communicate()
            stderr = stderr.decode('utf-8', 'replace')
            # print('-> %d' % p.returncode)
            msg = stderr.splitlines() or ['']
            code = 0
            reason = ''
            #charset = 'ascii'
            encoding = None
            header_i = -1
            in_headers = False
            for i in range(len(msg)):
                # print '%d: %s' % (i, msg[i])
                m = re.match(
                        r'''(?x)
                            \s*Location:\s+(?P<url>[\S]+?)($|\s+\[following\])|
                            \s+HTTP/[\d.]+\s+(?P<code>[1-5]\d\d)(?:\s+(?P<reason>.*))?$|
                            \s+Content-Type:\s+(?P<charset>.+)$|
                            \s+Content-Encoding:\s+(?P<encoding>\S+)|
                            (?P<other>\S.*)$
                        ''', msg[i])
                if m:
                    #print m.groups()
                    if m.group('url'):
                        new_url = urljoin(url, m.group('url'))
                        if new_url:
                            url = new_url
                    elif m.group('other'):
                        in_headers = False
                    else:
                        if m.group('code'):
                            code = int(m.group('code'))
                            reason = m.group('reason') or ''
                            if not in_headers:
                                header_i = i+1
                                in_headers = True
                        elif m.group('encoding'):
                            encoding = m.group('encoding').lower()
                        if debug_printtraffic:
                            self._downloader._write_string(msg[i].lstrip())
                else:
                    if debug_printtraffic:
                        self._downloader._write_string(msg[i].lstrip())
            #print 'ENCODING: %s' % encoding
            content = stdout
            if p.returncode != 0:
                if p.returncode == 8:
                    if not (code >= 300 and code < 310):
                        raise compat_HTTPError(url, code, reason, None, None)
                elif p.returncode == 6:
                    raise compat_HTTPError(url, 401, msg[-1], None, None)
                else:
                    raise compat_http_client.BadStatusLine(msg[-1])
            #print ' '.join( '%02x' % ord(x) for x in content[:12])
            if encoding in ('gzip', 'deflate'):
                # 47 = 32+15 = format:zlib,gzip + windowsize:32768
                content = zlib.decompress(content, 47)
            elif encoding:
                self.report_warning('Unsupported Content-Encoding: %s' % encoding)
            rsp = UrlHandle(
                      content, code, url,
                      '\n'.join(
                          [h.strip() for h in takewhile(
                              lambda x: x[0].isspace(), msg[header_i:])]))
            self._downloader.cookiejar.extract_cookies(rsp, url_or_request)
            if post1 and code >= 300 and code < 310:
                redir = YoutubeDLRedirectHandler()
                r_req = redir.redirect_request(url_or_request, None, code, reason, rsp.headers, url)
                if r_req:
                    return self._request_webpage(r_req, video_id, note=note, errnote=errnote, fatal=fatal, data=None, headers={}, query={}, expected_status=expected_status)
            if False and debug_printtraffic:
                if not encoding:
                    encoding = self._guess_encoding_from_content(rsp.headers.get('Content-Type', ''), content)
                try:
                    content = content.decode(encoding, 'replace')
                except LookupError:
                    content = content.decode('utf-8', 'replace')
                self._downloader._write_string(content)
            return rsp

        except tuple(exceptions) as err:
            if isinstance(err, compat_HTTPError):
                if self._InfoExtractor__can_accept_status_code(err, expected_status):
                    # Retain reference to error to prevent file object from
                    # being closed before it can be read. Works around the
                    # effects of <https://bugs.python.org/issue15002>
                    # introduced in Python 3.4.1.
                    err.fp._error = err
                    return err.fp

            if errnote is False:
                return False
            if errnote is None:
                errnote = 'Unable to download webpage'

            errmsg = '%s: %s' % (errnote, error_to_compat_str(err))
            if fatal:
                raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
            else:
                self._downloader.report_warning(errmsg)
                return False
 
Last edited:
Some standard openssl command applied to the certificate in iplfix. Even more than ffmpeg I find this an area of dark magic. Maybe
Code:
openssl pkcs12 -in hdrfoxt2_20201202.p12 -out hdrfoxt2_20201202.pem -nodes
Here interpret -nodes as -no-DES, nothing do with nodes.
 
Here is a modified BBC extractor
Where? And what's changed?
I've so far patched the 3 .py files from your PR and generated the .pem file. Neither of your suggested test streams above generates anything different in the format list for me when using the --client-certificate option. So I'm obviously missing something.
 
Where? In my proof of concept version. Not that I was being deliberately obtuse but the BBC extractor needs some modifications to access a different Mediator and it's not obvious how to implement those without breaking normal operation. In fact even the normal operation is controversial as to whether multiple Mediator requests are acceptable to produce a wider range of formats.

For a proof of concept, this should do:
Code:
--- /home/df/Documents/src/youtube-dl/youtube_dl/extractor/bbc.py
+++ /home/df/Documents/src/youtube-dl/youtube_dl/extractor/bbc.py.hak
@@ -57,13 +57,10 @@
     _LOGIN_URL = 'https://account.bbc.com/signin'
     _NETRC_MACHINE = 'bbc'
 
-    _MEDIA_SELECTOR_URL_TEMPL = 'https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/%s/vpid/%s'
+    _MEDIA_SELECTOR_URL_TEMPL = 'https://securegate.iplayer.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/%s/vpid/%s'
     _MEDIA_SETS = [
-        # Provides HQ HLS streams with even better quality that pc mediaset but fails
-        # with geolocation in some cases when it's even not geo restricted at all (e.g.
-        # http://www.bbc.co.uk/programmes/b06bp7lf). Also may fail with selectionunavailable.
-        'iptv-all',
-        'pc',
+        'legacy-iptv-all',
+        'iptv-uhd',
     ]
 
     _EMP_PLAYLIST_NS = 'http://bbc.co.uk/2008/emp/playlist'
However this won't produce anything without the certificate.

To be integrated properly, the extraction should fall back to the normal Mediator if no certificate is given or extraction using the certificate fails.
 
Back
Top