[youtube-dl] Download files from youtube.com or other video platforms
- Thread starter prpr
- Start date
bottletop
Active Member
I was curious as to what this did and tried it myself on a HDR - it works great!
Just some info - you can substitute 96 with 95 to 91 for decreasing quality to save on bandwidth at the cost of quality.
As the output goes to "/media/drive1/Video/Oporto.ts", I had to navigate to it via storage/USB/VirtualUSB/Oporto.ts
Code:
youtube-dl -v -F 'https://m.youtube.com/watch?v=mnO3QQY5Jwo'
[info] Available formats for mnO3QQY5Jwo:
format code extension resolution note
139 m4a audio only DASH audio 64k , m4a_dash container, mp4a.40.5 (22050Hz)
140 m4a audio only DASH audio 144k , m4a_dash container, mp4a.40.2 (44100Hz)
278 webm 256x144 DASH video 111k , webm_dash container, vp9, 30fps, video only
242 webm 426x240 DASH video 166k , webm_dash container, vp9, 30fps, video only
160 mp4 256x144 DASH video 212k , mp4_dash container, avc1.42c00b, 15fps, video only
243 webm 640x360 DASH video 292k , webm_dash container, vp9, 30fps, video only
133 mp4 426x240 DASH video 456k , mp4_dash container, avc1.4d4015, 30fps, video only
244 webm 854x480 DASH video 528k , webm_dash container, vp9, 30fps, video only
134 mp4 640x360 DASH video 1008k , mp4_dash container, avc1.4d401e, 30fps, video only
247 webm 1280x720 DASH video 1040k , webm_dash container, vp9, 30fps, video only
135 mp4 854x480 DASH video 1350k , mp4_dash container, avc1.4d401f, 30fps, video only
248 webm 1920x1080 DASH video 1816k , webm_dash container, vp9, 30fps, video only
136 mp4 1280x720 DASH video 2684k , mp4_dash container, avc1.4d401f, 30fps, video only
137 mp4 1920x1080 DASH video 5018k , mp4_dash container, avc1.640028, 30fps, video only
91 mp4 256x144 290k , avc1.42c00b, 15.0fps, mp4a.40.5
92 mp4 426x240 546k , avc1.4d4015, 30.0fps, mp4a.40.5
93 mp4 640x360 1209k , avc1.4d401e, 30.0fps, mp4a.40.2
94 mp4 854x480 1568k , avc1.4d401f, 30.0fps, mp4a.40.2
95 mp4 1280x720 2969k , avc1.4d401f, 30.0fps, mp4a.40.2
96 mp4 1920x1080 5420k , avc1.640028, 30.0fps, mp4a.40.2 (best)An alternative short clip for non footie fans to try
Code:
youtube-dl -f 22 'https://youtu.be/ZCzTse-8p6o' -o - | ffmpeg -hide_banner -i - -vcodec copy -acodec copy -mpegts_m2ts_mode true "/m
edia/drive1/Obuses1.ts"edit:
For the HDR a suitable output could be "/media/My Video/Ofilename1.ts" - so that it'll appear in the Media - Video lists.
Last edited:
Black Hole
May contain traces of nut
/df was using a HD-FOX. For HDR-FOX you should substitute "/media/My\ Video/Oporto.ts" (or some such). I presume it only worked for you because you happened to have a USB drive connected!
bottletop
Active Member
Ah yes I have virtual-disk2 installed - so no physical USB mass storage device attached.
I tried using "/mnt/hd2/My\ Video/Otest.ts" but that failed so I stuck to the original output path.
bottletop
Active Member
Something I noticed on my HDR, even though I had kept the package up to date - or so I thought, but
When I navigate to the packages webpage it seems to think it is up to date.
So to resolve the issue I have to remove the 2 packages
Note if you've made any changes /mod/etc/youtube-dl.conf you should back it up before the removal process.[/icode]
Code:
youtube-dl -v
[debug] System config: [u'--restrict-filenames', u'--prefer-ffmpeg', u'-f', u'best[height<=?576][fps<=?60]', u'-o', u'/mnt/hd2/My Video/40-Imported_Video/action_team/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2020.09.20
[debug] Python version 2.7.1 (CPython) - Linux-2.6.18-7.1-7405b0-smp-with-libc0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
Usage: youtube-dl [OPTIONS] URL [URL...]So to resolve the issue I have to remove the 2 packages
qtube youtube-dl, then install them again.Note if you've made any changes /mod/etc/youtube-dl.conf you should back it up before the removal process.[/icode]
Attachments
Last edited:
Pleasure!
I selected 96 on the grounds that (a) we are dealing with a box that's meant to show HD (b) there was more chance of seeing where the moving ball was at a given time rather than a smear. But I expect 720 (95) would have been quite adequate. The format codes had the same interpretations in 2019, but I suppose they could change at YT's whim.
A paranoid fan would have run the command from abduco in case the telnet session failed.
The fans were hooting up and down the streets of W London last night.
Black Hole
May contain traces of nut
It crosses my mind that as the destination string is enclosed in quotes in the command, it might be incorrect to escape the embedded space.
One or the other, not both.
The original post that I quoted did address how to vary the destination.
Although it's not yet a problem with iPlayer, some media sites are being more aggressive about the TLS version that they will accept.
If this is just for the final download, the
If a site demands a modern TLS version just for yt-dl to download the video page and/or its metadata, the extraction will fail.
A new build of Python 2.7, or at least its lib/ssl, is needed to solve this issue. Or a massive hack to insert
If this is just for the final download, the
--external-downloader and --hls-prefer-mpeg options enable yt-dl to punt the download to a program (ffmpeg, wget) that's been built with a new-ish OpenSSL and so manages to negotiate a secure connection.If a site demands a modern TLS version just for yt-dl to download the video page and/or its metadata, the extraction will fail.
A new build of Python 2.7, or at least its lib/ssl, is needed to solve this issue. Or a massive hack to insert
wget into yt-dl's web access routines.Such a massive hack is possible in a basic way although supporting all the functionality of the yt-dl routines may be trickier/too tricky. Overriding the
_request_webpage() method of InfoExtractor intercepts the web access performed during extraction (though not when downloading the media file(s) themselves).In another thread, it was observed that the iPlayer app provided different resolutions from qtube downloads. This occurs because the BBC extractor for yt-dl uses a different Mediator URL for listing the available media; the URL used by the iPlayer app requires a client certificate recognised by the BBC.
I wrote:
I wrote:
It's actually a straightforward change that should be offered in a PR, for what good that does. Here is a modified BBC extractor using the same Mediator service and Media Set as the iPlayer app, and passing the certificate using a new
--client-certificate option:
Code:
# youtube-dl -v -F --ignore-config --client-certificate /mod/tmp/hdrfoxt2_20201202.pem 'https://www.bbc.co.uk/iplayer/episode/m000j4wd/darkest-hour'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'--ignore-config', u'--client-certificate', u'/mod/tmp/hdrfoxt2_20201202.pem', u'https://www.bbc.co.uk/iplayer/episode/m000j4wd/darkest-hour']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2021.06.06.1
[debug] Python version 2.7.1 (CPython) - Linux-2.6.18-7.1-7405b0-smp-with-libc0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[bbc.co.uk] m000j4wd: Downloading video page
[bbc.co.uk] Downloading media selection JSON
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[info] Available formats for m000j4wb:
format code extension resolution note
mf_akamai-0 mp4 512x288 576k , h264
mf_akamai-1 mp4 512x288 576k , h264
mf_cloudfront-0 mp4 512x288 576k , h264
mf_cloudfront-1 mp4 512x288 576k , h264
mf_akamai-2 mp4 960x540 1732k , h264
mf_akamai-3 mp4 960x540 1732k , h264
mf_cloudfront-2 mp4 960x540 1732k , h264
mf_cloudfront-3 mp4 960x540 1732k , h264
mf_akamai-1800-0 mp4 704x396 1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_akamai-1800-1 mp4 704x396 1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_bidi-1800-0 mp4 704x396 1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_bidi-1800-1 mp4 704x396 1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_cloudfront-1800-0 mp4 704x396 1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k
mf_cloudfront-1800-1 mp4 704x396 1800k , avc1.64001F@1570k, 50.0fps, mp4a.40.2@128k (best)
#
Ezra Pound
Well-Known Member
Well, apparently the HDR can haz UHD, Media Set
We also get 1280x720 50fps playable in the HD/HDR as well, instead of 960x540, supposing that these can actually be downloaded.
Also
iptv-uhd:
Code:
# youtube-dl -F --ignore-config --client-certificate /mod/tmp/hdrfoxt2_20201202.pem 'https://www.bbc.co.uk/iplayer/episode/p04thmv7/blue-planet-ii-series-1-1-one-ocean'
[bbc.co.uk] p04thmv7: Downloading video page
[bbc.co.uk] p04thmv7: Downloading playlist JSON
[bbc.co.uk] Downloading media selection JSON
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading media selection JSON
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
[bbc.co.uk] Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[bbc.co.uk] Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading media selection JSON
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[bbc.co.uk] Downloading MPD manifest
[info] Available formats for p06mqywx:
format code extension resolution note
mf_akamai-audio_eng_1=96000-0 m4a audio only [en] DASH audio 96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_akamai-audio_eng_1=96000-1 m4a audio only [en] DASH audio 96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_bidi-audio_eng_1=96000-0 m4a audio only [en] DASH audio 96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_bidi-audio_eng_1=96000-1 m4a audio only [en] DASH audio 96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_cloudfront-audio_eng_1=96000-0 m4a audio only [en] DASH audio 96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_cloudfront-audio_eng_1=96000-1 m4a audio only [en] DASH audio 96k , m4a_dash container, mp4a.40.5 (48000Hz)
mf_akamai_uhd-7cf791a8-8f71-5069-fc09-a2d45515a71f_audio=192000_dur=3840-0 m4a audio only [eng] DASH audio 190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai_uhd-7cf791a8-8f71-5069-fc09-a2d45515a71f_audio=192000_dur=3840-1 m4a audio only [eng] DASH audio 190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_cloudfront_uhd-7cf791a8-8f71-5069-fc09-a2d45515a71f_audio=192000_dur=3840-0 m4a audio only [eng] DASH audio 190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_cloudfront_uhd-7cf791a8-8f71-5069-fc09-a2d45515a71f_audio=192000_dur=3840-1 m4a audio only [eng] DASH audio 190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai-video=281000-0 mp4 384x216 DASH video 281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_akamai-video=281000-1 mp4 384x216 DASH video 281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_bidi-video=281000-0 mp4 384x216 DASH video 281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_bidi-video=281000-1 mp4 384x216 DASH video 281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_cloudfront-video=281000-0 mp4 384x216 DASH video 281k , mp4_dash container, avc3.42C015, 25fps, video only
mf_cloudfront-video=281000-1 mp4 384x216 DASH video 281k , mp4_dash container, avc3.42C015, 25fps, video only
...
mf_akamai-video=5070000-0 mp4 1280x720 DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_akamai-video=5070000-1 mp4 1280x720 DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_bidi-video=5070000-0 mp4 1280x720 DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_bidi-video=5070000-1 mp4 1280x720 DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_cloudfront-video=5070000-0 mp4 1280x720 DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
mf_cloudfront-video=5070000-1 mp4 1280x720 DASH video 5070k , mp4_dash container, avc3.640020, 50fps, video only
...
mf_cloudfront_uhd-eb579e03-9dd0-e25f-bf1b-ec77bca4f66e_video=6300000_dur=3840-1 mp4 1920x1080 DASH video 8054k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-7c31b37e-2bec-7988-b87c-2409a6bf48e9_video=8080000_dur=3840-0 mp4 2560x1440 DASH video 10214k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-7c31b37e-2bec-7988-b87c-2409a6bf48e9_video=8080000_dur=3840-1 mp4 2560x1440 DASH video 10214k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-7c31b37e-2bec-7988-b87c-2409a6bf48e9_video=8080000_dur=3840-0 mp4 2560x1440 DASH video 10214k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-7c31b37e-2bec-7988-b87c-2409a6bf48e9_video=8080000_dur=3840-1 mp4 2560x1440 DASH video 10214k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-42a7dea3-7e7f-df7e-3d32-2b79d652ea30_video=12640000_dur=3840-0 mp4 3200x1800 DASH video 16086k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-42a7dea3-7e7f-df7e-3d32-2b79d652ea30_video=12640000_dur=3840-1 mp4 3200x1800 DASH video 16086k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-42a7dea3-7e7f-df7e-3d32-2b79d652ea30_video=12640000_dur=3840-0 mp4 3200x1800 DASH video 16086k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-42a7dea3-7e7f-df7e-3d32-2b79d652ea30_video=12640000_dur=3840-1 mp4 3200x1800 DASH video 16086k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-9dc5851e-d99b-10de-9315-0c5464e635c7_video=18160000_dur=3840-0 mp4 3840x2160 DASH video 21715k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-9dc5851e-d99b-10de-9315-0c5464e635c7_video=18160000_dur=3840-1 mp4 3840x2160 DASH video 21715k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-9dc5851e-d99b-10de-9315-0c5464e635c7_video=18160000_dur=3840-0 mp4 3840x2160 DASH video 21715k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-9dc5851e-d99b-10de-9315-0c5464e635c7_video=18160000_dur=3840-1 mp4 3840x2160 DASH video 21715k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
...
mf_akamai-2 mp4 960x540 1732k , h264
mf_akamai-3 mp4 960x540 1732k , h264
mf_cloudfront-2 mp4 960x540 1732k , h264
mf_cloudfront-3 mp4 960x540 1732k , h264
mf_akamai-1802-0 mp4 960x540 1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_akamai-1802-1 mp4 960x540 1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_bidi-1802-0 mp4 960x540 1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_bidi-1802-1 mp4 960x540 1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_cloudfront-1802-0 mp4 960x540 1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_cloudfront-1802-1 mp4 960x540 1802k , avc1.4D401F@1604k, 25.0fps, mp4a.40.5@ 96k
mf_akamai-4 mp4 960x540 3000k , h264
mf_akamai-5 mp4 960x540 3000k , h264
mf_cloudfront-4 mp4 960x540 3000k , h264
mf_cloudfront-5 mp4 960x540 3000k , h264
mf_akamai-3117-0 mp4 960x540 3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_akamai-3117-1 mp4 960x540 3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_bidi-3117-0 mp4 960x540 3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_bidi-3117-1 mp4 960x540 3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_cloudfront-3117-0 mp4 960x540 3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k
mf_cloudfront-3117-1 mp4 960x540 3117k , avc1.64001F@2812k, 50.0fps, mp4a.40.2@128k (best)
#Also
see here.
Last edited:
Create the Python source file
youtube_dl/extractor/fakehttp.py; then in an extractor that fails because of SSL/TLS version issues, insert this line before the extractor class definition:
Code:
from .fakehttp import FakeHTTP as InfoExtractorfakehttp.py (version 2021-10-06)
Code:
# coding: utf-8
from __future__ import unicode_literals
from .common import InfoExtractor
from ..compat import (
compat_str,
compat_http_client,
compat_HTTPError,
compat_urllib_request,
compat_urllib_error,
)
from ..utils import (
base_url,
error_to_compat_str,
ExtractorError,
sanitized_Request,
std_headers,
update_Request,
update_url_query,
urljoin,
YoutubeDLRedirectHandler,
)
from StringIO import StringIO
import email
def encode_hdr(string, encoding='iso8859-1', errors='strict'):
return string if isinstance(string, bytes) else string.encode(encoding, errors)
class UrlHandle(StringIO):
_code = 0
_url = ''
_info = {}
from types import MethodType
def __init__(self, text, code, url, info):
StringIO.__init__(self, text)
self._info = email.message_from_string(encode_hdr(info, errors='backslashreplace'))
# interface required by Py2 CookieJar.extract_cookies(); Py3 just uses info()
self._info.getheaders = self.MethodType(lambda s, n: s.get_all(n) or [], self._info)
self._code = code
self._url = url
def _read_only(self):
forbidden = PermissionError()
raise compat_urllib_error.URLError(forbidden)
def truncate(self, size=float('inf')):
self._read_only()
def write(self, str):
self._read_only()
def writelines(self, sequence):
self._read_only()
@property
def mode(self):
return 'r'
@property
def headers(self):
return self._info
def geturl(self):
return self._url
def info(self):
return self._info
def getcode(self):
return self._code
import subprocess
import re
import sys
import zlib
from itertools import takewhile
class FakeHTTP(InfoExtractor):
def _request_webpage(self, url_or_request, video_id, note=None, errnote=None, fatal=True, data=None, headers={}, query={}, expected_status=None):
if isinstance(url_or_request, compat_urllib_request.Request):
url_or_request = update_Request(
url_or_request, data=data, headers=headers, query=query)
else:
if query:
url_or_request = update_url_query(url_or_request, query)
url_or_request = sanitized_Request(url_or_request, data, headers)
# type may not be an attribute before Py3
def req_type(req):
return req.type if not callable(getattr(req, 'get_type')) else (req.type or req.get_type())
if req_type(url_or_request) not in ('http', 'https', ):
return super(FakeHTTP, self)._request_webpage(url_or_request, video_id, note=None, errnote=errnote, fatal=fatal, expected_status=expected_status)
if note is None:
self.report_download_webpage(video_id)
elif note is not False:
if video_id is None:
self.to_screen('%s' % (note,))
else:
self.to_screen('%s: %s' % (video_id, note))
# Some sites check X-Forwarded-For HTTP header in order to figure out
# the origin of the client behind proxy. This allows bypassing geo
# restriction by faking this header's value to IP that belongs to some
# geo unrestricted country. We will do so once we encounter any
# geo restriction error.
if self._x_forwarded_for_ip and not url_or_request.has_header('X-Forwarded-For'):
url_or_request.add_header('X-Forwarded-For', self._x_forwarded_for_ip)
for h, v in std_headers.items():
#if h.capitalize() not in url_or_request.headers and h != 'Accept-Encoding':
if not url_or_request.has_header(h.capitalize()) and h != 'Accept-Encoding':
url_or_request.add_header(h, v)
exceptions = [compat_urllib_error.URLError, compat_http_client.HTTPException] #, socket.error]
try:
cmd = ['wget', '-4', '-U ""', '-q', '-S', '-O-']
method = url_or_request.get_method()
if method != 'GET':
cmd.append('--method=%s' % method)
self._downloader.cookiejar.add_cookie_header(url_or_request)
for h_k, h_v in url_or_request.header_items():
cmd.append('--header=%s: %s' % (h_k, h_v))
#if encoding:
# cmd.append('--remote-encoding=%s ' % encoding)
cmd.append('--connect-timeout=%s ' % self._downloader._socket_timeout)
if data:
post1 = (method in ('GET', 'POST'))
cmd.append('--%s-data=%s' %
('post' if method == 'GET' else 'body', data))
else:
post1 = (method == 'POST')
# if POSTing, manually handle redirect to GET
if post1:
cmd.append('--max-redirect=0')
url = url_or_request.get_full_url()
cmd.append(url)
debug_printtraffic = self._downloader.params.get('debug_printtraffic', False)
if debug_printtraffic:
self._downloader._write_string(' '.join(cmd))
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
stderr = stderr.decode('utf-8', 'replace')
# print('-> %d' % p.returncode)
msg = stderr.splitlines() or ['']
code = 0
reason = ''
#charset = 'ascii'
encoding = None
header_i = -1
in_headers = False
for i in range(len(msg)):
# print '%d: %s' % (i, msg[i])
m = re.match(
r'''(?x)
\s*Location:\s+(?P<url>[\S]+?)($|\s+\[following\])|
\s+HTTP/[\d.]+\s+(?P<code>[1-5]\d\d)(?:\s+(?P<reason>.*))?$|
\s+Content-Type:\s+(?P<charset>.+)$|
\s+Content-Encoding:\s+(?P<encoding>\S+)|
(?P<other>\S.*)$
''', msg[i])
if m:
#print m.groups()
if m.group('url'):
new_url = urljoin(url, m.group('url'))
if new_url:
url = new_url
elif m.group('other'):
in_headers = False
else:
if m.group('code'):
code = int(m.group('code'))
reason = m.group('reason') or ''
if not in_headers:
header_i = i+1
in_headers = True
elif m.group('encoding'):
encoding = m.group('encoding').lower()
if debug_printtraffic:
self._downloader._write_string(msg[i].lstrip())
else:
if debug_printtraffic:
self._downloader._write_string(msg[i].lstrip())
#print 'ENCODING: %s' % encoding
content = stdout
if p.returncode != 0:
if p.returncode == 8:
if not (code >= 300 and code < 310):
raise compat_HTTPError(url, code, reason, None, None)
elif p.returncode == 6:
raise compat_HTTPError(url, 401, msg[-1], None, None)
else:
raise compat_http_client.BadStatusLine(msg[-1])
#print ' '.join( '%02x' % ord(x) for x in content[:12])
if encoding in ('gzip', 'deflate'):
# 47 = 32+15 = format:zlib,gzip + windowsize:32768
content = zlib.decompress(content, 47)
elif encoding:
self.report_warning('Unsupported Content-Encoding: %s' % encoding)
rsp = UrlHandle(
content, code, url,
'\n'.join(
[h.strip() for h in takewhile(
lambda x: x[0].isspace(), msg[header_i:])]))
self._downloader.cookiejar.extract_cookies(rsp, url_or_request)
if post1 and code >= 300 and code < 310:
redir = YoutubeDLRedirectHandler()
r_req = redir.redirect_request(url_or_request, None, code, reason, rsp.headers, url)
if r_req:
return self._request_webpage(r_req, video_id, note=note, errnote=errnote, fatal=fatal, data=None, headers={}, query={}, expected_status=expected_status)
if False and debug_printtraffic:
if not encoding:
encoding = self._guess_encoding_from_content(rsp.headers.get('Content-Type', ''), content)
try:
content = content.decode(encoding, 'replace')
except LookupError:
content = content.decode('utf-8', 'replace')
self._downloader._write_string(content)
return rsp
except tuple(exceptions) as err:
if isinstance(err, compat_HTTPError):
if self._InfoExtractor__can_accept_status_code(err, expected_status):
# Retain reference to error to prevent file object from
# being closed before it can be read. Works around the
# effects of <https://bugs.python.org/issue15002>
# introduced in Python 3.4.1.
err.fp._error = err
return err.fp
if errnote is False:
return False
if errnote is None:
errnote = 'Unable to download webpage'
errmsg = '%s: %s' % (errnote, error_to_compat_str(err))
if fatal:
raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
else:
self._downloader.report_warning(errmsg)
return False
Last edited:
OP
prpr
Well-Known Member
Where? In my proof of concept version. Not that I was being deliberately obtuse but the BBC extractor needs some modifications to access a different Mediator and it's not obvious how to implement those without breaking normal operation. In fact even the normal operation is controversial as to whether multiple Mediator requests are acceptable to produce a wider range of formats.
For a proof of concept, this should do:
However this won't produce anything without the certificate.
To be integrated properly, the extraction should fall back to the normal Mediator if no certificate is given or extraction using the certificate fails.
For a proof of concept, this should do:
Code:
--- /home/df/Documents/src/youtube-dl/youtube_dl/extractor/bbc.py
+++ /home/df/Documents/src/youtube-dl/youtube_dl/extractor/bbc.py.hak
@@ -57,13 +57,10 @@
_LOGIN_URL = 'https://account.bbc.com/signin'
_NETRC_MACHINE = 'bbc'
- _MEDIA_SELECTOR_URL_TEMPL = 'https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/%s/vpid/%s'
+ _MEDIA_SELECTOR_URL_TEMPL = 'https://securegate.iplayer.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/%s/vpid/%s'
_MEDIA_SETS = [
- # Provides HQ HLS streams with even better quality that pc mediaset but fails
- # with geolocation in some cases when it's even not geo restricted at all (e.g.
- # http://www.bbc.co.uk/programmes/b06bp7lf). Also may fail with selectionunavailable.
- 'iptv-all',
- 'pc',
+ 'legacy-iptv-all',
+ 'iptv-uhd',
]
_EMP_PLAYLIST_NS = 'http://bbc.co.uk/2008/emp/playlist'To be integrated properly, the extraction should fall back to the normal Mediator if no certificate is given or extraction using the certificate fails.