[youtube-dl] Download videos from youtube.com or other video platforms

/df

Active Member
It worked for me, but I think there's a couple of bugs related to subtitles:
...
I've now committed what I hope is a full fix for subtitles in the PR linked in post #319. It appears to work the same with Python 2.7 here and with 3.5 in a Linux derived from Ubuntu Xenial.
 

MontysEvilTwin

Well-Known Member
It worked for me, but I think there's a couple of bugs related to subtitles:
  1. subtitles may be returned as None rather than an empty list (but see #2) - initial fix in https://github.com/ytdl-org/youtube-dl/pull/26821;
  2. formats are returned as a list (doesn't have items()) but subtitles seem to be a dict (has items()).
Was an option like --list-subs involved?
You are right. It works (youtube-dl_2020.09.20-2) with the standard configuration file but fails with the '--write-sub' option added.
 

/df

Active Member
I haven't really looked properly at this, but the version it prints is 2020.07.28, which is backlevel somehow.
That's quite properly enough. I hadn't twigged that your last releases also pulled in the latest upstream, and I haven't yet set up a build from the forked repository. Round again.
 
OP
prpr

prpr

Well-Known Member
Code:
                # subtitles subtitles (a dict {(lang,sttl)})
Is there an unintentional duplicate word there?
Code:
                    if not subtitles:
                        subtitles = formatsAndSubtitles[1]
                    else:
                        # prioritise the first sttl for each lang
                        formatsAndSubtitles[1].update(subtitles)
                        subtitles = formatsAndSubtitles[1]
Can this not be simplified to:
Code:
                    if subtitles:
                        # prioritise the first sttl for each lang
                        formatsAndSubtitles[1].update(subtitles)
                    subtitles = formatsAndSubtitles[1]
 

/df

Active Member
Thx, the dup is clear; and yes, you could eliminate the extra line. In an ideal world the Python compiler would do that anyway (see following post).
 

/df

Active Member
Incidentally, the reason why the package doesn't run when unpacked is the lack of /usr/bin/env. If CF 3.14 ever escapes, this should be fixed.

If the installation detects that file, it should unpack the package into /mod/lib/python2.7/dist-packages/youtube-dl, with a stub file as /mod/bin/youtube-dl:
Code:
#!/bin/sh
python /mod/lib/python2.7/dist-packages/youtube-dl "$@"
This brings the elapsed time to complete youtube-dl --help from 45s to 10s, as the code doesn't have to be unpacked and (after the first run) the Python code has been compiled into Python byte code.

Also, to make the youtube-dl package work as before, /mod/bin/youtube:
Code:
#!/bin/sh
python /mod/lib/python2.7/dist-packages/youtube-dl --config-location /mod/etc/youtube-dl.conf "$@"

The missing /usr/bin/env can be provided by a bind mount, which is how that test ran; but my set-up, which tries to make various RO directories RW is quite tricky, so I'm not confident to deliver it in a package.

However, if there isn't a /usr/bin/env, we can patch the extracted file /mod/lib/python2.7/dist-packages/youtube-dl/__main__.py to change it to /bin/env, and still succeed:
Code:
sed -i -e 's@/usr/bin/@/bin/@' /mod/lib/python2.7/dist-packages/youtube-dl/__main__.py
 
Last edited:
OP
prpr

prpr

Well-Known Member
Incidentally, the reason why the package doesn't run when unpacked is the lack of /usr/bin/env.
Yes, I've been working on that...
If the installation detects that file
What and how exactly?
This brings the elapsed time to complete youtube-dl --help from 45s to 10s, as the code doesn't have to be unpacked and (after the first run) the Python code has been compiled into Python byte code.
Yes, I have it running like that currently. We're obviously thinking along the same lines.
we can patch the extracted file
Indeed! There are several with /usr/bin/env in, so I've been patching them all.
 

/df

Active Member
I found it was enough to patch the __main__.py per my updated post above. Maybe other entry-points would use some other top-level file with a #! but we don't offer them. I remember trying this originally, failing, and moving on, but obviously my Python and CF Foo has increased. My patch that provides /usr/bin/env has been in place for more than 2 years and I only checked this today :-(.

The answer to "what and how" is test -x /usr/bin/env.

Just fixed the DL link above.
 

/df

Active Member
That's telnet and vi for you. Time to enter the 1990s. Also, code formatting is, of all things, something that software should be handling, especially when (and I don't necessarily approve) it's actually syntactically significant.
 
OP
prpr

prpr

Well-Known Member
This still doesn't work:
Code:
humax # youtube --write-sub https://www.bbc.co.uk/iplayer/episode/m000n7g4/top-of-the-pops-04011990
...
Traceback (most recent call last):
  File "/mod/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/mod/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/mnt/hd2/mod/lib/python2.7/dist-packages/youtube-dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/__init__.py", line 474, in main
    _real_main(argv)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/__init__.py", line 464, in _real_main
    retcode = ydl.download(all_urls)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/YoutubeDL.py", line 2019, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/bbc.py", line 602, in _real_extract
    self._sort_formats(formats)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/bbc.py", line 498, in _download_playlist
    except ExtractorError as ee:
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/bbc.py", line 365, in _download_media_selector
    except BBCCoUkIE.MediaSelectionError as e:
ValueError: dictionary update sequence element #0 has length 5; 2 is required
from:
Code:
        formats = []
        subtitles = None
        for mediaselector_url in self._MEDIASELECTOR_URLS:
            try:
                formatsAndSubtitles = self._download_media_selector_url(
                    mediaselector_url % programme_id, programme_id)
                if formatsAndSubtitles[0]:
                    formats += formatsAndSubtitles[0]
                if formatsAndSubtitles[1]:
                    if subtitles:
                        formatsAndSubtitles[1].update(subtitles)
                    subtitles = formatsAndSubtitles[1]
            except BBCCoUkIE.MediaSelectionError as e:
 
Last edited:

/df

Active Member
"Works for me"
Code:
youtube-dl  --write-sub https://www.bbc.co.uk/iplayer/episode/m000n7g4/top-of-the-pops-04011990
[bbc.co.uk] m000n7g4: Downloading video page
[bbc.co.uk] m000n7g4: Downloading playlist JSON
[bbc.co.uk] m000n7g3: Downloading media selection XML
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
...
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
[info] Writing video subtitles to: Top of the Pops, 04_01_1990-m000n7g3.en-GB.ttml
[dashsegments] Total fragments: 450
^C
a) with the unpacked modified code that looks like what you posted above
b) with the packed program that was zipped and apparently uploaded
c) with the packed program from the zip file that was uploaded and is identical to the downloaded zip
Something weird? Is there a radiation experiment running near the CPU?
 
OP
prpr

prpr

Well-Known Member
"Works for me"
Does for me now. I think I might have confused it (and me) by not blasting the .pyc files.
Probably good to test on an unsullied system, which wouldn't be any of mine!
Anybody else want to volunteer then? Otherwise it'll be a case of publish and be damned :D.
That's telnet and vi for you. Time to enter the 1990s.
You could at least use nano.
especially when (and I don't necessarily approve) it's actually syntactically significant.
I certainly don't approve and I really dislike things like that.
 
OP
prpr

prpr

Well-Known Member
youtube-dl seems to timestamp all its .py files to 31-12-1999 18:01:00 though. Presumably they do that the same on every release, so the only way out of this would seem to be to remove any existing .pyc files on installation, to guarantee a re-compile. This seems to be a bit annoying/stupid to me.

And you should have quoted the 2.7 manual for our purposes!
 
Top