• The forum software that supports hummy.tv has been upgraded to XenForo 2.3!

    Please bear with us as we continue to tweak things, and feel free to post any questions, issues or suggestions in the upgrade thread.

[youtube-dl] Download files from youtube.com or other video platforms

/df said:
It worked for me, but I think there's a couple of bugs related to subtitles:
...
Click to expand...
I've now committed what I hope is a full fix for subtitles in the PR linked in post #319. It appears to work the same with Python 2.7 here and with 3.5 in a Linux derived from Ubuntu Xenial.
 
/df said:
It worked for me, but I think there's a couple of bugs related to subtitles:
  1. subtitles may be returned as None rather than an empty list (but see #2) - initial fix in https://github.com/ytdl-org/youtube-dl/pull/26821;
  2. formats are returned as a list (doesn't have items()) but subtitles seem to be a dict (has items()).
Was an option like --list-subs involved?
Click to expand...
You are right. It works (youtube-dl_2020.09.20-2) with the standard configuration file but fails with the '--write-sub' option added.
 
prpr said:
I haven't really looked properly at this, but the version it prints is 2020.07.28, which is backlevel somehow.
Click to expand...
That's quite properly enough. I hadn't twigged that your last releases also pulled in the latest upstream, and I haven't yet set up a build from the forked repository. Round again.
 
Code: 
                # subtitles subtitles (a dict {(lang,sttl)})
Is there an unintentional duplicate word there?
Code: 
                    if not subtitles:
                        subtitles = formatsAndSubtitles[1]
                    else:
                        # prioritise the first sttl for each lang
                        formatsAndSubtitles[1].update(subtitles)
                        subtitles = formatsAndSubtitles[1]
Can this not be simplified to:
Code: 
                    if subtitles:
                        # prioritise the first sttl for each lang
                        formatsAndSubtitles[1].update(subtitles)
                    subtitles = formatsAndSubtitles[1]
 
Thx, the dup is clear; and yes, you could eliminate the extra line. In an ideal world the Python compiler would do that anyway (see following post).
 
Incidentally, the reason why the package doesn't run when unpacked is the lack of /usr/bin/env. If CF 3.14 ever escapes, this should be fixed.

If the installation detects that file, it should unpack the package into /mod/lib/python2.7/dist-packages/youtube-dl, with a stub file as /mod/bin/youtube-dl:
Code: 
#!/bin/sh
python /mod/lib/python2.7/dist-packages/youtube-dl "$@"
This brings the elapsed time to complete youtube-dl --help from 45s to 10s, as the code doesn't have to be unpacked and (after the first run) the Python code has been compiled into Python byte code.

Also, to make the youtube-dl package work as before, /mod/bin/youtube:
Code: 
#!/bin/sh
python /mod/lib/python2.7/dist-packages/youtube-dl --config-location /mod/etc/youtube-dl.conf "$@"

The missing /usr/bin/env can be provided by a bind mount, which is how that test ran; but my set-up, which tries to make various RO directories RW is quite tricky, so I'm not confident to deliver it in a package.

However, if there isn't a /usr/bin/env, we can patch the extracted file /mod/lib/python2.7/dist-packages/youtube-dl/__main__.py to change it to /bin/env, and still succeed:
Code: 
sed -i -e 's@/usr/bin/@/bin/@' /mod/lib/python2.7/dist-packages/youtube-dl/__main__.py
 
Last edited:
/df said:
Incidentally, the reason why the package doesn't run when unpacked is the lack of /usr/bin/env.
Click to expand...
Yes, I've been working on that...
If the installation detects that file
Click to expand...
What and how exactly?
This brings the elapsed time to complete youtube-dl --help from 45s to 10s, as the code doesn't have to be unpacked and (after the first run) the Python code has been compiled into Python byte code.
Click to expand...
Yes, I have it running like that currently. We're obviously thinking along the same lines.
we can patch the extracted file
Click to expand...
Indeed! There are several with /usr/bin/env in, so I've been patching them all.
 
I found it was enough to patch the __main__.py per my updated post above. Maybe other entry-points would use some other top-level file with a #! but we don't offer them. I remember trying this originally, failing, and moving on, but obviously my Python and CF Foo has increased. My patch that provides /usr/bin/env has been in place for more than 2 years and I only checked this today :-(.

The answer to "what and how" is test -x /usr/bin/env.

Just fixed the DL link above.
 
That's telnet and vi for you. Time to enter the 1990s. Also, code formatting is, of all things, something that software should be handling, especially when (and I don't necessarily approve) it's actually syntactically significant.
 
This still doesn't work:
Code: 
humax # youtube --write-sub https://www.bbc.co.uk/iplayer/episode/m000n7g4/top-of-the-pops-04011990
...
Traceback (most recent call last):
  File "/mod/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/mod/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/mnt/hd2/mod/lib/python2.7/dist-packages/youtube-dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/__init__.py", line 474, in main
    _real_main(argv)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/__init__.py", line 464, in _real_main
    retcode = ydl.download(all_urls)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/YoutubeDL.py", line 2019, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/bbc.py", line 602, in _real_extract
    self._sort_formats(formats)
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/bbc.py", line 498, in _download_playlist
    except ExtractorError as ee:
  File "/mod/lib/python2.7/dist-packages/youtube-dl/youtube_dl/extractor/bbc.py", line 365, in _download_media_selector
    except BBCCoUkIE.MediaSelectionError as e:
ValueError: dictionary update sequence element #0 has length 5; 2 is required
from:
Code: 
        formats = []
        subtitles = None
        for mediaselector_url in self._MEDIASELECTOR_URLS:
            try:
                formatsAndSubtitles = self._download_media_selector_url(
                    mediaselector_url % programme_id, programme_id)
                if formatsAndSubtitles[0]:
                    formats += formatsAndSubtitles[0]
                if formatsAndSubtitles[1]:
                    if subtitles:
                        formatsAndSubtitles[1].update(subtitles)
                    subtitles = formatsAndSubtitles[1]
            except BBCCoUkIE.MediaSelectionError as e:
 
Last edited:
"Works for me"
Code: 
youtube-dl  --write-sub https://www.bbc.co.uk/iplayer/episode/m000n7g4/top-of-the-pops-04011990
[bbc.co.uk] m000n7g4: Downloading video page
[bbc.co.uk] m000n7g4: Downloading playlist JSON
[bbc.co.uk] m000n7g3: Downloading media selection XML
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
...
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
[bbc.co.uk] m000n7g3: Downloading captions
[info] Writing video subtitles to: Top of the Pops, 04_01_1990-m000n7g3.en-GB.ttml
[dashsegments] Total fragments: 450
^C
a) with the unpacked modified code that looks like what you posted above
b) with the packed program that was zipped and apparently uploaded
c) with the packed program from the zip file that was uploaded and is identical to the downloaded zip
Something weird? Is there a radiation experiment running near the CPU?
 
/df said:
"Works for me"
Click to expand...
Does for me now. I think I might have confused it (and me) by not blasting the .pyc files.
/df said:
Probably good to test on an unsullied system, which wouldn't be any of mine!
Click to expand...
Anybody else want to volunteer then? Otherwise it'll be a case of publish and be damned :D.
/df said:
That's telnet and vi for you. Time to enter the 1990s.
Click to expand...
You could at least use nano.
especially when (and I don't necessarily approve) it's actually syntactically significant.
Click to expand...
I certainly don't approve and I really dislike things like that.
 
prpr said:
Does for me now. I think I might have confused it (and me) by not blasting the .pyc files.
...
Click to expand...
I was pleased to confirm what I thought, plus some more:
https://docs.python.org/3/tutorial/modules.html#compiled-python-files said:
...
Python checks the modification date of the source against the compiled version to see if it’s out of date and needs to be recompiled. This is a completely automatic process. Also, the compiled modules are platform-independent, so the same library can be shared among systems with different architectures.

Python does not check the cache in two circumstances. First, it always recompiles and does not store the result for the module that’s loaded directly from the command line. Second, it does not check the cache if there is no source module. To support a non-source (compiled only) distribution, the compiled module must be in the source directory, and there must not be a source module.
...
Click to expand...

When I actually hadn't entered the 1990s I was a down-the-line Emacs person, completely incredulous that you would have to enter a special insert mode to type text (certainly didn't do that with EDT). But now the keystrokes have all gone from memory as one can't rely on there being anything other than vi.
 
Last edited:
youtube-dl seems to timestamp all its .py files to 31-12-1999 18:01:00 though. Presumably they do that the same on every release, so the only way out of this would seem to be to remove any existing .pyc files on installation, to guarantee a re-compile. This seems to be a bit annoying/stupid to me.

And you should have quoted the 2.7 manual for our purposes!
 
You must log in or register to reply here.
Back
Top