[youtube-dl] Download files from youtube.com or other video platforms

/df

Well-Known Member
Is the --config-location /mod/etc/youtube-dl.conf supposed to be missing from /mod/bin/youtube in the latest (-3)?
 
OP
prpr

prpr

Well-Known Member
Is the --config-location /mod/etc/youtube-dl.conf supposed to be missing from /mod/bin/youtube in the latest (-3)?
Yes. I modified the default as part of the package build, like I modified the /usr/bin/env stuff.
youtube now does the same as youtube-dl as they're identical scripts. The former is only retained for backwards compatibility.
 

/df

Well-Known Member
That seems entirely reasonable, since installable C packages should get configured with --prefix=/mod and thus find their conf in /mod/etc without any recourse to /etc.
 

/df

Well-Known Member
I've tested djcsdy's patch for BBC playlist support.

The patch seems to work (testing with --get-title), slowly, but with one of the listed URLs (https://www.bbc.co.uk/iplayer/episodes/b07xd230/josh?seriesId=p047lvcs, where it should find 6 episodes, but doesn't) python repeatedly took over the box; it first used 80% CPU as measured in top and gradually reached 80% memory, at which point the picture stuttered and froze and then humaxtv crashed. As it was a memory issue, the CF package cpulimit didn't help.

The operation of the entire extractor is pretty obscure and this patch just piles it on. More study needed.
 

Private61

Member
Hi,
Can anyone help?
I use YouTube-dl on Windows and the update option fails, youtube-dl-exe -U reports ERROR: can't find the current version. Please try again later.
I've visited https://yt-dl.org/ and only receive an old version not the latest November version.
Can anyone say whats happening or suggest an alternative source, the GitHub source has been taken down?
 

/df

Well-Known Member
The Windows .exe on the linked page is the version from 2020-11-01. I believe it's a self-contained package that you just download and run. You'll need to be Administrator (or "elevate") to install it system-wide: perhaps you're accidentally running an existing system-wide installation instead of the one you downloaded.
 

Private61

Member
Hi,
Thanks for the comment. Your comment of a self-contained package is correct. I don't run a it system wide but it is installed with Administration rights.
I run the programme from a known folder and it's version does not report the November version and the update, -U, fails.
Thanks for your comment.
 
OP
prpr

prpr

Well-Known Member
But you used /mod/bin/busybox/env and didn't make busybox a dependency. If it's not a dependency, /bin/env would be fine.
Oh, so I did. How slapdash of me. I'll fix it. But busybox is a dependency of webif, so it's rather unlikely anyone would have youtube-dl and not webif.
 

Wildebeest

Member
I know I'm interupting an expert conversation with a newbie and stupid question so apologies, but when downloading from iplayer what option(s) do I need to use to get subtitles please? (I've looked in the wiki and noted the comments a page or two back but am still unclear). Thanks.
 
OP
prpr

prpr

Well-Known Member
From here, you could try:
--write-sub --convert-subs srt
You might want to play with the other options mentioned. I have never tried any of this myself.
 

/df

Well-Known Member
... when downloading from iplayer what option(s) do I need to use to get subtitles please? (I've looked in the wiki and noted the comments a page or two back but am still unclear). ...
You are right to be unclear, because it's not trivial. Actually, people and gnus everywhere are baffled by the procedure.

The following needs to be done (other combinations may be possible, as in the link quoted above):
  1. use the yt-dl options --write-sub (to force the subtitle file to be written) and --convert-subs srt (to get the file in a format that the Humax player has a chance of understanding); you might also try --subtitle-format srt/best to get srt format subtitles if they are available;
  2. edit the resulting UTF-8 .srt file and save it in olde worlde ANSI text format, making the new subtitle file name match the media file name up to the extension (which means removing the .en-GB); the libiconv package includes the iconv command that can do this;
  3. strip out HTML-style tags that are left over from the ttml->srt conversion, which can be done with an editor that supports regular expressions, or on-box with sed.
The character encodings supported by the Humax player (what the manual calls subtitle languages) are displayed as you press the Sub button with a valid .srt file in place:
Latin 1
Latin 2
Latin 4
Cyrillic
Greek
Turkish

It has been suggested that UTF-7 may be an acceptable encoding, but it's not one of the above. The level of encoding is an option in UTF-7: characters like <> can be sent as-is or encoded using an escape sequence beginning with +; in the latter case the Humax player won't recognise a Latin 1 .srt.

We'll be going with Latin 1 which corresponds to ANSI.
Code:
#!/bin/sh
# Usage: fixsttl media_file
# for a media file, convert its .locale.srt, if any, to plain text .srt

# can be overriden STTL_LANG=da-DK, etc
STTL_LANG=${STTL_LANG:-en-GB}

main() {
        local ext froot srt
        [ -n "$1" ] || return
        # any other extensions?
        for ext in mp4 mpg mkv; do 
                froot=${1%.$ext}
                [ "$1" != "$froot" ]  && break
        done
        [  "$1" != "$froot" ] || return
        srt=${froot}.${STTL_LANG}.srt
        [ -r "$srt" ] || return
        # *.en-GB.srt -> *.srt
        iconv -f UTF-8 -t LATIN1 "$srt" | 
                # strip <tags> and </tags>
                sed -r -e 's@<[/a-zA-Z]+( [^>]*)?>@@g' > "${froot}.srt"  &&
                { rm -f -- "$srt"; return 0; }
}

main "$@"
Updated: preferred pattern is test_that_must_be_true || return, so if the test fails the return value is false by default.

Ob. instructions: use the File Editor in WebIf>Diagnostics, or your preferred editor in a telnet or WebShell, to create /mod/bin/fixsttl, make it executable, add --exec "fixsttl {}" to your yt-dl or qtube options. And you must have installed libiconv.

Example:
Code:
# youtube --write-sub --sub-format srt/best --convert-subs srt --exec "fixsttl {}" 'https://www.bbc.co.uk/iplayer/episode/m000n7g4/top-of-the-pops-04011990'
[bbc.co.uk] m000n7g4: Downloading video page
[bbc.co.uk] m000n7g4: Downloading playlist JSON
[bbc.co.uk] m000n7g3: Downloading media selection XML
[bbc.co.uk] m000n7g3: Downloading captions
...
[bbc.co.uk] m000n7g3: Downloading m3u8 information
...
[bbc.co.uk] m000n7g3: Downloading MPD manifest
...
[bbc.co.uk] m000n7g3: Downloading m3u8 information
...
[bbc.co.uk] m000n7g3: Downloading MPD manifest
...
[bbc.co.uk] m000n7g3: Downloading m3u8 information
[bbc.co.uk] m000n7g3: Downloading m3u8 information
...
[bbc.co.uk] m000n7g3: Downloading media selection XML
[bbc.co.uk] m000n7g3: Downloading captions
...
[bbc.co.uk] m000n7g3: Downloading MPD manifest
...
[info] Writing video subtitles to: /media/drive1/Video/Top_of_the_Pops_04_01_1990.en-GB.ttml
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 225
[download] Destination: /media/drive1/Video/Top_of_the_Pops_04_01_1990.mp4
[download] 100% of 367.73MiB in 06:48
[ffmpeg] Fixing malformed AAC bitstream in "/media/drive1/Video/Top_of_the_Pops_04_01_1990.mp4"
[ffmpeg] Converting subtitles
WARNING: You have requested to convert dfxp (TTML) subtitles into another format, which results in style information loss
Deleting original file /media/drive1/Video/Top_of_the_Pops_04_01_1990.en-GB.ttml (pass -k to keep)
[exec] Executing command: fixsttl /media/drive1/Video/Top_of_the_Pops_04_01_1990.mp4
#
And see the dreadfully grainy image.

Now what we need is for the BBC to show some programme about web page coding to see what happens to a line like "You shouldn't use the <marquee> tag but you can create a scrolling text effect with CSS Level 3 instead".
 

Attachments

  • sttl.jpg
    sttl.jpg
    20.4 KB · Views: 12
Last edited:

Wildebeest

Member
Thanks to both of you but especially to /df for taking the time to write such a complete and clear explanation & instructions of a complex topic (far more complex than I was expecting!). Much appreciated.
 
OP
prpr

prpr

Well-Known Member
use the File Editor in WebIf>Diagnostics, or your preferred editor in a telnet or WebShell, to create /mod/bin/fixsttl, make it executable,
Are you happy for this to be added to the package?
 

/df

Well-Known Member
Are you happy for this to be added to the package?
Wouldn't have posted otherwise! Though the caveat is that it needs testing by people who actually use subtitles, which is generally not me.

I don't think adding the libiconv dependency is a problem.
 

Luke

Well-Knwοn Мember

/df

Well-Known Member
In the new unpacked version of yt-dl, there is the problem that the upstream bin/youtube-dl doesn't launch the program, even if the #! is tweaked to match /bin/env.

Following a Debian/Ubuntu desktop Linux installation, I invented a directory /mod/lib/python2.7/dist-packages as the location for the unpacked module directory for yt-dl, and any other Python module installed by the CF package manager. Without the aid of a time machine, this choice wasn't configured into the CF python package, so Python doesn't find a module there.

There are at least these ways of solving the problem:
  1. set the PYTHONPATH environment variable, which is inserted in the configured module search path;
  2. add the directory using site.addsitedir() in a module with the special name sitecustomize, which is run when Python starts;
  3. rebuild the python package with the modified module search path.
Obviously #1 is the simplest, although, as PYTHONPATH is inserted at the start of the module search path, a package installed in Python wouldn't override the same package installed from the CF repo, contrary to the Debian case. In theory it should be possible to combine this with the #! tweak as below:
Code:
#!/bin/env PYTHONPATH=/mod/lib/python2.7/dist-packages python
But, this hangs without ever appearing to run Python. So the solution is to add the setting to /mod/etc/profile/python.

This, together with the changes in the latest release from the temporary upstream GitLab repo, is implemented at the latest https://git.hpkg.tv/df/youtube-dl/src/branch/Hummy.
 
Top