• The forum software that supports hummy.tv has been upgraded to XenForo 2.3!

    Please bear with us as we continue to tweak things, and feel free to post any questions, issues or suggestions in the upgrade thread.

[youtube-dl] Download files from youtube.com or other video platforms

Is the --config-location /mod/etc/youtube-dl.conf supposed to be missing from /mod/bin/youtube in the latest (-3)?
 
/df said:
Is the --config-location /mod/etc/youtube-dl.conf supposed to be missing from /mod/bin/youtube in the latest (-3)?
Click to expand...
Yes. I modified the default as part of the package build, like I modified the /usr/bin/env stuff.
youtube now does the same as youtube-dl as they're identical scripts. The former is only retained for backwards compatibility.
 
That seems entirely reasonable, since installable C packages should get configured with --prefix=/mod and thus find their conf in /mod/etc without any recourse to /etc.
 
I've tested djcsdy's patch for BBC playlist support.

The patch seems to work (testing with --get-title), slowly, but with one of the listed URLs (https://www.bbc.co.uk/iplayer/episodes/b07xd230/josh?seriesId=p047lvcs, where it should find 6 episodes, but doesn't) python repeatedly took over the box; it first used 80% CPU as measured in top and gradually reached 80% memory, at which point the picture stuttered and froze and then humaxtv crashed. As it was a memory issue, the CF package cpulimit didn't help.

The operation of the entire extractor is pretty obscure and this patch just piles it on. More study needed.
 
Hi,
Can anyone help?
I use YouTube-dl on Windows and the update option fails, youtube-dl-exe -U reports ERROR: can't find the current version. Please try again later.
I've visited https://yt-dl.org/ and only receive an old version not the latest November version.
Can anyone say whats happening or suggest an alternative source, the GitHub source has been taken down?
 
The Windows .exe on the linked page is the version from 2020-11-01. I believe it's a self-contained package that you just download and run. You'll need to be Administrator (or "elevate") to install it system-wide: perhaps you're accidentally running an existing system-wide installation instead of the one you downloaded.
 
Hi,
Thanks for the comment. Your comment of a self-contained package is correct. I don't run a it system wide but it is installed with Administration rights.
I run the programme from a known folder and it's version does not report the November version and the update, -U, fails.
Thanks for your comment.
 
/df said:
But you used /mod/bin/busybox/env and didn't make busybox a dependency. If it's not a dependency, /bin/env would be fine.
Click to expand...
Oh, so I did. How slapdash of me. I'll fix it. But busybox is a dependency of webif, so it's rather unlikely anyone would have youtube-dl and not webif.
 
I know I'm interupting an expert conversation with a newbie and stupid question so apologies, but when downloading from iplayer what option(s) do I need to use to get subtitles please? (I've looked in the wiki and noted the comments a page or two back but am still unclear). Thanks.
 
From here, you could try:
--write-sub --convert-subs srt
You might want to play with the other options mentioned. I have never tried any of this myself.
 
Wildebeest said:
... when downloading from iplayer what option(s) do I need to use to get subtitles please? (I've looked in the wiki and noted the comments a page or two back but am still unclear). ...
Click to expand...
You are right to be unclear, because it's not trivial. Actually, people and gnus everywhere are baffled by the procedure.

The following needs to be done (other combinations may be possible, as in the link quoted above):
  1. use the yt-dl options --write-sub (to force the subtitle file to be written) and --convert-subs srt (to get the file in a format that the Humax player has a chance of understanding); you might also try --subtitle-format srt/best to get srt format subtitles if they are available;
  2. edit the resulting UTF-8 .srt file and save it in olde worlde ANSI text format, making the new subtitle file name match the media file name up to the extension (which means removing the .en-GB); the libiconv package includes the iconv command that can do this;
  3. strip out HTML-style tags that are left over from the ttml->srt conversion, which can be done with an editor that supports regular expressions, or on-box with sed.
The character encodings supported by the Humax player (what the manual calls subtitle languages) are displayed as you press the Sub button with a valid .srt file in place:
Latin 1
Latin 2 (Central/Eastern Europe)
Latin 4 (Northern Europe/Baltic)
Cyrillic
Greek
Turkish

It has been suggested that UTF-7 may be an acceptable encoding, but it's not one of the above. The level of encoding is an option in UTF-7: characters like <> can be sent as-is or encoded using an escape sequence beginning with +; in the latter case the Humax player won't recognise a Latin 1 .srt.

We'll be going with Latin 1 which corresponds to ANSI.
Code: 
#!/bin/sh
# Usage: fixsttl media_file
# for a media file, convert its .locale.srt, if any, to plain text .srt

# can be overriden STTL_LANG=da-DK, etc
STTL_LANG=${STTL_LANG:-en-GB}

main() {
        local ext froot srt
        [ -n "$1" ] || return
        # any other extensions?
        for ext in mp4 mpg mkv; do 
                froot=${1%.$ext}
                [ "$1" != "$froot" ]  && break
        done
        [  "$1" != "$froot" ] || return
        srt=${froot}.${STTL_LANG}.srt
        [ -r "$srt" ] || return
        # *.en-GB.srt -> *.srt
        iconv -f UTF-8 -t LATIN1 "$srt" | 
                # strip <tags> and </tags>
                sed -r -e 's@<[/a-zA-Z]+( [^>]*)?>@@g' > "${froot}.srt"  &&
                { rm -f -- "$srt"; return 0; }
}

main "$@"
Updated: preferred pattern is test_that_must_be_true || return, so if the test fails the return value is false by default.

Ob. instructions: use the File Editor in WebIf>Diagnostics, or your preferred editor in a telnet or WebShell, to create /mod/bin/fixsttl, make it executable, add --exec "fixsttl {}" to your yt-dl or qtube options. And you must have installed libiconv.

Example:
Code: 
# youtube --write-sub --sub-format srt/best --convert-subs srt --exec "fixsttl {}" 'https://www.bbc.co.uk/iplayer/episode/m000n7g4/top-of-the-pops-04011990'
[bbc.co.uk] m000n7g4: Downloading video page
[bbc.co.uk] m000n7g4: Downloading playlist JSON
[bbc.co.uk] m000n7g3: Downloading media selection XML
[bbc.co.uk] m000n7g3: Downloading captions
...
[bbc.co.uk] m000n7g3: Downloading m3u8 information
...
[bbc.co.uk] m000n7g3: Downloading MPD manifest
...
[bbc.co.uk] m000n7g3: Downloading m3u8 information
...
[bbc.co.uk] m000n7g3: Downloading MPD manifest
...
[bbc.co.uk] m000n7g3: Downloading m3u8 information
[bbc.co.uk] m000n7g3: Downloading m3u8 information
...
[bbc.co.uk] m000n7g3: Downloading media selection XML
[bbc.co.uk] m000n7g3: Downloading captions
...
[bbc.co.uk] m000n7g3: Downloading MPD manifest
...
[info] Writing video subtitles to: /media/drive1/Video/Top_of_the_Pops_04_01_1990.en-GB.ttml
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 225
[download] Destination: /media/drive1/Video/Top_of_the_Pops_04_01_1990.mp4
[download] 100% of 367.73MiB in 06:48
[ffmpeg] Fixing malformed AAC bitstream in "/media/drive1/Video/Top_of_the_Pops_04_01_1990.mp4"
[ffmpeg] Converting subtitles
WARNING: You have requested to convert dfxp (TTML) subtitles into another format, which results in style information loss
Deleting original file /media/drive1/Video/Top_of_the_Pops_04_01_1990.en-GB.ttml (pass -k to keep)
[exec] Executing command: fixsttl /media/drive1/Video/Top_of_the_Pops_04_01_1990.mp4
#
And see the dreadfully grainy image.

Now what we need is for the BBC to show some programme about web page coding to see what happens to a line like "You shouldn't use the <marquee> tag but you can create a scrolling text effect with CSS Level 3 instead".
 

Attachments

  • sttl.jpg
    sttl.jpg
    20.4 KB · Views: 15
Last edited:
Thanks to both of you but especially to /df for taking the time to write such a complete and clear explanation & instructions of a complex topic (far more complex than I was expecting!). Much appreciated.
 
/df said:
use the File Editor in WebIf>Diagnostics, or your preferred editor in a telnet or WebShell, to create /mod/bin/fixsttl, make it executable,
Click to expand...
Are you happy for this to be added to the package?
 
prpr said:
Are you happy for this to be added to the package?
Click to expand...
Wouldn't have posted otherwise! Though the caveat is that it needs testing by people who actually use subtitles, which is generally not me.

I don't think adding the libiconv dependency is a problem.
 
trog said:
Not sure how relevant this is here but ............................."Two weeks ago the RIAA asked GitHub to remove the open-source stream-ripper software youtube-dl." https://torrentfreak.com/riaa-takedowns-backfire-as-pirated-mp3s-now-surface-on-github-201104/
https://torrentfreak.com/github-warns-users-reposting-youtube-dl-they-could-be-banned-201102/
Click to expand...
Have a look at this discussion https://hummy.tv/forum/threads/qtube-webif-front-end-for-youtube-dl.8948/page-8#post-146866
 
In the new unpacked version of yt-dl, there is the problem that the upstream bin/youtube-dl doesn't launch the program, even if the #! is tweaked to match /bin/env.

Following a Debian/Ubuntu desktop Linux installation, I invented a directory /mod/lib/python2.7/dist-packages as the location for the unpacked module directory for yt-dl, and any other Python module installed by the CF package manager. Without the aid of a time machine, this choice wasn't configured into the CF python package, so Python doesn't find a module there.

There are at least these ways of solving the problem:
  1. set the PYTHONPATH environment variable, which is inserted in the configured module search path;
  2. add the directory using site.addsitedir() in a module with the special name sitecustomize, which is run when Python starts;
  3. rebuild the python package with the modified module search path.
Obviously #1 is the simplest, although, as PYTHONPATH is inserted at the start of the module search path, a package installed in Python wouldn't override the same package installed from the CF repo, contrary to the Debian case. In theory it should be possible to combine this with the #! tweak as below:
Code: 
#!/bin/env PYTHONPATH=/mod/lib/python2.7/dist-packages python
But, this hangs without ever appearing to run Python. So the solution is to add the setting to /mod/etc/profile/python.

This, together with the changes in the latest release from the temporary upstream GitLab repo, is implemented at the latest https://git.hpkg.tv/df/youtube-dl/src/branch/Hummy.
 
You must log in or register to reply here.
Back
Top