[youtube-dl] Download videos from youtube.com or other video platforms

MymsMan

Ad detector
qtube doesn't change the download options

I have had problems with dowloading that I haven't investigated yet, when specifying lower resolution height<=540 I have been getting format unavailable but omitting that I have got sound but no video - perhaps it has downloaded too high a resolution
 

Black Hole

May contain traces of nut
Since this latest update all downloads from the BBC seem to be lowest quality. Has the default config changed?
How do I get it to default to best?
It may be intertwined with th Qtube package.

Edit:
This is my config file at /mod/etc/youtube-dl.conf

--restrict-filenames
--prefer-ffmpeg
-f "best[height<=?1080][fps<=?30]"
-o "/mnt/hd2/My Video/%(title)s.%(ext)s"
I am under the impression that qtube doesn't automatically use the user default settings in youtube-dl.conf, certainly when it comes to the -o switch... unless that's been cured in the mean time. Does that have any bearing?
It has into youtube not youtube-dl .
And as qtube seems to call youtube then...

Try using just "-o" in the Process options box (without the quotes). Then it should default to what's in the config. file.
Is there a youtube.conf? Should qtube be calling youtube-dl instead of youtube?
 
OP
prpr

prpr

Well-Known Member
-f "best[height<=?1080][fps<=?30]"
Sorry, this is probably supposed to say [fps<=?60] rather than 30. Try it and see what happens.
Constant confusion over the actual meaning of fps (fields/frames), although I don't see why it should affect resolution, but it does - 704x396 vs. 1280x720 on a quick test.
 

4ndy

Forum Supporter
I am under the impression that qtube doesn't automatically use the user default settings in youtube-dl.conf, certainly when it comes to the -o switch... unless that's been cured in the mean time. Does that have any bearing?

Is there a youtube.conf? Should qtube be calling youtube-dl instead of youtube?
-o in the options jus throws up an error.
 

4ndy

Forum Supporter
Sorry, this is probably supposed to say [fps<=?60] rather than 30. Try it and see what happens.
Constant confusion over the actual meaning of fps (fields/frames), although I don't see why it should affect resolution, but it does - 704x396 vs. 1280x720 on a quick test.
That fixed it. Thank you.
 

/df

Active Member
However there's a bit more complexity here.

The upstream FM explains that the default format is bestvideo+bestaudio/best to handle both sites that only send separate video and audio streams as well as sites with combined streams. It would be nice if a combined stream were preferred if the best separate streams had the same resolution or less, and one test seems to show that this is what happens.

Therefore something like -f '(bestvideo+bestaudio/best)[height<=?1080][fps<=?60]' would be better.

A further nicety, but not relevant for qtube, is that a combined stream is preferred if the output is a pipe.
 

Ezra Pound

Well-Known Member
I have recently had a problem with Youtube-dl files, I downloaded all 12 Car Share files from BBC I-Player and S1E2 stopped downloading half way through, not sure why, the PuTTY session just froze, anyway I restarted the download and it continued from where it left off and created an MP4 file, however the video had a break-up at the point where PuTTY stopped and it looks like the file didn't seamlessly join when the download re-started, so here is the question does anyone know of an MP4 file tester that could find any break-ups like this so that I can have some confidence in my downloaded files without having to watch then all?
 

Black Hole

May contain traces of nut
Whatever you discover might be applicable in my situation too, rather than migrating the content to a PC.
 

Ezra Pound

Well-Known Member
Well, VLC stats did find 4 'Lost' frames on S1E2-bad and no lost frames on S1E2 but it runs in real time so it took 28 Mins for each file, I had more luck with (Windows) ffmpeg version git-2020-01-06-1e3f4b5 e.g :-

Code:
ffmpeg -v error -i S1E1.mp4 -f null - 2>S1E1.log
took 63 seconds to scan each file (A batch file of 13 lines took 14 Mins) and produced a zero length log file for the 12 good files and this for the 'bad' file :-

Code:
[h264 @ 000000810cd2f2c0] co located POCs unavailable
[h264 @ 000000810cda6b00] co located POCs unavailable
[null @ 000000810cceed00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3074 >= 3074
[null @ 000000810cceed00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3082 >= 3082

{10400 more lines starting null}

[null @ 000000810cceed00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 86292 >= 86292
[null @ 000000810cceed00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 86300 >= 86300
So all in all a good result for ffmpeg

BTW
The above ffmpeg command line will run on the Humax but I wouldn't bother trying it, is is still running on the same file used for the 63 seconds (Windows) test after some 85 Minutes
 
Last edited:

/df

Active Member
So apparently the problem was the repetition of the frame across the split in the download, and it should be possible with something like Avidemux, or indeed a magical ffmpeg incantation, to remove the first of the repeated frames.

Running EP's check against a 1.5GB MP4 ("Stewart Lee - Content Provider") took most of 19 hours on my HDR with ffmpeg 4.1.
 

Black Hole

May contain traces of nut
IIRC VideoReDo has a "quick stream fix" function... but I guess that might simply act as a GUI for ffmpeg.
 

MymsMan

Ad detector
So apparently the problem was the repetition of the frame across the split in the download, and it should be possible with something like Avidemux, or indeed a magical ffmpeg incantation, to remove the first of the repeated frames.
Or it might be less hassle to just rename/delete the file and repeat the download
 

Ezra Pound

Well-Known Member
Or it might be less hassle to just rename/delete the file and repeat the download
That's the path I took, I though if there was a chunk missing It can't be fixed and even with my crappy broadband link the new download only took about 25 Mins.
 

Black Hole

May contain traces of nut
If you call that crappy, god knows what you would call mine. My signal is now so bad that if I really want something in HiDef I have to download it. Rick Stein in Mexico took over 90 minutes each.
 

bottletop

New Member
Subtitles for ITV and BBC downloads
I use qtube (that calls youtube-dl) to download files from BBC or ITV, and use this to check to see if a program has subtitles
youtube-dl --list-subs <url>. As many will know, the downlaoded files are mp4 format, and the Humax will be able to pick up subtitles if a matching srt file has the exact name with srt extention <filename>.srt
Background info regarding using srt files on Humax
https://wiki.hummy.tv/wiki/Custom_Firmware_Package_Notes#SRT
https://hummy.tv/forum/threads/handbrake-files-subtitles.8273/

ITV has lang en, type vtt. We can use the youtube-dl convert option, then change the filename slightly.
I find using these options work well
--sub-lang en --write-sub --convert-subs srt
After download, I manually locate the paired file <filename>.en.srt and rename it to <filename>.srt

BBC has lang en or en-GB (for Click), type ttml. The youtube-dl convert works ok for ttml (but the result is not that great for the Humax) and we still need to change the filename. I find using these options work well
--sub-lang en,en-GB --write-sub -k
After download, I manually locate the paired file <filename>.en.ttml or <filename>.en-GB.ttml
convert the ttml file into srt by using python script found here https://github.com/yuppity/ttml2srt
eg python ttml2srt.py <filename>.en-GB.ttml <filename>.srt

So for ease of use and simplicity I just add this to the options file /mod/etc/youtube-dl.conf
--sub-lang en,en-GB --write-sub -k

I will add --convert-subs srt to qtube when kicking off ITV downloads.
Note this is just a workaround for the Humax, as srt files often have suffix en.srt for english, but Humax expects it to be just .srt

I tried to write this script to semi automate the process - eg calling via cron. It works on Linux Mint shell, but not on the Humax.
The offeding contruct seems to be while IFS= read -r -d '' iitem; do so I got stuck!
Code:
#find files of type ttml and checks to see if there is a matching srt for it.
#If srt does NOT exist, create it using the python script ttml2srt.py

#find "/mnt/hd2/My Video/60-Downloaded_Video/" -type f -name '*.en*.ttml' -print0 |
find "Documents/My tv-subtitles/"           -type f -name '*.en*.ttml' -print0 |
while IFS= read -r -d ''  iitem; do
#    printf '%s\n' "$iitem"
   ipath="${iitem%/*}"
   ifilename="${iitem##*/}"
   ibasename="${ifilename%%.*}"
   isrtname="$ibasename.srt"
   inewfile="$ipath/$isrtname"

   if [ ! -f "$inewfile" ]
   then
     echo "Creating srt-"  "$inewfile"
     #python /mod/tmp/ztools/ttml2srt.py "$iitem"  > "$inewfile" ;
     python Documents/ttml2srt2.py "$iitem"  > "$inewfile" ;
   fi
done
 
Last edited:

/df

Active Member
...
I tried to write this script to semi automate the process - eg calling via cron. It works on Linux Mint shell, but not on the Humax.
The offending construct seems to be while IFS= read -r -d '' iitem; do so I got stuck!
...
read doesn't have a -d (end of record delimiter) option in POSIX, only -r (shellcheck.net is good for validating shell scripts). The default Humax shell (Busybox ash) adds -p (prompt). You need to run the script with bash (ie install that package and change the #!), or work around it with what ash offers. awk may be a more amenable tool than the shell. Or of course Python.
 
Last edited:

bottletop

New Member
read doesn't have a -d (end of record delimiter) option in POSIX, only -r (shellcheck.net is good for validating shell scripts). The default Humax shell (Busybox ash) adds -p (prompt). You need to run the script with bash (ie install that package and change the #!)....
Thank you! That was the bit I overlooked completely.
So having it as a script and this at the beginning helps #!/mod/bin/bash
..or work around it with what ash offers. awk may be a more amenable tool than the shell.
Good idea, I'll need to look into this option.
..Or of course Python.
Erm I've never written in Python before so this is a little out of my league.
 
Last edited:

/df

Active Member
Just that the called script is in Python, so could easily be imported into a script by someone more skilled than you or me.

In your script, I suppose you don't have .ttml files with embedded newlines in the names. In that case this
Code:
...
find ... -type f |
    while IFS='' read -r iitem; do
...
ought to be fine in a POSIX shell like ash.

The pain of the find command is that it's a separate program, and consequently -exec actions don't know anything about the script from which they're being called. So you have to go through the sort of malarkey that you had if you want to do anything much more than searching the files.

Incidentally you can provide a #! line in the Python script, make it executable and then run it explicitly, instead of python .... Normally people use #!/usr/bin/env python, but that won't work until the new CF 3.14; instead just #!/mod/bin/python.
 
Top