[youtube-dl] Download files from youtube.com or other video platforms

Is it possible to download an entire BBC series with one request?
Or must each episode be individually downloaded?

I am attempting to download the Granddaughter's current favourite programme https://www.bbc.co.uk/iplayer/episodes/p06m8m6y

I put in the URL for the series into youtube-dl and it claimed to have downloaded a playlist but downloaded 0 videos
Code:
113    25/10/2018 19:53:02 - completed download of ? options
112    25/10/2018 19:53:02 - [download] Finished downloading playlist: p06m8m6y
111    25/10/2018 19:53:01 - [bbc.co.uk:iplayer:playlist] playlist p06m8m6y: Downloading 0 videos
110    25/10/2018 19:53:01 - [download] Downloading playlist: p06m8m6y
109    25/10/2018 19:53:00 - [bbc.co.uk:iplayer:playlist] p06m8m6y: Downloading webpage
108    25/10/2018 19:52:02 - --newline https://www.bbc.co.uk/iplayer/episodes/p06m8m6y?
107    25/10/2018 19:52:02 - Starting download URL https://www.bbc.co.uk/iplayer/episodes/p06m8m6y? Options  QID 20425
It says it downloaded a playlist but I cant find any corresponding file
It does not work with iPlayer. I did download a series from CBC (Windows version of youtube-dl): inputting the address for the main series page (where links to all episodes were present) downloaded all episodes with just one command line. In iPlayer you seem to need to input the URL of each episode individually. To simplify the process a bit you can open multiple abduco sessions and download several episodes at once. I have done this for up to four episodes at a time. It does not make it quicker as the download rate slows down with each episode you add, but you can at least set a few going and let it get on with it.
 
Is it possible to download an entire BBC series with one request?
Or must each episode be individually downloaded?

I am attempting to download the Granddaughter's current favourite programme https://www.bbc.co.uk/iplayer/episodes/p06m8m6y

I put in the URL for the series into youtube-dl and it claimed to have downloaded a playlist but downloaded 0 videos
...

Apparently youtube-dl's iplayer extractor supports asx playlists but obviously that isn't working here.

This might, though.
Code:
!/bin/sh
# scrape iplayer programme URLs from a BBC web page
                                                                                
# args: 1=iplayer_series_url
                                                                                
mung_url() { # prefix     
    local url
    while read url; do
        url=${url##href=\"};
        echo $1${url%%\"}   
    done                    
}                                                                               

# get BBC's base address                                                                                
bbc="$1"; bbc="${bbc%%/iplayer*}" 

# parse the web page for episode URLs, extract and prepare them for youtube-dl
# curl: -k insecure, needed due to Humax's old SSL libs, -s silent, -S show errors anyway
# grep: -o print matching substring, -E match extended regular expression
curl -k -s -S $1 | grep -oE "href=('|\")/iplayer/episode/[^'\"]+\\1" | mung_url $bbc
[corrected copy/paste error in setting BBC base address above]
If the above is saved as /mod/bin/iplayer_episodes and of course after making it executable
Code:
# chmod a+x /mod/bin/iplayer_episodes
then you could use it to create a playlist, or feed the resulting URLs to youtube-dl manually, or just go whole hog:
Code:
# abduco -f -c Catie /mod/bin/iplayer_episodes https://www.bbc.co.uk/iplayer/episodes/p06m8m6y | youtube -a -

HTH
 
Last edited:
A bit of googling seems to indicate that the BBC dropped support for playlists in JSON format in May requiring the use of screen scraping :(

Qtube makes it easy to queue the download requests and now this tool will save the need for individual cutting and pasting of episode URLs - I will try to incorporate it into the webif
 
The youtube-dl documentation indicates that ITV is supported. I tried it out but it seems that ITV are still serving Flash streams. The download failed, asking for the installation of rtmpdump. Would this be feasible with rtmpdump?

I tried downloading an ITV one and ended up with the following log in qtube ...

20:32:17 - starting download of https://www.itv.com/hub/joanna-lumleys-silk-road-adventure/2a5513a0003 options
20:33:06 - [ITV] 2a5513a0003: Downloading webpage
20:33:07 - [ITV] 2a5513a0003: Downloading XML
20:33:27 - [ITV] 2a5513a0003: Downloading JSON metadata
20:33:27 - [ITV] 2a5513a0003: Downloading m3u8 information
20:33:32 - [hlsnative] Downloading m3u8 manifest
20:33:33 - [download] Destination: /mnt/hd2/My Video/Series 1 - Episode 3.mp4
20:33:34 - 20:33:34 - 20:33:35 - Caught error: WARNING: Unable to download XML: HTTP Error 504: Gateway Time-out
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg
ffmpeg version 2.8 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 4.2.0 (GCC) 20070124 (prerelease) - BRCM 11ts-20090508
configuration: --cross-prefix=mipsel-linux- --enable-cross-compile --arch=mipsel --target-os=linux --enable-gpl --enable-libmp3lame --disable-static --enable-shared --disable-asm --prefix=/mod --extra-cflags='-fPIC -DDEBUG' --disable-doc --disable-mips32r2 --disable-mipsdspr2 --disable-mipsdspr1
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101 l
ibpostproc 53. 3.100 / 53. 3.100
https protocol not found, recompile FFmpeg with openssl, gnutls, or securetransport enabled.
https://itvpnpdotcom-a.akamaihd.net...2_18_1_VAR012-audio=128000-video=1438350.m3u8: Protocol not found
ERROR: ffmpeg exited with code 1

It looks like it should download the MP4 file, surely that is what I need?
 
ITV doesn't work (on the Humax). It does on a PC. See earlier in the thread.

Indeed, the output from the command-line is more helpful.
Code:
humax# youtube https://www.itv.com/hub/joanna-lumleys-silk-road-adventure/2a5513a0003
[ITV] 2a5513a0003: Downloading webpage
[ITV] 2a5513a0003: Downloading XML
WARNING: Unable to download XML: HTTP Error 504: Gateway Time-out
[ITV] 2a5513a0003: Downloading JSON metadata
[ITV] 2a5513a0003: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg
[download] Destination: /media/drive1/Video/Series 1 - Episode 3.mp4
ffmpeg version 2.8 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.2.0 (GCC) 20070124 (prerelease) - BRCM 11ts-20090508
  configuration: --cross-prefix=mipsel-linux- --enable-cross-compile --arch=mipsel --target-os=linux --enable-gpl --enable-libmp3lame --disable-static --enable-shared --disable-asm --prefix=/mod --extra-cflags='-fPIC -DDEBUG' --disable-doc --disable-mips32r2 --disable-mipsdspr2 --disable-mipsdspr1
...
https protocol not found, recompile FFmpeg with openssl, gnutls,
or securetransport enabled.
https://itvpnpdotcom-a.akamaihd.net/2-5513-0003-002/18/1/VAR012/2-5513-0003-002_18_1_VAR012.ism/2-5513-0003-002_18_1_VAR012-audio=128000-video=1438350.m3u8: Protocol not found


ERROR: ffmpeg exited with code 1
humax#
To save people going back 5 or so pages in the thread, the issue is that ITV Hub, and many other sites, expect up-to-date https: protocol support, whereas
  • the Humax HDx Fox-T2, even with CF3.13, has quite an old version of the OpenSSL library to support https:;
  • anyway the CF Python package isn't built with https: support, so, when the youtube-dl Python program determines that https: is needed ("features it does not support"), it tries to invoke ffmpeg to download the stream;
  • boo-hoo, the CF ffmpeg package isn't built with https: support either, as it clearly reports.
When I commented on this originally, I said that I was trying to rebuild ffmpeg against LibreSSL (a simpler alternative to OpenSSL, which I had already built). This effort reached an apparently impassable configuration and dependency swamp, to the point that I wondered whether a rebuild of Python might be easier. And the answer appears to be yes.

Python 2.7.15 compiles but seems to be much larger than the CF version and segfaults when run. Perhaps the config.status file from the CF build would clarify what's going wrong - is that something that @af123 would have?
 
Indeed, the output from the command-line is more helpful.

Ah, thanks for the analysis. I (wrongly, obviously) assumed that qtube was reporting everything from youtube-dl to the screen, but obviously it didn't quite.

I'll see how I get on doing a PC download.
 
Right, using the Windows version looks like it has downloaded it, but I get the following: -

Code:
C:\Datasheets\Humax>youtube-dl https://www.itv.com/hub/joanna-lumleys-silk-road-adventure/2a5513a0003
[ITV] 2a5513a0003: Downloading webpage
[ITV] 2a5513a0003: Downloading XML
WARNING: Unable to download XML: HTTP Error 504: Gateway Time-out
[ITV] 2a5513a0003: Downloading JSON metadata
[ITV] 2a5513a0003: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 234
[download] Destination: Series 1 - Episode 3-2a5513a0003.mp4
[download] 100% of 533.75MiB in 17:25
WARNING: 2a5513a0003: malformed AAC bitstream detected. Install ffmpeg or avconv to fix this automatically.

Does this mean I need to do something to it to reform the bitstream when I load it on the Hummy? It seems to play fine on the PC.
 
malformed AAC bitstream detected.
That occurs normally using youtube-dl on the Humax and ffmpeg is run under the covers to do the fixing, since it automatically runs I have no idea what the impact of not doing the correction would be - possibly none - try it and see.

If you are going to do this regularly install ffmpeg on the PC so that it can do the autofixing on the PC/
 
If you are going to do this regularly install ffmpeg on the PC so that it can do the autofixing on the PC/

You need to put ffmpeg.exe, ffplay.exe and ffprobe.exe in the same directory as youtube-dl.exe. It will then automatically tidy up the downloaded file.
Without it the mp4 seems to judder and break up when using trick play (well it does when I use Windows Media Player).
 
That occurs normally using youtube-dl on the Humax and ffmpeg is run under the covers to do the fixing, since it automatically runs I have no idea what the impact of not doing the correction would be - possibly none - try it and see.

If you are going to do this regularly install ffmpeg on the PC so that it can do the autofixing on the PC/
The mp4 format uses headers and an audio stream doesn't. The audio still has timestamps though and should play correctly. The initial download also has the streams in the wrong order (audio = stream 0, video = stream 1). Ffmpeg fixes the order and header, making the mp4 file fully compliant with the mp4 standards.
The easiest way to install ffmpeg on Windows is to download a standalone build: search for Zeranoe ffmpeg.
 
You need to put ffmpeg.exe, ffplay.exe and ffprobe.exe in the same directory as youtube-dl.exe. It will then automatically tidy up the downloaded file.
Without it the mp4 seems to judder and break up when using trick play (well it does when I use Windows Media Player).
The standard way to install ffmpeg on Windows is to place the binaries in C:\ffmpeg\bin and then add ffmpeg to the system path. The steps are outlined here.
 
Thanks for the help everyone. I can report that the ITV program downloaded on the PC played quite happily on the Hummy without needing to have its audio manipulated by ffmpeg.

Many thanks to those who set up this tool on the Hummy as well - and for the qtube front end. I still have a BBC program to watch that I downloaded with the Hummy version.

A very useful tool to have around when an episode is missed.
 
I have noticed that if you download a programme twice with youtube-dl it skips the actual download because it already exists but still reports the 'malformed AAC bitstream detected. ' and runs ffmpeg again so it is not obvious whether it has actually fixed anything
 
I downloaded a file from the iplayer, ended up with two similarly named files, one had temp in the name.

I played the normal file. I stopped it. It took about ten seconds to stop. I then tried to play it again it gave me the option to resume and there it hangs. Forever. The t2 is still fine underneath all that, it's not crashed. What do you think is happening?
 
The download-and-fixup process has not completed properly - the temp file should self-destruct on completion. Try downloading it again.
 
I have downloaded again, it said already downloaded, then fixing audio forever. I left it overnight. This morning the temp file is gone. Played the file given option to resume slow to act, played, then I stopped it slow to act again. When I played again from resume it stopped responding.

Shall I just delete everything and start again?

Also, how do you know that the download AND all processes have finished in the cmd line?
 
Back
Top