OP
prpr
Well-Known Member
That just gives me an unexpected "(" type of error (same on my Linux box).
[youtube-dl] Download files from youtube.com or other video platforms
- Thread starter prpr
- Start date
MymsMan
Ad detector
It was copied and pasted from the jim source code and includes the escaping of braces and quotes etc needed to survive the interpreter,
It is a long time since I last touched that bit of code but it does work - I must have cribbed it from somewhere
Code:
07/08/2018 17:59:48 NS(22665)- Wrote 67818 entries to /media/My Video/Chase/The Chase_20180807_1659-crop. Stripped 1196345 packets (224314k) of EPG data
Black Hole
May contain traces of nut
Here is another example, this time of a ~400MB download that took maybe half an hour:
The clusters of activity are when I started it off (and watched for a while), then 20 mins later when I checked back to see what was going on, having closed WebShell in the mean time. I closed it again and returned maybe half an hour later to discover the process had completed, but it looks like the download completed during the second burst.
I think it is reasonable to suspect the very low data rate points between the bursts are red herrings. Bearing in mind the x-axis is derived, it could be that the first few packets transferred during the second burst were regarded as at a slow average rate (looks like it averages the time for the last 8 packets), and those points actually occurred at the beginning of the second burst and not in between bursts as plotted. I might be able to correct that with some deconvolution, but that would be a distraction from what I am supposed to be doing.
Does anyone else see these kinds of effect? My next test will be to see whether WebShell is to blame by running the same in Telnet.
The clusters of activity are when I started it off (and watched for a while), then 20 mins later when I checked back to see what was going on, having closed WebShell in the mean time. I closed it again and returned maybe half an hour later to discover the process had completed, but it looks like the download completed during the second burst.
I think it is reasonable to suspect the very low data rate points between the bursts are red herrings. Bearing in mind the x-axis is derived, it could be that the first few packets transferred during the second burst were regarded as at a slow average rate (looks like it averages the time for the last 8 packets), and those points actually occurred at the beginning of the second burst and not in between bursts as plotted. I might be able to correct that with some deconvolution, but that would be a distraction from what I am supposed to be doing.
Does anyone else see these kinds of effect? My next test will be to see whether WebShell is to blame by running the same in Telnet.
OP
prpr
Well-Known Member
Showing your age there!
(Once I'd got annoyed with the limitations on Basic, I taught myself Fortran, aged about 16, from the manual, at the local Poly where the school 'borrowed' the computing facilities. That didn't last too long as it was also quite an annoying language. Then on via varying machine code/assemblers to C and C++.)
I typed "youtube-dl how to use" into Google and it came up with this:
https://linuxconfig.org/download-video-from-the-command-line-with-youtube-dl
You can ignore most of it to do with installation (as that is handled for you by the WebIf) and skip to the Downloading Video bit. You just need to go to a command prompt (telnet, ssh, Webshell) and do something like this:
Code:
humax# youtube https://www.youtube.com/watch?v=yVpbFMhOAwE
MontysEvilTwin
Well-Known Member
The best resource I have found is the youtube-dl documentation. There is a link to it in the first post of this thread. This is straightforward but has a lot of information: it is not an 'idiots' guide. However, what do you want to do? With the default configuration, if you wanted to download an iPlayer programme, for example, all you have to to is find the link to the programme on the iPlayer website, copy it, open a telnet console and connect to the HDR-FOX, select the 'CLI' option to get the command line then type 'youtube' leave a space, paste a link and hit return. This will download the recording in good (if not full HD) quality. If you want other options, e.g. to download the subtitles, you can look in the documentation to find the extra commands you need.
One other thing I would do is to make sure you have the abduco package installed, then before starting the download, type the following at the command line in telnet:
abduco -A x. This will make the download run on the HDR-FOX alone; if your telnet connection drops, the download will still proceed.
OP
prpr
Well-Known Member
Just to make it clear, you would type something like:
Code:
humax# abduco -A x youtube https://www.youtube.com/watch?v=yVpbFMhOAwEMontysEvilTwin
Well-Known Member
Black Hole
May contain traces of nut
OP
prpr
Well-Known Member
But it's so easy, especially with Webshell...
1. Install the Webshell package
2. Reboot
3. Go to the WebIf's Diagnostics page and select "Command Line" from the Utilities section
4. Enter the PIN (probably 0000)
5. Enter "cli"
6. Type "youtube http://whatever-your-link-is"
7. When it's finished type "exit" and then option "x" to close the Webshell
Any idiot can follow this, surely? Even an "idiot" like you. It's only typing. And you seem to be able to do that OK in the forum!
I think you severely underestimate your capabilities.
MontysEvilTwin
Well-Known Member
There is a problem downloading from the ITV hub. After the 22nd August update to youtube-dl this is what I get on a Windows installation:
The download was successful using the 'hlsnative' downloader. However, the download fails on the HDR-FOX with the same version of youtube-dl:
The program tries to download using the 'hlsnative' downloader but this fails: 'WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg'. Downloading with ffmpeg also fails, as expected, since the version of ffmpeg as compiled for the HDR-FOX does not support https downloads.
Although on different platforms with different versions of python (and ffmpeg) installed, these alternate versions are supported by youtube-dl. I don't know why the hlsnative downloader works on the Windows version but not on the HDR-FOX. There is debug information in the console output; a large chunk pertaining to python, but I don't know enough about python to debug it myself.
Code:
C:\Downloads>youtube-dl https://www.itv.com/hub/itv-news-weekend-teatime/2a4546a0275 --verbose
[debug] System config: []
[debug] User config: ['--prefer-ffmpeg', '-f', 'best[height<=?1080]', '--geo-bypass', '--write-sub', '--sub-format', 'srt', '--convert-subs', 'srt']
[debug] Custom config: []
[debug] Command-line args: ['https://www.itv.com/hub/itv-news-weekend-teatime/2a4546a0275', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2018.08.22
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134
[debug] exe versions: ffmpeg 4.0, ffprobe 4.0
[debug] Proxy map: {}
[debug] Using fake IP 25.33.190.59 (GB) as X-Forwarded-For.
[ITV] 2a4546a0275: Downloading webpage
[ITV] 2a4546a0275: Downloading XML
WARNING: Unable to download XML: HTTP Error 503: Service Unavailable
[ITV] 2a4546a0275: Downloading JSON metadata
[ITV] 2a4546a0275: Downloading m3u8 information
[debug] Invoking downloader on 'https://itvpnpdotcom-a.akamaihd.net/2-4546-0275-001/18/1/VAR012/2-4546-0275-001_18_1_VAR012.ism/2-4546-0275-001_18_1_VAR012-audio=96041-video=1516000.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 75
[download] Destination: ITV News Weekend Teatime - Saturday 25 Aug 6pm-2a4546a0275.mp4
[download] 100% of 180.28MiB in 06:40
[debug] ffmpeg command line: ffprobe -show_streams "file:ITV News Weekend Teatime - Saturday 25 Aug 6pm-2a4546a0275.mp4"
[ffmpeg] Fixing malformed AAC bitstream in "ITV News Weekend Teatime - Saturday 25 Aug 6pm-2a4546a0275.mp4"
[debug] ffmpeg command line: ffmpeg -y -i "file:ITV News Weekend Teatime - Saturday 25 Aug 6pm-2a4546a0275.mp4" -c copy -f mp4 "-bsf:a" aac_adtstoasc "file:ITV News Weekend Teatime - Saturday 25 Aug 6pm-2a4546a0275.temp.mp4"
[ffmpeg] There aren't any subtitles to convert
C:\Downloads>
Code:
Lounge# youtube https://www.itv.com/hub/itv-news-weekend-teatime/2a4546a0275 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: [u'--prefer-ffmpeg', u'-f', u'best[height<=?1080]', u'--write-sub', u'--sub-format', u'srt', u'--convert-subs', u'srt']
[debug] Command-line args: [u'--config-location', u'/mod/etc/youtube-dl.conf', u'-o', u'/mnt/hd2/My Video/%(title)s.%(ext)s', u'https://www.itv.com/hub/itv-news-weekend-teatime/2a4546a0275', u'--verbose']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2018.08.22
[debug] Python version 2.7.1 (CPython) - Linux-2.6.18-7.1-7405b0-smp-with-libc0
[debug] exe versions: ffmpeg 2.8, ffprobe 2.8
[debug] Proxy map: {}
[debug] Using fake IP 25.24.106.147 (GB) as X-Forwarded-For.
[ITV] 2a4546a0275: Downloading webpage
[ITV] 2a4546a0275: Downloading XML
WARNING: Unable to download XML: HTTP Error 503: Service Unavailable
[ITV] 2a4546a0275: Downloading JSON metadata
[ITV] 2a4546a0275: Downloading m3u8 information
[debug] Invoking downloader on u'https://itvpnpdotcom-a.akamaihd.net/2-4546-0275-001/18/1/VAR012/2-4546-0275-001_18_1_VAR012.ism/2-4546-0275-001_18_1_VAR012-audio=96041-video=1516000.m3u8'
[hlsnative] Downloading m3u8 manifest
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg
[download] Destination: /mnt/hd2/My Video/ITV News Weekend Teatime - Saturday 25 Aug 6pm.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
X-Forwarded-For: 25.24.106.147
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Cookie: test_cookie=allowed; hdntl=exp=1535316094~acl=%2f*~data=empty~hmac=18d4630658f0a89c356cd6c8003e1c86f0c71d2613ee909da58bf1c73a6e4bb6
' -i 'https://itvpnpdotcom-a.akamaihd.net/2-4546-0275-001/18/1/VAR012/2-4546-0275-001_18_1_VAR012.ism/2-4546-0275-001_18_1_VAR012-audio=96041-video=1516000.m3u8' -c copy -f mp4 '-bsf:a' aac_adtstoasc 'file:/mnt/hd2/My Video/ITV News Weekend Teatime - Saturday 25 Aug 6pm.mp4.part'
ffmpeg version 2.8 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 4.2.0 (GCC) 20070124 (prerelease) - BRCM 11ts-20090508
configuration: --cross-prefix=mipsel-linux- --enable-cross-compile --arch=mipsel --target-os=linux --enable-gpl --enable-libmp3lame --disable-static --enable-shared --disable-asm --prefix=/mod --extra-cflags='-fPIC -DDEBUG' --disable-doc --disable-mips32r2 --disable-mipsdspr2 --disable-mipsdspr1
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
https protocol not found, recompile FFmpeg with openssl, gnutls,
or securetransport enabled.
https://itvpnpdotcom-a.akamaihd.net/2-4546-0275-001/18/1/VAR012/2-4546-0275-001_18_1_VAR012.ism/2-4546-0275-001_18_1_VAR012-audio=96041-video=1516000.m3u8: Protocol not found
ERROR: ffmpeg exited with code 1
File "/mod/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/mod/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/mod/bin/youtube-dl/__main__.py", line 19, in <module>
youtube_dl.main()
File "/mod/bin/youtube-dl/youtube_dl/__init__.py", line 472, in main
_real_main(argv)
File "/mod/bin/youtube-dl/youtube_dl/__init__.py", line 462, in _real_main
retcode = ydl.download(all_urls)
File "/mod/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2001, in download
url, force_generic_extractor=self.params.get('force_generic_extractor', False))
File "/mod/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 803, in extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/mod/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 857, in process_ie_result
return self.process_video_result(ie_result, download=download)
File "/mod/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1635, in process_video_result
self.process_info(new_info)
File "/mod/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1908, in process_info
success = dl(filename, info_dict)
File "/mod/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1847, in dl
return fd.download(name, info)
File "/mod/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
return self.real_download(filename, info_dict)
File "/mod/bin/youtube-dl/youtube_dl/downloader/hls.py", line 76, in real_download
return fd.real_download(filename, info_dict)
File "/mod/bin/youtube-dl/youtube_dl/downloader/external.py", line 64, in real_download
self.get_basename(), retval))
File "/mod/bin/youtube-dl/youtube_dl/downloader/common.py", line 165, in report_error
self.ydl.report_error(*args, **kargs)
File "/mod/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 620, in report_error
self.trouble(error_message, tb)
File "/mod/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 582, in trouble
tb_data = traceback.format_list(traceback.extract_stack())
Lounge#Although on different platforms with different versions of python (and ffmpeg) installed, these alternate versions are supported by youtube-dl. I don't know why the hlsnative downloader works on the Windows version but not on the HDR-FOX. There is debug information in the console output; a large chunk pertaining to python, but I don't know enough about python to debug it myself.
Black Hole
May contain traces of nut
I'm in the process of downloading a Proms concert from BBC FOUR, but I have bypassed the HomePlugs and not seeing the same stops and starts (although I forgot to capture the progress report).
If the ffmpeg downloader needs newer versions of TLS than supported by openSSL-0.9.8r from the package repo, then it's a fair guess that Python does too. That could be what youtube-dl means when it reports "hlsnative has detected features it does not support, extraction will be delegated to ffmpeg".
I made reasonable progress with rebuilding ffmpeg-3.4.2 (builds, liable to hang/crash) and wget-1.19.4 (builds, runs) against libreSSL-2.6.4 to address this but I'm not with the relevant HDR now. libreSSL because openSSL's build is a nightmare. On reflection perhaps Python should have been rebuilt, but I imagine that even ffmpeg is simpler.
Code:
# ls -l HDR/mod/lib/ | grep "^[^ld]"
...
-rw-r--r-- 1 root root 9985520 Jul 12 19:04 libcrypto.a
-rw-r--r-- 1 root root 929 Jul 12 19:04 libcrypto.la
-rw-r--r-- 1 root root 6439216 Jul 12 19:04 libcrypto.so.42.0.0
...
-rw-r--r-- 1 root root 2245424 Jul 12 19:20 libssl.a
-rw-r--r-- 1 root root 930 Jul 12 19:20 libssl.la
-rw-r--r-- 1 root root 1441744 Jul 12 19:20 libssl.so.44.0.1
...
-rw-r--r-- 1 root root 454672 Jul 12 19:24 libtls.a
-rw-r--r-- 1 root root 949 Jul 12 19:24 libtls.la
-rw-r--r-- 1 root root 307858 Jul 12 19:24 libtls.so.16.0.1
...
# ls -le HDR/mod/bin/ | grep "^[^ld].* 2018 .*"
...
-rwxr-xr-x 1 root root 80730 Thu Jul 12 19:24:58 2018 ocspcheck
-rwxr-xr-x 1 root root 1422133 Thu Jul 12 19:38:18 2018 openssl
...
-rwxr-xr-x 1 root root 1029876 Thu Jul 12 17:02:07 2018 wget
...MontysEvilTwin
Well-Known Member
In the first instance it would be good to understand why the hlsnative downloader fails. You may be right about python, but without an update to the version of python on the HDR-FOX it is hard to test. Is there anything useful in the debug output?
I expected ffmpeg (strictly version 2.8, but called 2.8.1 because of an applied patch) to fail since the variant compiled for the HDR-FOX MIPS platform is quite stripped down and does not have the features required to support https downloads. Recompiling with the extra features may be enough. Who knows?
The Python backtrace just repeats what we already knew, that yt-dl's HLS module punted to the external downloader and that failed.
My expectation was that the repo versions of Python and ffmpeg must have been built against the OpenSSL version that is provided with them. Not really so.
As you point out, ffmpeg is actually built against no version of OpenSSL.
python has a module ssl.py apparently linked (_ssl.so) against OpenSSL 1.0.0 but that only supports https, not general encryption. The system (at least on the HD I'm examining) has OpenSSL 1.0.0 in /usr/lib, presumably as part of the Opera installation used by the Humax settop binary, but perhaps python can use it as well.
A bit of digging in yt-dl's GitHub reveals that the hlsnative error message relates to the absence of support for encryption. Indeed, the repo Python doesn't have either of the packages (Crypto, pycrypto) that might do so. Sadly it doesn't support the documented ways of acquiring them (no doubt installation will involve building an interface library similar to the existing _ssl.so).
I still don't understand how to choose a particular file produced by the -F switch
I don't get this:
The -f switch is how you specify a particular download from the list produced by the -F switch. "options" needs to be enclosed in quotes and not contain any spaces.
"best[height=540][fps=25][ext=mp4]" works for me,
how do I choose this?
stream-uk-iptv_streaming_concrete_combined_hd_mf_limelight_uk_hls_https-5714 mp4 1280x720 5714k , avc1.640020@5070k, 50.0fps, mp4a.40.2@320k (best)
I don't get this:
The -f switch is how you specify a particular download from the list produced by the -F switch. "options" needs to be enclosed in quotes and not contain any spaces.
"best[height=540][fps=25][ext=mp4]" works for me,
how do I choose this?
stream-uk-iptv_streaming_concrete_combined_hd_mf_limelight_uk_hls_https-5714 mp4 1280x720 5714k , avc1.640020@5070k, 50.0fps, mp4a.40.2@320k (best)
Black Hole
May contain traces of nut
youtube -f "best[height=720][fps=50][ext=mp4]" <url>(replace "<url>" with whatever the URL is for your download)
Surely you can see how this relates to the listing in your post? The first field (in the listing entry) identifies the type of stream, followed by the container type, screen resolution in pixels (width x height), bit rate, video codec, frame rate, audio codec. So all I have done is substitute your entries into my template.
Or maybe simply:
youtube -f "best[ext=mp4]" <url>If you specify "best", you get whatever stream has the greatest bit rate (in your case 5714kbps). But you want to make sure you get an MP4 out of it, so "best[ext=mp4]" means you get the MP4 download with the greatest bit rate. I suspect the -f flag stands for "filter", applying the specified criteria to find the particular download you want. In my case, because I have a slow broadband and don't want to use up much disk space with large files, I specify [height=540][fps=25] to get a download which is roughly StDef in quality rather than HiDef (which is what you seem to want).
If you are happy to have the largest possible download (and hope that is MP4), all you need on your command line is:
youtube <url>...because it defaults to the greatest bit rate anyway.
Anyone wanting to use youtube-dl on an HD-FOX (rather than an HDR-FOX) needs to read the comments in my guide HERE (click), because when large downloads are subsequently automatically cleaned up by passing them through ffmpeg, the HD-FOX can run out of memory.
Last edited:
Of course I could see the relation. But there were several files with the attributes you put in your answer to me above. How do I know which one those attributes will download.?
Black Hole
May contain traces of nut
How do I know unless you show me all the matches? You only showed one of them.