"&" does cause the process to persist even when the telnet session is closed.
To bring a background process to the foreground use 'fg %n' where 'n' is the job number.
e.g.
will start a new shell in a session called "x", or re-attach if the session already exists. You can even attach to the same session from multiple places at once.
Interesting. I've had a quick glance at the abduco documentation, and checked to see that the command is available even though it is not listed as a package.
What is the need to start a new shell in the abduco session? Is there no command line interpretation without a shell running?
I'm playing with youtube-dl to grab the first episode of Flatpack Empire. -f "worst" gave me a horrible 70MB file.
This is grabbing an 830MB file, which is coming in on my pittance of a broadband connection at roughly real-time, so that feels like what I would get if I used the iPlayer app and saved the stream.
My suggestion to limit the frames per second was intended to enable programmes to be downloaded at 1280 x 720 @ 25 fps (like the iPlayer app used to do when set to 'HD', before this changed to 960 x 540 @ 50 fps). However, it seems that the 1280 x 720 @ 25 fps profile is unavailable as a HLS stream (used by youtube-dl), so limiting the frames per second probably gave you 960 x 540 @ 25 fps: in webif > browse media files, if you click on the title of the mp4 file it will tell you exactly what you have got.
By default, 1280 x 720 @ 50 fps will be downloaded (if available). If you want 960 x 540 @ 25 fps, limiting the frame rate should be enough. If you want 960 x 540 @ 50 fps (like the iPlayer app) setting the height parameter to '<=540' should work.
In the absence of a secondary preferences file, I have been adding extra parameters to '/mod/etc/youtube-dl.conf'. This works but the file gets overwritten every time the package is updated. Would it be possible to configure updates to not overwrite modified configuration files? Other packages already do this, for example, a modified 'smb.conf' does not get overwritten, the default configuration file is saved as 'smb.conf-opkg' instead.
It is available for most programmes, but it requires filtering by bit rate ([tbr<3000]) because youtube-dl doesn't pick up the frame rate. Be aware that the 1280 x 720 (and 832 x 468) @ 25 fps streams have 96k audio. If you have used get_iplayer, this is the default stream it downloads.
If you want 320k audio in BBC programmes, restrict youtube-dl to HLS streams ([protocol=m3u8_native]). 128k audio is the best available with DASH from iPlayer. 320k audio appears to be available for HLS streams with video width>=448 However, this does not apply to the 1280 x 720 (or 832 x 468) @ 25 fps streams.
Post #39 says it is possible but does not seem to recommend it. My suggestion keeps everything in one place and is in line with the behaviour of other packages. Is prpr packaging and uploading the updates? I suppose he will decide if my suggestion is worth implementing.
It is available for most programmes, but it requires filtering by bit rate ([tbr<3000]) because youtube-dl doesn't pick up the frame rate. Be aware that the 1280 x 720 (and 832 x 468) @ 25 fps streams have 96k audio. If you have used get_iplayer, this is the default stream it downloads.
If you want 320k audio in BBC programmes, restrict youtube-dl to HLS streams ([protocol=m3u8_native]). 128k audio is the best available with DASH from iPlayer. 320k audio appears to be available for HLS streams with video width>=448 However, this does not apply to the 1280 x 720 (or 832 x 468) @ 25 fps streams.
I had used the '--list-formats' option to see which streams are available to youtube-dl. I had assumed that a 1280 x 720/ 25 fps stream was not available as it wasn't listed using this command.
I had used the '--list-formats' option to see which streams are available to youtube-dl. I had assumed that a 1280 x 720/ 25 fps stream was not available as it wasn't listed using this command.
I rarely find it missing, but perhaps it wasn't available for whatever programme you were downloading. If it is always missing from all BBC programmes, that would be unusual unless you are falling foul of some sort of geoblocking, but that seems unlikely given that you apparently can access other streams.
I created the config file with my extra options in '/mod/.config/youtube-dl/' but it is ignored. The file is called 'config' with no file extension and Unix line terminators. Has anyone got this working?
I created the config file with my extra options in '/mod/.config/youtube-dl/' but it is ignored. The file is called 'config' with no file extension and Unix line terminators. Has anyone got this working?
So if extra parameters are added to ''/mod/etc/youtube-dl.conf'' they will taken into account but will be deleted when the package is updated, and if "/mod/.config/youtube-dl/config" is used it will be ignored? Oh yes, I see post #39 has now been updated, which answers my second query.
I created the config file with my extra options in '/mod/.config/youtube-dl/' but it is ignored. The file is called 'config' with no file extension and Unix line terminators. Has anyone got this working?
I haven't installed prpr's package but the raw package installed as I originally described (and see my #18 in this thread) definitely reads options from ~/.config/youtube-dl/config = /mod/.config/youtube-dl/config.
If the packaging didn't alter this, your approach should have worked. Perhaps you need to quote the option values?
So if extra parameters are added to ''/mod/etc/youtube-dl.conf'' they will taken into account but will be deleted when the package is updated, and if "/mod/.config/youtube-dl/config" is used it will be ignored?
I've been looking at downloading with youtube-dl from different sites. One problem I have come across is that if the built in HLS downloader does not like the stream, the program will try and use ffmpeg which can fail with the following error message: "https protocol not found, recompile FFmpeg with openssl, gnutls, or securetransport enabled." af123 - would it be possible for you to compile a beta version of ffmpeg with https support? There is more information here.
Black Hole's Summary Guide to Using youtube-dl and qtube
youtube-dl provides an alternative to the TV Portal iPlayer and YouTube apps for downloading "streamed" content, particularly with the decreasing functionality of the TV Portal. It might also extract media from other sites besides iPlayer and YouTube.
youtube-dl is a command-line utility. For those of a nervous disposition, the qtube package provides a GUI add-on to the WebIF as a front-end to youtube-dl - see heading "qtube" (below). However, in order to have full control of qtube, the user still needs to be aware of the options available in the youtube-dl package, so a glance through the info under heading "youtube-dl" is advisable.
...will result in no response for a considerable period, then status reports while the youtube-dl program reads the listings from the programme web page to detect the best quality media file not greater than 1080 lines (see below), then the download starts. <URL> is the web address for the programme you want to download: find the programme on https://www.bbc.co.uk/iplayer, copy the URL from your browser address bar, and paste it in as above.
The resulting video file is located in (HDR-FOX) Media >> Storage (blue) >> HDD >> My Video or (HD-FOX) Media >> Storage (blue) >> USB-1* >> Video. Note that Media >> Media (yellow) >> Video must be selected (this is the default). * Your recording drive (HD-FOX) might be called something else in the USB drive list.
If the command stalls or you have to terminate it for any reason, issuing the same command again at a later time will resume where the last left off (providing you don't delete any intermediate files in the mean time).
A problem with the above simplistic approach is that the best possible quality file available will probably be in HiDef and a huge download - okay if you have a fast Internet connection and don't mind how much data you download or how much disk space it takes up, but there are other options. Another problem is that you might end up with a download that is incompatible with the 'Fox and invisible in the media list (eg a .webm file).
Code:
# youtube -F <URL>
Instead of downloading the programme, the -F switch produces a manifest (catalogue) of the video files available from the source, from which you can choose the one that suits you. The first field in each listing is a unique format code. The remaining fields in each listing specify the container format, resolution, bit rate of the encoded data, video codec, frame rate, and audio codec. When a group of listings all have the same details except for the first field, they all refer to the same resulting download (just different download mechanisms).
Code:
# youtube -f <options> <URL>
The -f (filter) switch is how you specify a particular download from the list produced by the -F (files) switch. To select a specific line from the -F catalogue, just use youtube -f <format_code> <URL>. Take care to ensure the listing includes video and audio (if that's what you want), and uses suitable codecs.
NB: In some instances the format code is just a short number (like "192", for example). BBC iPlayer (in particular, but not necessarily exclusively) uses a long descriptive string, which makes things less convenient (copy-and-paste required!).
Alternatively, <options> can specify a series of requirements and then youtube-dl will choose the best match for itself. <options> may need to be enclosed in quotes (to ensure the string is parsed correctly), and must not contain any spaces. The file downloaded is the best one (highest bit rate) of those which match your options (assuming any do).
For example: <options> = "best[height=540][fps=25][ext=mp4]" works for me, ensuring I get an StDef quality (960x540 @ 25fps) MP4 file which takes about the same space on disk as a normal StDef recording. "best" is probably superfluous, but I'm a belt-and-braces type.
The -f "best[height<=?1080]" switch is included in the default settings file for youtube-dl. This will download the best available version as long as the video is not more than 1080 lines.
Once downloaded, the data packets are stitched together and then any post-processing applied (iPlayer downloads seem to need audio stream correction through ffmpeg). See notes below if using an HD-FOX.
If, like me, you want to download MP3s of radio programmes for use in the car, you will find that (a) downloads are moving over to MP4 Audio (.m4a), and (b) bbc.co.uk/sounds won't co-operate with youtube-dl.
The BBC Sounds problem can be side-stepped by obtaining the equivalent BBC iPlayer URL. The BBC identify their programmes on the website using an arbitrary code string (eg "p08023tz"), which is often the last part of the URL (after the final "/"). By prefixing the programme code string with "https://www.bbc.co.uk/programmes/", it is often possible to access the same programme via iPlayer, and thus for it to become downloadable.
The MP4 Audio problem can be overcome using format conversion within youtube-dl. The command switch -x --audio-format mp3 tells youtube-dl to convert the file to MP3 after download (but takes an awfully long time with the limited processing power of the HDR-FOX). Thus the full command line to download programme "p08023tz" and convert to MP3 would be:
If using qtube (see below), just put "https://www.bbc.co.uk/programmes/p08023tz" in the "URL" box and "-x --audio-format mp3" in the "Process options" box.
Note that the same switch can be used to produce an MP3 soundtrack-only file from a video download (if you happened to want that – it's useful for extracting a music track from YouTube without the obligatory accompanying video).
Grabbing a Series of Programmes
If the URL is a page listing a number of programmes instead of just one, eg the master page for a series with links to the individual programmes within the series, youtube-dl will download all the media it can find.
abduco
A problem with running a long process from a Telnet session or the webshell package (a command terminal available as a web page - access via WebIF >> Diagnostics >> Command Line) is that the session inconveniently drops out and terminates any active processes if you take the focus away to do something else. Then you have to reconnect and restart the youtube-dl command, which then has to do its initial thinking before the download resumes...
The command abduco is already available to support other long processes such as fix-disk, and can be used here to create a protected command session which carries on regardless of the terminal session dropping out. So, before launching the main download, use the following to create a protected session:
Code:
# abduco -A yt
This creates a protected session called "yt" (call it what you like), and the next command prompt is within that session. Then start your youtube-dl process as described above. You can now "do other stuff" and the session will carry on regardless - to break out of the session and do other things on the command line, use Ctrl+\.
To come back later, use the same abduco line again. This time, as there already exists a session called "yt", the command connects to it rather than starting a new one.
To close the session (from within the session), type "exit".
To inspect open abduco sessions, type "abduco" (with no other parameters).
youtube-dl + abduco
Running a process within an abduco session leads to another problem: you can't monitor status, or recover status afterwards, if you are not connected to the session at the time. If the download process completes or crashes while the session is not active, and you then reconnect to the session later, all you see is a command prompt with no information what happened.
A solution is to copy all status output to a file where you can look at it, even while it is being accumulated (but note that the download progress status line continually re-writes, so won't appear in the file until it completes):
Code:
# youtube -f "options" <URL> | tee /mod/tmp/youtube.log
directs status output to the file youtube.log, which can be accessed via WebIF >> Diagnostics >> View Log Files >> youtube.log.
Unless you are seriously welded to the command line, it's simpler to use qtube (see below) – which takes care of running the processes in the background and capturing status to a log file.
Large Files, and/or Running On HD-FOX
There is a specific problem with the HD-FOX when it comes to the ffmpeg stage of the process to "fix" the downloaded stream suitable for playback - that is it can easily run out of memory for downloads ~1GB+. The solution is to install the swapper package, which engages the OS's ability to page RAM into virtual memory on the HDD/external drive.
The HDR-FOX runs out of steam around 6GB, even with swapper, we assume because there comes a point where the system spends more processing time paging memory than performing the task, and the task is competing with the normal processing as a PVR. ffmpeg is not optimised for target systems with such minimal resources, and the HD-/HDR-FOX is only endowed with sufficient hardware to be a PVR!
If you really must download that multi-gigabyte movie using your HD-/HDR-FOX rather than a "real" computer, the easiest work-around is to disable youtube-dl's fix-up pass:
So far I've found the easiest option is to bypass fixup processing using --fixup warn or --fixup never for youtube-dl. The result will be an MP4 without cue/review functions.
If you require fixup (so that cue/review on MP4 will work) then try the following - at the expense of extra 1-2 Hours processing and HDR crawling to glacier speed, losing audio & video output to HDMI and eventually requiring reboot(s).
I used these options in /mod/etc/youtube-dl.conf
Better resolutions are available, but I don't know how well the HDR will cope. -f "best[height<=?576][fps<=?60]" (to keep file size down) -o "/media/drive1/dir1/%(title)s.%(ext)s" (store file out of DLNA indexing reach, easy access via Humax UI eg Media-Video/Storage(blue)/USB)
Worked on
Eastbourne - uninterrupted day 4
Requires 400MB swapfile
Result 5.65GB mp4
Eastbourne - uninterrupted day 5
Requires 625MB swapfile
Result 9.78GB mp4
Note that this only applies to HDR-FOX, and "/media/drive1" means the virtual USB storage available when the virtual-disk2 package is installed. If you have external storage plugged in then the relevant path string might be something different. Swapfile size is 256MiB by default, but can be adjusted – see https://hummy.tv/forum/threads/swapper-virtual-memory.8844/post-171196.
qtube
The qtube package adds a GUI front-end for youtube-dl to the WebIF, with the benefit of pushing downloads onto the auto-processing queue for processing in the background (along with other background processes such as decryption). This obviates the need for abduco as discussed above (background processes are already "abducoed"). Just paste the URL of a web page containing the video you want to download into the URL entry box, and click "Run download in background"... and wait for the job to finish (monitor the process queue and/or qtube log).
An immediate processing option is available, but this is best only used for obtaining the manifest ("-F" in the "Process options" box) as it is not "abducoed". Running an actual download in the foreground has all the same problems mentioned previously.
For control of the file selected for download, insert the string you would have used as options for the youtube command into the "Process options" box verbatim (everything from the equivalent youtube-dl command line except "youtube" and the URL)*. With no options at all, the download defaults to process options defined in a config file set to select the best quality file of no more than 1080 pixels vertical resolution (not a problem if you have fast broadband, but those with slower connections, limited data, or limited disk space might want to restrict the resolution to 540 lines - ie StDef). The config file can be edited if you want a different default (accessible directly via a link on the WebIF qtube page).
* NB: Do not use the single quote ' as a delimiter in the qtube process options box. {..} or ".." are acceptable.
On HDR-FOX, the resulting download is in My Video. For HD-FOX, it will be in Video on the recording drive. An alternative destination can be specified using the "-o" option. Provided the download is MP4 (or another compatible container format), it will be listed along with existing recordings on the MEDIA button... but it will only play provided the video and audio codecs within are also compatible.
As an alternative to get_iplayer, if you have the Customised Firmware (and that is where we are) you can install youtube-dl, which is currently capable of downloading iPlayer content, on the HD/HDR-Fox T2 itself.
You need to be happy with telnet and Unix shell to use youtube-dl like this. get_iplayer has a web interface that would make it easier to use on the HDx-Fox, but also endless dependencies that make its installation more of a term project.
To set up youtube-dl, first make sure you have these CF packages installed (using WebIf>Package Management or the opkg utility):
ffmpeg
wget
python.
Connect to the HDx-Fox by telnet and download the package:
set the save location (adapt the path according to HD vs HDR and your personal preference; adapt the filename template according to your preference - see 'man youtube-dl' section 'OUTPUT TEMPLATE' online):
if you don't want best available quality, you can eg limit the total bit rate to 1900kbit/s (again see 'man youtube-dl' section 'FORMAT SELECTION' online):
Obviously avoid any format option that might force re-encoding the video stream.
Reconnect the telnet session and finally you can run the program. Options specified on the command line override the defaults in the config file. Find the programme URL online, copy and paste into the command. Eg, with a topical programme:
Make a cup of tea|dry martini|whatever while the tiny CPU struggles through the Python code. Then you should see some output like this:
Code:
[bbc.co.uk] b09ptgt9: Downloading video page
... [~20 lines snipped]
[download] Destination: /media/drive1/My Video/Australian Open Tennis, 2018, Day 11 Highlights.mp4
and eventually:
Code:
[download] 100% of 760.46MiB in 17:12
[ffmpeg] Fixing malformed AAC bitstream in "/media/drive1/My Video/Australian Open Tennis, 2018, Day 11 Highlights.mp4"
will start a new shell in a session called "x", or re-attach if the session already exists. You can even attach to the same session from multiple places at once.
You could use >>/mod/tmp/youtube.log to redirect the output to a log viewable on the diag panel
You can use pgrep youtube to check if it is still running
will start a new shell in a session called "x", or re-attach if the session already exists. You can even attach to the same session from multiple places at once.
I still don't understand the need to start a shell session when instigating an abduco session. Just "abduco -A x" seems to work fine. Can anyone enlighten me?
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
