youtube-dl: no formats available for some BBC programmes

T

Thaddeus

Member
I've recently had a problem with the iPlayer app returning a "Something went wrong loading this programme" error, regardless of which programme I try, so I've been exploring the youtube-dl and qtube packages.

In the main they've worked fine but I've got an issue with Atlanta on iPlayer, as per the error below. Does anyone know what causes it, as the programmes themselves are there to play:

Code: 
07:56:26 - --------------------------------------------------------------
07:56:26 - Starting immediate download of https://www.bbc.co.uk/iplayer/episode/b0b3px4q/atlanta-series-1-1-the-big-bang Options -F
07:57:15 - [bbc.co.uk] b0b3px4q: Downloading video page
07:57:16 - [bbc.co.uk] b0b3px4q: Downloading playlist JSON
07:57:16 - [bbc.co.uk] b0b3ptgc: Downloading media selection XML
07:57:17 - Caught error: ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.


=== Done ===
 
Thaddeus said:
In the main they've worked fine but I've got an issue with Atlanta on iPlayer, as per the error below. Does anyone know what causes it, as the programmes themselves are there to play:
Click to expand...
What do you get if you run qtube in immediate mode with -F in the options? (update: already done in the previous post :oops: )

You could also try shortening the URL - you only need the URL up to the unique ID: https://www.bbc.co.uk/iplayer/episode/b0b3px4q
 
Last edited:
Thaddeus said:
I've got an issue with Atlanta on iPlayer
Click to expand...
I don't think this is a problem with qtube or youtube-dl on the Humax, I think that the files are missing on the Iplayer server, some of the episodes are present, see the list below:-
Code: 
no  https://www.bbc.co.uk/iplayer/episode/b0b3px4q/atlanta-series-1-1-the-big-bang
no  https://www.bbc.co.uk/iplayer/episode/b0b3px4s/atlanta-series-1-2-streets-on-lock
no  https://www.bbc.co.uk/iplayer/episode/b0b4jlbb/atlanta-series-1-3-go-for-broke
no  https://www.bbc.co.uk/iplayer/episode/b0b4jlbd/atlanta-series-1-4-the-streisand-effect
yes https://www.bbc.co.uk/iplayer/episode/b0b6486r/atlanta-series-1-5-nobody-beats-the-biebs
yes https://www.bbc.co.uk/iplayer/episode/b0b6486y/atlanta-series-1-6-value
no  https://www.bbc.co.uk/iplayer/episode/b0b6y213/atlanta-series-1-7-ban
no  https://www.bbc.co.uk/iplayer/episode/b0b6y2m7/atlanta-series-1-8-the-club
yes https://www.bbc.co.uk/iplayer/episode/b0b7rh1n/atlanta-series-1-9-juneteenth
no  https://www.bbc.co.uk/iplayer/episode/b0b7rh1q/atlanta-series-1-10-the-jacket
 
This is related to this yt-dl issue.

The yt-dl extractor for iPlayer uses these templates to find the playlist XML listing available programme streams:
Code: 
 _MEDIASELECTOR_URLS = [
        # Provides HQ HLS streams with even better quality that pc mediaset but fails
        # with geolocation in some cases when it's even not geo restricted at all (e.g.
        # http://www.bbc.co.uk/programmes/b06bp7lf). Also may fail with selectionunavailable.
        'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/iptv-all/vpid/%s',
        'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/%s',
]
In this case the first one matches but only provides captions. If the extractor tried the second one, it would find the video and audio too, but it gives up once it successfully parses a playlist XML.
 
/df said:
This is related to this yt-dl issue.

The yt-dl extractor for iPlayer uses these templates to find the playlist XML listing available programme streams:
Code: 
_MEDIASELECTOR_URLS = [
        # Provides HQ HLS streams with even better quality that pc mediaset but fails
        # with geolocation in some cases when it's even not geo restricted at all (e.g.
        # http://www.bbc.co.uk/programmes/b06bp7lf). Also may fail with selectionunavailable.
        'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/iptv-all/vpid/%s',
        'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/%s',
]
In this case the first one matches but only provides captions. If the extractor tried the second one, it would find the video and audio too, but it gives up once it successfully parses a playlist XML.
Click to expand...

Thanks. Do you know if there's a workaround? The last paragraph on the page you linked to, suggests there might be but I don't understand it well enough to apply it to my situation.

Thanks also to Black Hole (I'll cross my fingers that the iPlayer app gets sorted) and to EzraPound, as you're right that a few of the episodes are available in the normal way.
 
Last edited by a moderator:
Thaddeus said:
Thanks. Do you know if there's a workaround? The last paragraph on the page you linked to, suggests there might be but I don't understand it well enough to apply it to my situation.
...
Click to expand...
One idea would be to look at the second URL template with the desired playlist ID (b0b3ptgc) instead of %s.

Or wait for the issue to be fixed in the original package and then shortly after in the CF repo.

Or fix it yourself. This is possible using the telnet command line but I imagine it's a bit too challenging. Nonetheless, here are the steps (and yes, this discussion should be in the CF forum!):
  • connect to the Humax using a graphical telnet client, or Webshell;
  • extract the bbc.py file, using 7zip (make sure the package is installed):
    Code: 
    cd /tmp
7z x /mod/bin/youtube-dl youtube_dl/extractor/bbc.py
  • edit the file: vi youtube_dl/extractor/bbc.py; find the method /def _download_media_selector\([Enter]; delete it 12dd; disable autoindent :set noai[Enter]; copy the replacement text from the issue (NB initial spaces are significant); go into insert mode in vi i; using the telnet client's paste function, paste the replacement text at the insert point; exit insert mode [Esc]; save and quit :wq[Enter]; phew;
  • make an updated Python archive:
    Code: 
    dd if=/mod/bin/youtube-dl skip=1 bs=22 of=youtube-dl.zip
7z -tzip u youtube-dl.zip youtube_dl/extractor/bbc.py
cp /mod/bin/youtube-dl .
dd if=youtube-dl count=1 bs=22 | cat - youtube-dl.zip >/mod/bin/youtube-dl
  • check your work:
    Code: 
    # ls -l /mod/bin/youtube-dl
-rwxr-xr-x 1 root root 1756161 Oct  6 14:45 /mod/bin/youtube-dl
# python /mod/bin/youtube-dl -F 'https://www.bbc.co.uk/iplayer/episode/b0b3px4q'
[bbc.co.uk] b0b3px4q: Downloading video page
[bbc.co.uk] b0b3px4q: Downloading playlist JSON
[bbc.co.uk] b0b3ptgc: Downloading media selection XML
[bbc.co.uk] b0b3ptgc: Downloading media selection XML
[bbc.co.uk] b0b3ptgc: Downloading MPD manifest
[bbc.co.uk] b0b3ptgc: Downloading MPD manifest
[bbc.co.uk] b0b3ptgc: Downloading MPD manifest
[bbc.co.uk] b0b3ptgc: Downloading MPD manifest
[info] Available formats for b0b3ptgc:
format code                                                                             extension  resolution note
stream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-audio_eng=96000           m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
stream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash_https-audio_eng=96000     m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
stream-uk-pc_streaming_concrete_combined_sd_mf_limelight_uk_dash-audio_eng=96000        m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
stream-uk-pc_streaming_concrete_combined_sd_mf_limelight_uk_dash_https-audio_eng=96000  m4a        audio only [en] DASH audio   96k , m4a_dash container, mp4a.40.5 (48000Hz)
stream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-video=281000              mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015, 25fps, video only
stream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash_https-video=281000        mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015, 25fps, video only
stream-uk-pc_streaming_concrete_combined_sd_mf_limelight_uk_dash-video=281000           mp4        384x216    DASH video  281k , mp4_dash container, avc3.42C015
...
#
On my Linux PC this was all much easier because the youtube-dl package was unpacked by the installer and so the file could be edited, then the .pyc cached compiled version removed, then run the package again to recompile it (all as root).
 
Last edited:
At a guess it fails in the same way that running youtube-dl -F does: no stream formats are listed because it finds only captions; and the same for the abbreviated URL.
 
/df said:
7z -tzip -u ...
Click to expand...
This should of course be 7z -tzip u ...

I updated my copy (assuming I did it correctly) and can get the -F list, but it still can't download: "ERROR: requested format not available"
 
Thanks /df

I tried this:
/df said:
One idea would be to look at the second URL template with the desired playlist ID (b0b3ptgc) instead of %s.
Click to expand...

but I still get an error.

I don't think I'm confident enough to try and fix the code but I appreciate you taking the time to post it.
 
Thaddeus said:
...
I tried this:
...
but I still get an error.
Click to expand...
You might have to open the URL in your browser and then look at the URLs embedded in the returned playlist. Luckily "it's only telly".
I don't think I'm confident enough to try and fix the code but I appreciate you taking the time to post it.
Click to expand...
You can always reinstall the package if it gets broken. Also, others might be braver/rasher!

Anyway, we can only hope for a fix to be accepted into the original package.
 
Last edited:
prpr said:
This should of course be 7z -tzip u ...
Click to expand...
Obvs, thx. Obsolete shell history shown instead of the working command :-(
I updated my copy (assuming I did it correctly) and can get the -F list, but it still can't download: "ERROR: requested format not available"
Click to expand...
After the successful -F this worked for me on the Humax:
Code: 
# mod/bin/youtube-dl -f 'stream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-video=281000+stream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-audio_eng=96000' 'https://www.bbc.co.uk/iplayer/episode/b0b3px4q'
[bbc.co.uk] b0b3px4q: Downloading video page
[bbc.co.uk] b0b3px4q: Downloading playlist JSON
[bbc.co.uk] b0b3ptgc: Downloading media selection XML
[bbc.co.uk] b0b3ptgc: Downloading media selection XML
[bbc.co.uk] b0b3ptgc: Downloading MPD manifest
[bbc.co.uk] b0b3ptgc: Downloading MPD manifest
[bbc.co.uk] b0b3ptgc: Downloading MPD manifest
[bbc.co.uk] b0b3ptgc: Downloading MPD manifest
[dashsegments] Total fragments: 381
[download] Destination: Atlanta, Series 1, The Big Bang-b0b3ptgc.fstream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-video=281000.mp4
[dashsegments] Total fragments: 381
[download] Destination: Atlanta, Series 1, The Big Bang-b0b3ptgc.fstream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-video=281000.mp4
[download] 100% of 47.19MiB in 02:29
[dashsegments] Total fragments: 381
[download] Destination: Atlanta, Series 1, The Big Bang-b0b3ptgc.fstream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-audio_eng=96000.m4a
[download] 100% of 16.86MiB in 01:47
[ffmpeg] Merging formats into "Atlanta, Series 1, The Big Bang-b0b3ptgc.mp4"
Deleting original file Atlanta, Series 1, The Big Bang-b0b3ptgc.fstream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-video=281000.mp4 (pass -k to keep)
Deleting original file Atlanta, Series 1, The Big Bang-b0b3ptgc.fstream-uk-pc_streaming_concrete_combined_sd_mf_akamai_uk_dash-audio_eng=96000.m4a (pass -k to keep)
#
I got a similar result on the laptop with -f 'bestvideo[width<400]+bestaudio', but without having to wait so long ...

Is this series any good?
 
This recent portion of the thread should be split out and moved to the CF section of the forum since it has nothing to do with the Humax TV Portal !
 
/df said:
You can always reinstall the package if it gets broken
Click to expand...

I took heart from this and decided to try to fix the code using your instructions. Unfortunately I must have gone wrong somewhere as I'm now left with this error when I try to use a "valid" url in Q Download:

Code: 
22:33:48 - --------------------------------------------------------------
22:33:48 - Starting immediate download of https://www.bbc.co.uk/iplayer/episode/b00txmkk/battle-of-britain-the-real-story Options -F
22:34:08 - Caught error: Traceback (most recent call last):
  File "/mod/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/mod/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/mod/bin/youtube-dl/__main__.py", line 16, in <module>
  File "/mod/bin/youtube-dl/youtube_dl/__init__.py", line 43, in <module>
  File "/mod/bin/youtube-dl/youtube_dl/extractor/__init__.py", line 9, in <module>
  File "/mod/bin/youtube-dl/youtube_dl/extractor/extractors.py", line 84, in <module>
  File "/mod/bin/youtube-dl/youtube_dl/extractor/bbc.py", line 360
    formats += formatsAndSubtitles[0]
          ^
SyntaxError: invalid syntax

Full disclosure: I tried to copy and paste your "def _download_media_selector(self, programme_id):..." code with "copy" and a right mouse click (in Putty) but each time the formatting went wrong, so in the end I typed it out from scratch, so it's quite probable that I've copied something wrongly, in the same way I copied out programs from books as a child, barely understanding a word.

I've tried to "remove" the youtube.dl package, which seems to work fine but then it doesn't disappear from the "installed" section (or appear in the "available" section. I've tried resetting the Humax box but it doesn't change anything.

It might be because it's late and my brain is tired so I'm going to switch off and leave it alone tonight but is there a simpler solution than just reinstalling the custom firmware from scratch?

And yes, apologies to MymanMan and others, as I didn't think about the forum sections when I started this question today, as it seemed to follow on from the previous discussion.
 
Thaddeus said:
I've tried to "remove" the youtube.dl package, which seems to work fine but then it doesn't disappear from the "installed" section (or appear in the "available" section. I've tried resetting the Humax box but it doesn't change anything.
Click to expand...
If you have the Qtube front end for Youtube-dl installed then removing Youtube-dl will fail if you don't uninstall Qtube first. Alternatively, from a terminal:
Code: 
opkg remove youtube-dl --force-depends
will remove it.
 
You must log in or register to reply here.
Back
Top