Youtube-dl mp4 issues

R

rodp

Member
Hi All,

I've been using Youtube-dl to download a few full series mp4 files (bbc iplayer) but i am having issues in that the box front panel and remote becomes unresponsive on occasions. Webif is ok but is stating it's playing the mp4 files (green play icon when you tap the top frame / border) but it just stays there until you reboot (via webif). I'm not watching it on the tv but it's as if it's processing it or indexing it.

I read a thread talking about a similar issue and they came to the conclusion that it was an indexing issue which perhaps is the same problem. If it is how can I test for this and how might I fix this? I have downloaded mp4 files via the same process in the past and they have been stored on the hdd however that's only been one or two, not 20/30 files.

Link to other thread https://hummy.tv/forum/threads/unit-freezes-when-playing-mp4-files.3850/

Once it gets stuck playing one of the mp4 files i have to reboot in order for the things waiting in the queue to be processed.

Screenshot_20201017-225229_Chrome.jpg

Any thoughts or any tests to do please let me know. The HDD and installation is brand new. I can do a disk check but think it's something else.

Thanks

Rodp
 
This is almost certainly something to do with the MP4 itself (or the HDR-FOX's media player, which we can do nothing about). Compare with a known good MP4 (I presume you are not saying all MP4s do the same thing).

To do any diagnosis, I think you will need to export the errand file to a proper computer for examination.
 
Ok will take that file and a good known mp4 file and do some discovery. What sort of differences would i look for?
 
So, I've had a look at a good file and a bad file. Both acquired via Qtube albeit the good file was acquired some time ago. Other than the difference in video bitrate there is a difference in the Audio format and some metadata. If it is an indexing problem could something like that cause an issue?

I've since rebooted the humax and it no longer showed it was playing the file in the background.

I then tried to see if I could re-create the error and deleted the bad file off the internal HDD and then copied it back on. the normal Humax DLNA service took the file off (checked via VLC / windows Media player) and then when I copied it back on, it reappeared in the UPNP list (well it did in windows media player but vlc is playing up and isn't wanting to show all the files for some reason despite restarting VLC). There was no message in Webif saying it was playing the file. So, I wasn't my able to repeat the issue so I'm now going to assume it was something to do with qtube / youtube-dl processing in collaboration with ffmpeg.

If the problem re-appears or the front panel and remote become unresponsive again, I'll investigate again further.

Thanks

Rodp

1603051112389.png
 
rodp said:
Hi All,

I've been using Youtube-dl to download a few full series mp4 files (bbc iplayer) but i am having issues in that the box front panel and remote becomes unresponsive on occasions. Webif is ok but is stating it's playing the mp4 files (green play icon when you tap the top frame / border) but it just stays there until you reboot (via webif). I'm not watching it on the tv but it's as if it's processing it or indexing it.

I read a thread talking about a similar issue and they came to the conclusion that it was an indexing issue which perhaps is the same problem. If it is how can I test for this and how might I fix this? I have downloaded mp4 files via the same process in the past and they have been stored on the hdd however that's only been one or two, not 20/30 files.
...
Click to expand...
I don't think you're alone. I have this issue on occasions. It just potentially locks up when playing back a mp4 file, and doesn't register when it finishes the playback. It doesn't happen frequently (hope I haven't jinxed it by saying that)
https://hummy.tv/forum/threads/status-occassionally-reports-wrong-state.9690/
 
HI All,

The issue has coem back again but this time it's a temp.mp4 file

1609671277145.png


Same symptoms, remote not responsive but webif is fine. I set 3 files to download overnight.

2612​
02/01/2021 23:46​
Https://www.bbc.co.uk/iplayer/episode/m000qplxqtubeCOMPLETE
00:42:48​
Check qtube.log
03/01/2021 01:45​
2611​
02/01/2021 23:46​
Https://www.bbc.co.uk/iplayer/episode/m000qpkfqtubeCOMPLETE
00:40:28​
Check qtube.log
03/01/2021 01:02​
2610​
02/01/2021 23:45​
Https://www.bbc.co.uk/iplayer/episode/m000qpjkqtubeFAILED
00:36:23​
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
03/01/2021 00:22​

The first one (Engine_Earth) reported an issue during download (see above) but infact does seem to have downloaded ok and can be played (via smb) but the temp.mp4 didn't delete (and I can't play it). The text of the error is below - I've abbreviated some of it once it stopped giving errors

Code: 
02/01/2021 23:45:02    Https://www.bbc.co.uk/iplayer/episode/m000qpjk    qtube    FAILED    00:36:23    WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden ERROR: frame= 307 fps=0.0 q=-1.0 size= 3072kB time=00:00:06.12 bitrate=4112.1kbits/s speed=12.2x frame= 536 fps=536 q=-1.0 size= 5888kB time=00:00:10.73 bitrate=4495.0kbits/s speed=10.7x frame= 762 fps=506 q=-1.0 size= 8960kB time=00:00:15.22 bitrate=4822.6kbits/s speed=10.1x frame= 1012 fps=504 q=-1.0 size= 12032kB time=00:00:20.22 bitrate=4874.7kbits/s speed=10.1x frame= 1271 fps=507 q=-1.0 size= 15616kB time=00:00:25.40 bitrate=5036.5kbits/s speed=10.1x frame= 1524 fps=506 q=-1.0 size= 18688kB time=00:00:30.46 bitrate=5026.0kbits/s speed=10.1x frame= 1781 fps=507 q=-1.0 size= 22016kB time=00:00:35.60 bitrate=5066.2kbits/s speed=10.1x frame= 2007 fps=500 q=-1.0 size= 25088kB time=00:00:40.12 bitrate=5121.6kbits/s speed= 10x frame= 2302 fps=510 q=-1.0 size= 28672kB time=00:00:46.03 bitrate=5102.0kbits/s speed=10.2x frame= 2561 fps=511 q=-1.0 size= 31744kB time=00:00:51.20 bitrate=5079.0kbits/s speed=10.2x frame= 2821 fps=512 q=-1.0 size= 34816kB time=00:00:56.40 bitrate=5057.0kbits/s speed=10.2x frame= 3159 fps=525 q=-1.0 size= 37632kB time=00:01:03.16 bitrate=4881.0kbits/s speed=10.5x frame= 3409 fps=523 q=-1.0 size= 40704kB time=00:01:08.16 bitrate=4892.1kbits/s speed=10.5x frame= 3657 fps=521 q=-1.0 size= 44288kB time=00:01:13.12 bitrate=4961.8kbits/s speed=10.4x frame= 3915 fps=521 q=-1.0 size= 47616kB time=00:01:18.28 bitrate=4983.0kbits/s speed=10.4x frame= 4225 fps=527 q=-1.0 size= 50688kB time=00:01:24.48 bitrate=4915.2kbits/s speed=10.5x frame= 4479 fps=526 q=-1.0 size= 54272kB time=00:01:29.56 bitrate=4964.2kbits/s speed=10.5x frame= 4703 fps=521 q=-1.0 size= 57088kB time=00:01:34.04 bitrate=4973.0kbits/s speed=10.4x frame= 4939 fps=518 q=-1.0 size= 60160kB time=00:01:38.79 bitrate=4988.4kbits/s speed=10.4x frame= 5179 fps=516 q=-1.0 size= 63232kB time=00:01:43.57 bitrate=5001.3kbits/s speed=10.3x frame= 5417 fps=514 q=-1.0 size= 66304kB time=00:01:48.32 bitrate=5014.4kbits/s speed=10.3x frame= 5674 fps=514 q=-1.0 size= 69376kB time=00:01:53.46 bitrate=5009.1kbits/s speed=10.3x frame= 5926 fps=514 q=-1.0 size= 72448kB time=00:01:58.50 bitrate=5008.4kbits/s speed=10.3x frame= 6043 fps=502 q=-1.0 size= 73728kB time=00:02:00.84 bitrate=4998.2kbits/s speed= 10x frame= 6160 fps=491 q=-1.0 size= 75264kB time=00:02:03.18 bitrate=5005.4kbits/s speed=9.82x frame= 6329 fps=485 q=-1.0 size= 77312kB time=00:02:06.56 bitrate=5004.3kbits/s speed= 9.7x frame= 6458 fps=477 q=-1.0 size= 79360kB time=00:02:09.17 bitrate=5032.9kbits/s speed=9.54x frame= 6629 fps=472 q=-1.0 size= 81408kB time=00:02:12.56 bitrate=5030.9kbits/s speed=9.43x frame= 6885 fps=473 q=-1.0 size= 83968kB time=00:02:17.68 bitrate=4996.1kbits/s speed=9.46x frame= 7105 fps=472 q=-1.0 size= 87040kB time=00:02:22.08 bitrate=5018.5kbits/s speed=9.44x frame= 7368 fps=474 q=-1.0 size= 90624kB time=00:02:27.34 bitrate=5038.6kbits/s speed=9.47x frame= 7670 fps=478 q=-1.0 size= 93696kB time=00:02:33.38 bitrate=5004.3kbits/s speed=9.55x frame= 7912 fps=478 q=-1.0 size= 97024kB time=00:02:38.22 bitrate=5023.2kbits/s speed=9.55x frame= 8161 fps=478 q=-1.0 size= 100352kB time=00:02:43.22 bitrate=5036.6kbits/s speed=9.57x frame= 8412 fps=479 q=-1.0 size= 103424kB time=00:02:48.22 bitrate=5036.6kbits/s speed=9.58x frame= 8675 fps=480 q=-1.0 size= 107008kB time=00:02:53.48 bitrate=5053.1kbits/s speed= 9.6x

......... 

frame=169866 fps=448 q=-1.0 size= 2103552kB time=00:56:37.30 bitrate=5072.4kbits/s speed=8.95x frame=170034 fps=448 q=-1.0 size= 2105856kB time=00:56:40.66 bitrate=5072.9kbits/s speed=8.95x frame=170217 fps=447 q=-1.0 size= 2108160kB time=00:56:44.32 bitrate=5073.0kbits/s speed=8.95x frame=170378 fps=447 q=-1.0 size= 2110464kB time=00:56:47.54 bitrate=5073.7kbits/s speed=8.94x frame=170552 fps=447 q=-1.0 size= 2112768kB time=00:56:51.02 bitrate=5074.1kbits/s speed=8.94x frame=170704 fps=447 q=-1.0 size= 2114304kB time=00:56:54.06 bitrate=5073.2kbits/s speed=8.94x frame=170895 fps=447 q=-1.0 size= 2116352kB time=00:56:57.88 bitrate=5072.5kbits/s speed=8.94x frame=171063 fps=447 q=-1.0 size= 2118656kB time=00:57:01.24 bitrate=5073.0kbits/s speed=8.93x frame=171218 fps=447 q=-1.0 size= 2120448kB time=00:57:04.34 bitrate=5072.7kbits/s speed=8.93x frame=171370 fps=446 q=-1.0 size= 2122496kB time=00:57:07.39 bitrate=5073.1kbits/s speed=8.93x frame=171561 fps=446 q=-1.0 size= 2125056kB time=00:57:11.20 bitrate=5073.6kbits/s speed=8.92x frame=171715 fps=446 q=-1.0 size= 2127360kB time=00:57:14.28 bitrate=5074.5kbits/s speed=8.92x frame=171891 fps=446 q=-1.0 size= 2129152kB time=00:57:17.80 bitrate=5073.6kbits/s speed=8.92x frame=172065 fps=446 q=-1.0 size= 2131456kB time=00:57:21.28 bitrate=5074.0kbits/s speed=8.92x frame=172238 fps=446 q=-1.0 size= 2134016kB time=00:57:24.75 bitrate=5074.9kbits/s speed=8.91x frame=172422 fps=446 q=-1.0 size= 2136320kB time=00:57:28.42 bitrate=5075.0kbits/s speed=8.91x frame=172616 fps=445 q=-1.0 size= 2138624kB time=00:57:32.30 bitrate=5074.8kbits/s speed=8.91x frame=172806 fps=445 q=-1.0 size= 2141184kB time=00:57:36.10 bitrate=5075.3kbits/s speed=8.91x frame=173003 fps=445 q=-1.0 size= 2143744kB time=00:57:40.05 bitrate=5075.5kbits/s speed=8.91x frame=173217 fps=445 q=-1.0 size= 2146048kB time=00:57:44.32 bitrate=5074.7kbits/s speed=8.91x frame=173413 fps=445 q=-1.0 size= 2148608kB time=00:57:48.24 bitrate=5075.0kbits/s speed= 8.9x frame=173619 fps=445 q=-1.0 size= 2151168kB time=00:57:52.36 bitrate=5075.0kbits/s speed= 8.9x frame=173792 fps=445 q=-1.0 size= 2153216kB time=00:57:55.82 bitrate=5074.8kbits/s speed= 8.9x frame=173998 fps=445 q=-1.0 size= 2155776kB time=00:57:59.94 bitrate=5074.8kbits/s speed= 8.9x frame=174189 fps=445 q=-1.0 size= 2158336kB time=00:58:03.79 bitrate=5075.2kbits/s speed= 8.9x frame=174355 fps=445 q=-1.0 size= 2160384kB time=00:58:07.08 bitrate=5075.3kbits/s speed= 8.9x frame=174531 fps=445 q=-1.0 size= 2162944kB time=00:58:10.60 bitrate=5076.2kbits/s speed=8.89x

We can see the humax has index the water world file ok but because it's got hung up on the Engine_Earth.temp.mp3 file it won't continue and as a result the humax freezes.
1609671787239.png

I've tried rebooting a number of times but the problem continues and no doubt will continue until I delete the temp.mp4 file. Is there a way to stop the indexing looking at temp.mp4 files? Is this a custom firmware / package issue or a humax issues.

I won't change anything in case someone has some suggestions first and want to use my issue as a test bed.

Thanks

Rodp
 
I think something in the post-download ffmpeg post-processing crapped out and has left the temp file locked. I doubt there is anything to be done about it other than to dig in and clean up.
 
Does that mean there is nothing I can do except wait for the problem to occur and then delete it manually each time? Is there no way to stop the dnla from analysing temp.mp4 files? Could the youtube-dl / ffmpeg process use a file names mp4.temp instead? Assuming that this would then stop the dlna playing/indexing the file?

The reason I ask is that this issue also seems to stop anything that is scheduled to record.

Thanks

Rodp
 
Last edited:
rodp said:
Is there no way to stop the dnla from analysing temp.mp4 files?
Click to expand...
None that I can think of; DLNA indexing is a core Humax settop process.

rodp said:
Could the youtube-dl / ffmpeg process use a file names mp4.temp instead?
Click to expand...
Maybe, but my guess is that ffmpeg decides what to call its temp files itself and is not user-configurable. However, that said, ffmpeg may well have been compiled specifically for use on the HDR-FOX so possibly some enterprising individual could hack the code to alter the naming scheme, and recompile. This could be fraught with problems if any obscure references get missed, and maybe ffmpeg itself relies on even temp files being ".mp4".

rodp said:
Assuming that this would then stop the dlna playing/indexing the file?
Click to expand...
I guess so, presuming that is the actual problem.
 
A download submitted by qtube will continue to be run by the queue system until it either succeeds or fails.

If the post-processing wasn't completed, youtube-dl will run again, identify the preferred download, see that it has already been downloaded and then (I believe) run the post-processing again. Once the entire queued operation succeeds (or fails), the queue item is marked and won't be re-run unless it's manually re-submitted.

The post-processor writes to a file with the name of the downloaded file with ".temp" inserted before the dot-extension, and then renames the .temp file over the downloaded file. So if a .temp is left it may indicate that the post-processing failed, and needs to be repeated, or it may just be that the Python os.rename() primitive failed to delete the .temp file (the files will be the same size and bit-identical).
 
I suppose it's possible that the DLNA scanner is reading the .temp.mp4 file and so preventing it from being renamed.

qtube could be modified to use a separate directory that isn't scanned but that would leave other failure modes where the show gets downloaded twice, because the already-downloaded file has already been moved to its final location, or where the moving goes wrong. I don't think yt-dl has an option for using a temporary location for the post-processing phase. Creating the file with a qtube user and making it RW only by that user until the file is ready would only work if the set-top program was not run as root, which would no doubt introduce more gotchas.

"Playing" status is only shown for a file that is opened by the set-top program itself. Just being opened by ffmpeg doesn't count. DLNA scanning doesn't usually show up as it's transient, but streaming does. Possibly some media issue could cause the scan to take longer, or even hang.
 
Maybe this will muddy the water but I was able to rename the file from 'Royal_Institution_Christmas_Lectures_2020_-Planet_Earth-A_User_s_Guide_Engine_Earth.temp.mp4' to 'Royal_Institution_Christmas_Lectures_2020-Planet_Earth-_A_User_s_Guide_Engine_Earth.temp.mp4.tmp' via FTP and that stopped it showing up as playing. I've since rebooted and the remote is responsive again.


...and the Royal_Institution_Christmas_Lectures_2020_-Planet_Earth-_A_User_s_Guide_Engine_Earth.mp4 that was there whilst the temp file was there seems to play ok
 
rodp said:
Maybe this will muddy the water but I was able to rename the file from 'Royal_Institution_Christmas_Lectures_2020_-Planet_Earth-A_User_s_Guide_Engine_Earth.temp.mp4' to 'Royal_Institution_Christmas_Lectures_2020-Planet_Earth-_A_User_s_Guide_Engine_Earth.temp.mp4.tmp' via FTP and that stopped it showing up as playing. I've since rebooted and the remote is responsive again.
Click to expand...
I think this is as expected. The DLNA scan doesn't look at .tmp files in the "My Video" directory, as per the configuration in /mnt/hd2/dms_cds.db :
Code: 
sqlite> select path, filter from tblScanSetup ;
/mnt/hd2/tmp|*
/mnt/hd2/My Video|mpg;avi;mkv;ts;wmv;tp;mp4;asf;mpeg;trp
/mnt/hd2/My Photo|jpeg;jpg
/mnt/hd2/My Music|mp3
sqlite>
...and the Royal_Institution_Christmas_Lectures_2020_-Planet_Earth-_A_User_s_Guide_Engine_Earth.mp4 that was there whilst the temp file was there seems to play ok
Click to expand...
... which is consistent with it and the .temp.mp4 being identical copies of the qtube output file.
 
/df said:
I think this is as expected. The DLNA scan doesn't look at .tmp files in the "My Video" directory, as per the configuration in /mnt/hd2/dms_cds.db :
Code: 
sqlite> select path, filter from tblScanSetup ;
/mnt/hd2/tmp|*
/mnt/hd2/My Video|mpg;avi;mkv;ts;wmv;tp;mp4;asf;mpeg;trp
/mnt/hd2/My Photo|jpeg;jpg
/mnt/hd2/My Music|mp3
sqlite>

... which is consistent with it and the .temp.mp4 being identical copies of the qtube output file.
Click to expand...
Out of interest, can you edit that tblScanSetup to include mp3 and mp2 or anything else? eg. wav or wmv

Thanks

Rodp
 
Theoretically, yes. Practically, no.

For instance, you can go into a maintenance mode telnet session, modify an existing database and then restart. If the edit makes the filters more strict, existing non-matching entries won't be automatically removed, I think, so you would also have to work out how to do that manually. Probably the best solution is to have an empty database saved that you can restore and edit, and then let it be repopulated automatically.

You can make an empty database by some procedure like this:
  • in maintenance mode, move the database file to a safe place or new name in case it needs to be restored, make a new top-level directory and move the "My xxx" directories into it;
  • restart into normal mode;
  • back in maintenance mode, copy the newly created empty database to a safe place;
  • if desired, now make any desired edits and back up the result;
  • move the "My xxx" back to their original locations;
  • restart and enjoy.

If the modified filters select media formats that the DLNA server doesn't understand, I'd expect problems.

However, it's difficult to make any automatic changes stick, because changes have to be made before the settop program loads the configuration from the database, yet the settop program is what creates the database if it's bad or missing and what mounts the file system in which it is created; even manual changes (as above) could be reversed at any time if the settop program decides to reinitialise the database (it's prone to do that if it sees a database error). A simple change like removing one extension might be achieved by binary editing the schema text in the settop program, but of course that is in the read-only storage. You might hope to be able to hook the database access calls but libsqlite3 is statically linked into the settop program instead of using /usr/lib/libsqlite3.so (is that a CF addition?).

All of this makes one wonder again why the DLNA server is embedded in the settop program instead of being a separate executable.
 
You must log in or register to reply here.
Back
Top