Copying Subtitles and additional audio streams - ffmpeg patch?

MontysEvilTwin

MontysEvilTwin

Well-Known Member
Nicesplice is handy for removing adverts from recordings, but I have noticed that when the cropped files are played back on the HDR-FOX, the sound drops out for a few seconds after each join. If the dialogue starts quickly after a join it is sometimes necessary to leave a few seconds of the advert in to avoid sound loss (there is a work around, wait until the sound resumes and rewind to just after the join and then play). Also when played on other devices, the length of the recording is usually misreported and there can be problems when a join is reached (some players are better than others, Kodi is pretty good). These issues are caused by timestamps in the ts file and can be fixed by remuxing. If you just want to remux the video and main audio stream, this can be done on the unit itself with ffmpeg:
Code: 
ffmpeg -i input.ts -c copy output.m2ts
This is quick (no transcoding) and will fix the timestamps. It works for both MPEG2/ MP2 and H.264/ AAC recordings. If the file extension of the output file is then changed to ts (from m2ts) you can use the Sidecar package to create hmt and nts files for playback on the HDR-FOX with full transport controls.
Copying subtitles with ffmpeg was recently mentioned here. It is possible to copy additional streams using the map option with ffmpeg see here.
There seem to be up to four relevant streams in a DVB-T transmission, 0:0 (video), 0:1 (main audio),
0:2 (audio description or subtitle) and 0:3 (subtitle). If there are no subtitles, MPEG2 recordings tend
to have just two streams, with subtitles four: it seems there is a dummy audio stream at 0:2 if there is no audio description track. H.264 recordings have 3 or 4 streams, 3 if no audio description. For both MPEG2 and H.264 video, with audio description and subtitles the following can be used to copy the four streams into a new file:
Code: 
ffmpeg -i input.ts -map 0:0 -map 0:1 -map 0:2 -map 0:3 -c copy output.m2ts
The above can be adjusted depending on which streams you want to copy. For example, ffmpeg falls over if you try and copy an MPEG2 recording with a dummy audio description stream: in this case remove '-map 0:2' to stop ffmpeg attempting to copy that stream.
Copying true audio description streams works fine by this method, but subtitles are more challenging. Cropped files can sometimes cause ffmpeg to fall over when a subtitle track is included (see above post from xyz321). Also, with the version of ffmpeg on the HDR-FOX, the subtitles are copied but cannot be displayed due errors in the header. Both of these are known issues with ffmpeg for which a patch is available see here. I wonder if it is feasible to patch ffmpeg (or use a later version?) to solve this problem without affecting Detectads, for example? Or can more than one version of ffmpeg be incorporated into the custom firmware?
 
Last edited:
I would like to remove the silences after crop points

DetectAds is not. AFAIK, dependent on any specific version of ffmpeg - it just needs the audio extraction function, if possible I would like to eliminate the use of ffmpeg and extract audio directly from ts but I haven't looked at what would be involved.

Similarly we could possibly fix up the audio timestamps and subtitle tracks within nicesplice if we knew what needed to be done but it might be easiest to write the nicesplice output to stdout so that it could be piped into ffmpeg for remuxing.
 
Version 2.1.4 of ffmpeg used to be available in the test section of the repository, but is no longer. This version is more recent than the patch I mentioned above. af123 - would it be possible to get a copy of 2.1.4 (or later) to see if it copies the subtitle stream correctly? Was there a particular problem with this version which prevented it being placed in the main repository, or was it just that ffmpeg is suffering from middle-aged spread?
 
I tried about 6 different versions of ffmpeg at one point. None of them were as fast, small or stable as the version that's in the repository now! I'll dig out a copy of 2.1.4 and upload it for you..
 
af123 provided later versions of ffmpeg and I have had some success with ffmpeg 2.8. This version can copy the subtitles, in addition to the video and audio streams. For unedited recordings I have had 100% success rate to date with both MPEG2 and H264 video. With nicesplice cropped recordings, ffmpeg gives a warning about 'non-monotonous DTS in output stream' when it reaches a join but mostly it copes with this, fixes the stream timestamp and carries on copying. Occasionally though a join can make the process quit. This seems to happen when nicesplice gives particularly pixelated output at the join. I tend to leave a few spare seconds of video after a nicesplice join because of the sound dropouts. I found that recropping with nicesplice and removing a few seconds at the join can fix the problem, with ffmpeg then able to complete the copy and remux. This fix probably won't work on every occasion.

The following commands are useful as they make ffmpeg copy and remux all video and audio streams plus the subtitles whilst ignoring any unknown data streams:
Code: 
ffmpeg -i input.ts -map 0 -c copy -ignore_unknown output.m2ts
If the recording has a dummy audio description stream the above probably won't work. In this case, use ffmpeg to see which streams are present:
Code: 
ffmpeg -i input.ts
Then use the map option to choose the streams you want to copy (see post #1).

If you change the file extension of the output to 'ts' you can then use the Sidecar package to create new hmt and nts files. I encountered a minor issue with the Sidecar generated hmt file for ffmpeg processed H264 recordings (the Sidecar package was designed for native files). The recording length is correct but when you play it from the start it seems like the recording is already at 2 hours 30 minutes although it is less than 2 hours long. I have not located where the problem entry is in the hmt file, but I have a work around which I think is better. If you copy the hmt file from the input recording, renaming it as necessary, all you need to do is change the Prog ID field (offset=0x0488) to match the processed recording. This will do the trick:
Code: 
hmt +patch16=1160:0001 output.hmt
This hmt file contains the original timestamps and synopsis data.

I have also been using ffmpeg 2.8 with Detectads on two machines for two days with no problems to date.
 
Hi

I know this thread is 6 months old but I have a issue with streaming to a Kodi box for some (mostly Channel 4) files after detectads has been used. The issue is fixed after post-processing using ffmpeg, but similarly to MontysEvilTwin, I lose the subtitles if I process on the Hummy. Could either of you kind sirs provide me with a copy of ffmpeg 2.8 or point me in the direction of where to download?
 
From the command line you can do:

Code: 
humax# opkg install http://hpkg.tv/hdrfoxt2/test/ffmpeg_2.8_mipsel.opk
 
Sometimes when remuxing recordings with subtitles that have been cropped with Nicesplice, ffmpeg can fail at the points where sections are joined. This is due to errors in the decode timestamps in one or more of the streams: not necessarily the subtitle stream itself. Subtitle extraction and remuxing seems particularly sensitive to these errors. It is also not uncommon for subtitle extraction to fail when the source recording has not been edited.
I now have a way of processing subtitles which I use when the standard method fails. I have found this to be reliable with recordings made on the HDR-FOX. First remux and copy the subtitles to a new file, resetting the timestamp data:
Code: 
ffmpeg -use_wallclock_as_timestamps 1 -i File.ts -map 0:1 -map 0:3 -c copy File2.m2ts
The timestamp data is reset using the '-use_wallclock_as_timestamps 1' command. In this example, I have included the main audio stream. You need to copy a second stream for this to work. Then you can merge the subtitles into a new file with the video and audio streams from the original recording:
Code: 
ffmpeg -i File.ts -itsoffset 3 -i File2.m2ts -map 0:0 -map 0:1 -map 0:2 -map 1:1 -c copy File3.m2ts
Usually doing this makes the subtitle stream get out of sync with the other streams: in this example, '-itsoffset 3' makes the subtitle stream start 3 seconds later than the other streams to compensate. You will need to tweak this. I typically start with a 3 second offset and while processing is ongoing, I start playback on a tablet to see how good the sync is. If the offset is wrong, typing a 'q' in Telnet will quit the processing, then you can change the offset and try again. It may be possible to extract the start time for the subtitle stream from the original recording which would make merging the streams more efficient.
I have been using ffmpeg 2.8 for a few months now without any significant problems. It works fine with Detectads. The only problem I have found is that the mp2/ mp3 extraction built into Webif does not function. This is fine from the command line so there is probably an easy fix. af123 - it may be worth rolling out a later ffmpeg version at some point. If you are amenable, I would like to try out an even later version of ffmpeg. I have been using a Windows build of ffmpeg 3.2. I don't think this update makes subtitle extraction more robust, but the aac audio encoder is much improved (even compared to version 2.8, where this function is still experimental). This may facilitate the conversion of recordings from DVB-T2 multiplexes to mp4 files. Currently this is slow and prone to audio/ video desync.
 
I am just confused!
Sometimes when remuxing recordings with subtitles that have been cropped with Nicesplice, ffmpeg can fail at the points where sections are joined.
Click to expand...
"Sometimes" implies that the error is not consistent and that it is possible to crop recordings without introducing timestamp errors so I wonder whether it is feasible to improve nicesplice's cropping to avoid the times that cause problems!

I am no expert in recording file formats (virtually totally ignorant) but I do know that nicesplice attempts to crop at an I-frame and will normally look for one up to 25 frames either side of the nominal crop point. Could the fix be as simple as increasing the I-frame search range so it always crops at an I-frame?
 
In my limited examination of streams (or rather the .nts files), I've seen GoP lengths up to about 30 frames, so 25 does seem too small.
 
MymsMan said:
I am just confused!

"Sometimes" implies that the error is not consistent and that it is possible to crop recordings without introducing timestamp errors so I wonder whether it is feasible to improve nicesplice's cropping to avoid the times that cause problems!

I am no expert in recording file formats (virtually totally ignorant) but I do know that nicesplice attempts to crop at an I-frame and will normally look for one up to 25 frames either side of the nominal crop point. Could the fix be as simple as increasing the I-frame search range so it always crops at an I-frame?
Click to expand...
I did find that if you move the second crop point (i.e. the one after the ad break) by a second or two this may fix the problem for that particular section.
I don't know much about recording formats either, so forgive me if I am talking nonsense. If the end of a section is cut at an I-frame (i.e. the last frame of the output section is an I-frame) would you want the next section to start at an I-frame + 1 frame, rather than having 2 adjacent I-frames in the output?
You can do all the cropping in ffmpeg (cutting to be consistent with ffmpeg terminology; in their documentation cropping refers to removing part of the video image). Joining sections of video created by ffmpeg does not suffer from the same problem with the subtitles. Annoyingly most of the processes needed to cut and join video in ffmpeg are amenable to automation but a few issues make this difficult. If anyone is interested I can create a new thread about this as others may be able to solve the problems that stumped me.
I found a particular problem when editing recordings made on Film4+1 (H.264 video/ LATM-AAC audio, but standard definition) both with nicesplice and ffmpeg. When played back on the HDR-FOX, sometimes the video can jump up to about 10 seconds after a join. This could be remedied by moving the cut points on a trial and error basis. However, with other players (e.g. VLC on Windows, MX on Android, Kodi on OSMC) the 'jumpy' video played without problems.
 
I was confused! ;)

The frame search is actually +/- 50 frames - a 4 second range and it searches for a non B frame at the start of the cut and an I frame when the recording is to resume.

If you run nicesplice with the -DebugOutput flag it will write a messages showing whether crop points have been moved to the correct frame type
 
MontysEvilTwin said:
Sometimes when remuxing recordings with subtitles that have been cropped with Nicesplice, ffmpeg can fail at the points where sections are joined. This is due to errors in the decode timestamps in one or more of the streams: not necessarily the subtitle stream itself. Subtitle extraction and remuxing seems particularly sensitive to these errors. It is also not uncommon for subtitle extraction to fail when the source recording has not been edited.
Click to expand...
Here is a patch to ffmpeg which seems to help with this:

Code: 
--- ffmpeg.c.old	2015-01-10 04:21:15.000000000 +0000
+++ ffmpeg.c	2017-03-25 09:02:37.000000000 +0000
@@ -676,7 +676,7 @@
                      - FFMAX3(pkt->pts, pkt->dts, ost->last_mux_dts + 1);
         }
      if(
-        (avctx->codec_type == AVMEDIA_TYPE_AUDIO || avctx->codec_type == AVMEDIA_TYPE_VIDEO) &&
+        (avctx->codec_type == AVMEDIA_TYPE_AUDIO || avctx->codec_type == AVMEDIA_TYPE_VIDEO || avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) &&
         pkt->dts != AV_NOPTS_VALUE &&
         ost->last_mux_dts != AV_NOPTS_VALUE) {
       int64_t max = ost->last_mux_dts + !(s->oformat->flags & AVFMT_TS_NONSTRICT);

https://trac.ffmpeg.org/ticket/4450
 
[vob @ 0x8526700] Non-monotonous DTS in output stream 0:1; previous: 331320960, current: 331319487; changing to 331320961. This may result in incorrect timestamps in the output file.
Click to expand...
:rolling:

I think maybe that should read "non-monotonic". A case of auto-correct getting in on the act?
 
xyz321 said:
Here is a patch to ffmpeg which seems to help with this:

Code: 
--- ffmpeg.c.old    2015-01-10 04:21:15.000000000 +0000
+++ ffmpeg.c    2017-03-25 09:02:37.000000000 +0000
@@ -676,7 +676,7 @@
                      - FFMAX3(pkt->pts, pkt->dts, ost->last_mux_dts + 1);
         }
      if(
-        (avctx->codec_type == AVMEDIA_TYPE_AUDIO || avctx->codec_type == AVMEDIA_TYPE_VIDEO) &&
+        (avctx->codec_type == AVMEDIA_TYPE_AUDIO || avctx->codec_type == AVMEDIA_TYPE_VIDEO || avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) &&
         pkt->dts != AV_NOPTS_VALUE &&
         ost->last_mux_dts != AV_NOPTS_VALUE) {
       int64_t max = ost->last_mux_dts + !(s->oformat->flags & AVFMT_TS_NONSTRICT);

https://trac.ffmpeg.org/ticket/4450
Click to expand...
This fault is a real annoyance. It seems that the solution is quite straightforward. Ffmpeg 2.8 is now in general use; it supports copying of DVB subtitles but does suffer from this problem. How easy is it to include this patch and compile a mips version? af123: if is not too much trouble, I would be happy to test a patched version, at your convenience, of course.
 
You must log in or register to reply here.
Back
Top