Bookmark changes between 5.1 and Stereo audio?

MymsMan said:
I am wondering about the possibility of using the possibility of using the ffmpeg copy and concat technique as a replacement for nicespclice on the humax.
Click to expand...
That sounds like a very good thing to explore to me.
 
florca said:
After quite a few "learning opportunities" I have a basic BASH script which works on both my Raspberry Pi and the Humax to remove the Stereo sections from a mixed 5.1 and Stereo .ts (.m2ts) Humax recording. It's my first ever BASH Script ...
Click to expand...
Suggestions (apologies for not addressing the media processing aspects, which I take on trust):
  • introduce your script to http://shellcheck.net;
  • the idiom echo "text text"$var"text text" is a bit strange and probably not what you want - instead echo "text text${var}text text" or printf "text text%stext text\n" "$var" ;
  • consider using /bin/sh, instead of bash, as the new ffmpeg is already quite hefty program and you won't want 2 bashes running as well on the Humax; I think only your arithmetic will be affected, but shellcheck will help;
  • check the functionality of the sh, sed, grep, uniq that come in the CF's /bin, as these are the only guaranteed versions (though a package could be made to depend on eg coreutils).
 
Last edited:
Here are some suggestions:

This could be a possible replacement for the 'sed, uniq, sed, grep, grep, sed' section after the initial ffmpeg command.
Code: 
#!/bin/sh

/mod/bin/awk -F, '
    BEGIN {c=-1} 
    (c != $2) {
        if (last) {
            gsub(/\./,"",last);
            print last;
        }
        gsub(/\./,"");
        print $0; 
        c=$2
    }
    {
        last=$0;
    }
 ' temp.pts > temp.ptw

As an alternative, this includes the above plus the functionality of the following while loop and sed command in the original code.
Code: 
awk -F, '
    BEGIN {c=-1; f=0}
    (c != $2) {
         c=$2
         if (f == 0) f=$1;
         printf ("%f,%d\n", $1 - f, $2);
    }
 ' "root".pts > "root".ptx
Note, similar to 'uniq', 'awk' is not available in the CF's /bin but is available in either of the busybox or coreutils packages.

Bit of a style thing, in the ffmpeg commands where you are redirecting both stdout and stderr to the same file using 1>>\""$root".log\" 2>>\""$root".log\", it would be more usual to use this as a replacement >>\""$root".log\" 2>&1.
 
xyz321 said:
As an alternative, this includes the above plus the functionality of the following while loop and sed command in the original code.
Click to expand...
Well that's certainly much neater! Must admit I became rather carried away with the sed / uniq / grep stuff and should have stood back and thought harder about direct processing of the .pts file. Working nicely with your second version and also doing some other general tidying and making more compliant with non-bash shells. Hopefully another, rather cleaner, version tomorrow - quite enjoying learning something new :)
 
I mistakenly thought it was included as part of coreutils but it is in the busybox or gawk packages. I used busybox for testing.
 
/df said:
It does introduce a dependency on the gawk package for the Humax CF, which isn't necessarily a bad thing
Click to expand...
Providing you recognize you have a dependency and can declare it in the dependencies it is better to use a tool rather than reinventing wheels,
But too often you only realize something is an add on when someone else fails when trying to run your packge
 
MymsMan said:
...
But too often you only realize something is an add on when someone else fails when trying to run your packge
Click to expand...
Old hands will be familiar with this, but you have to distinguish between the busybox that comes in the CF's /bin and the installable busybox package.

For instance, xargs is in the latter but not the former; potentially more confusing, the ash(1) shell in the package is differently configured from the CF one (eg, the type built-in is in the package version but not the CF), so that shell scripts tested with /mod/bin/busybox in the PATH can fail when run, eg, at boot time when only /bin is available.

I posted a side-by-side comparison here.
 
Last edited:
Here's a version incorporating some of the suggestions (thanks..) plus some general tidying. One slightly strange thing which may inhibit adoption of the ffmpeg cut / concat approach for DetectAds is that I was greeted this morning by an error message on the Humax to say that automatic decryption had failed for the new file - which wasn't encrypted anyway, but there's obviously a flag which makes the CF believe that the file needs decryption?
The new version runs a couple of minutes slower on the same sample file (30mins vs. 28 for a full detect / cut / splice) but the overall improvement in readability and reduction in spurious work files make it a better option I think. It also now runs cleanly in the native shell as well as bash.
Bash: 
#!/bin/sh
#------------------------------------------------------------------------------------------------------------
# Shell script to determine whether a Humax .ts media file contains 5.1 audio and if so remove any Stereo section(s)
# Input parameters: $1 - Name of a .ts media file, with or without the .ts extension
# Output: the same .ts file, cut and spliced to remove any Stereo sections if they exist. Exits with file untouched if no 5.1 audio at +240 seconds
#                  .log file containing STDERR output from ffmpeg / ffprobe and other progress info
#        
# Calls / depends on: ffmpeg / ffprobe to analyse and edit the programme
#                     awk Linux command
#                    
#---------------------------------------------------------------------------------------------------------------

print_banner () {
echo "Shell script to determine whether a Humax .ts media file contains 5.1 audio and if so remove any Stereo section(s)"
echo "Input parameter:  Name of a .ts media file, with or without the .ts extension"
echo "Output:           the same .ts file, cut and spliced to remove any Stereo sections if they exist."
echo "                  .log file containing STDERR output from ffmpeg / ffprobe and other progress info"
echo " "
echo "Exits with file untouched if no 5.1 audio at +240 seconds or no Stereo section(s)"
echo " "        
echo "Calls / depends on: ffmpeg / ffprobe to analyse and edit the programme"
echo "                    awk Linux command"
echo " "
echo " "
echo "***WARNING*** This will overwrite the existing .ts file with no backup - use with caution! ***"
echo " "
}

cleanup () {
echo "*** Copy of "$root".ptx - the derived list of sections" >>"$root".log
cat "$root".ptx >>"$root".log
echo "*** Copy of "$root".sh - the ffmpeg cut-and-splice list" >>"$root".log
cat "$root".sh >>"$root".log
rm "$root".pt*
rm "$root".sh
ls -l "$root".* >>"$root".log
date >>"$root".log
echo "----- *** -----"
ls -l "$root".ts
date
echo ""$@"" >>"$root".log
echo ""$@" See \""$root.log"\" for more details"
}

# MAIN - Start here...

# If no input then exit with banner

if [ -z "$1" ]; then
print_banner
exit 5;
fi

# Derive basename of input file
root="${1%.*}"

# Check we have the required files and exit with banner and message if not found!

if [ ! -f "$root".ts ]; then
print_banner  
echo "**ERROR**       Required file \""$root".ts\" not found!"
exit 5;
fi

# Look at the number of channels four minutes into the programme and exit if less than three channels of audio

channels=$(ffprobe -hide_banner -loglevel warning -read_intervals 240%+#1 -select_streams a:0 -i "$root".ts -show_frames -show_entries frame=channels -of csv=nk=1:p=0 2>>"$root".log);

if [ $channels -lt 3 ]; then
echo "\""$root".ts\" only has "$channels" audio channels at +4 minutes - exiting with no action"
date
ls -l "$root".ts
exit 1
fi ;

echo "\""$root".ts\" has "$channels" audio channels at +4 minutes - processing file to remove any Stereo sections"
date
ls -l "$root".ts

#
#  We have multi-channel audio!
#
#  Initialise a log file to capture progress and ffmpeg errors


date >"$root".log
echo "File has "$channels" audio channels at +4 minutes - processing file to remove any Stereo sections" >>"$root".log
ls -l "$root".* >>"$root".log


# Initialise the script file for cutting and splicing work

echo "#!/bin/sh" >"$root".sh

# Generate a .pts file with a list of the number of audio chanels in each frame

ffprobe -hide_banner -loglevel warning -select_streams a:0 -i "$root".ts -show_frames -show_entries frame=pkt_pts_time,channels -of csv=nk=1:p=0  1>"$root".pts 2>>"$root".log;

# Select the start and end of each section from the frames

awk -F, '
    BEGIN {c=-1; f=0; pp=0}{
    if (c != $2) {
         pc=c;
         c=$2;
         if (f == 0) {
          f=$1;
          printf ("%f,%d\n", $1 - f, $2);
          } else {
            printf ("%f,%d\n", pp - f, pc);
            printf ("%f,%d\n", $1 - f, $2);
            }
    }
    else
     {
     pp=$1;
     }
    }
' "$root".pts > "$root".ptx

# Generate the Cut and Splice file

phase="Starts"
concat="concat:"
count6=0
count2=0

while IFS=, read -r pts chan; do
  if [ $phase = "Starts" ]; then
  spts="$pts"
  schan="$chan"
  phase="Ends"
  else
   phase="Starts"
    if [ $chan -gt 2 ]; then
     count6=$(($count6+1))
     echo "ffmpeg -hide_banner -loglevel error -stats -i \""$root".ts\" -c:v copy -c:a copy -ss "$spts" -to "$pts" -async 1 \""$root""$count6".m2ts\" >>\""$root".log\" 2>&1" >>"$root".sh
     concat=""$concat"\""$root""$count6".m2ts\"\\|"
     else
     count2=$(($count2+1))
    fi
  fi
done < "$root".ptx


# If there are no Stereo sections then just clean up and exit, leaving file untouched

  if [ $count2 -eq 0 ]; then
  cleanup "No Stereo sections found - exiting with no changes to file."
  exit 1
  fi

# If input ends on a 6 channel segment then include this to end

  if [ $schan -gt 2 ]; then
  count6=$(($count6+1))
  echo "ffmpeg -hide_banner -loglevel error -stats -i \""$root".ts\" -c:v copy -c:a copy -ss "$spts" -async 1 \""$root""$count6".m2ts\" >>\""$root".log\" 2>&1" >>"$root".sh
  concat=""$concat"\""$root""$count6".m2ts\"\\|"
  fi

# Concatonate all the segments together...

echo "ffmpeg -hide_banner -loglevel error -stats -i "${concat%??}" -y -c copy \""$root".m2ts\" >>\""$root".log\" 2>&1" >>"$root".sh;

# And go do the edit

echo "***Timestamp at the end of ffprobe and PTS scan processing as we start the Cut and Join..." >>"$root".log
date >>"$root".log

sh "$root".sh

# Overwrite the original .ts file with the spliced .m2ts version

  [ -f "$root".m2ts ] && \
  mv -f "$root".m2ts "$root".ts

# Clean up the work files and exit..

rm "$root"?.m2ts
cleanup "Exiting cleanly - "$root".ts has been cut into "$count6" sections with 5.1 audio and spliced"
exit 0
 
Last edited:
florca said:
I was greeted this morning by an error message on the Humax to say that automatic decryption had failed for the new file
Click to expand...
Are you creating a new .ts without any sidecar files?

WebIF auto-decrypt scans for recordings that have not been decrypted, which is signalled by the relevant flags in the .hmt file. If you use sidecar to regenerate the .hmt, or just plain copy the sidecars from the input recording to the output, your output .ts will not then trigger auto-decrypt.

Alternatively, ensure your working folder is somewhere auto-decrypt won't touch, such as a folder prefixed "[".
 
I replaced the .ts file into the same directory but had left the HMT / NTS / THM files untouched and also in the same directory - I realise the HMT and NTS files probably need to be rebuilt with the sidecar utility but hadn't done this yet - is there something that recognises that the .ts file has changed??
 
Hmm.. This throws up the need for a bit of a rethink, certainly for the Cut / Concat part.
The new .ts file reports "1" (encrypted I assume?) from stripts -E
It won't play on the Humax (it shows a message "Channel encrypted or not available") - don't know why I didn't try to play it from the Humax before...
It will play perfectly using VLC on a Windows 10 PC with sound in sync
If I convert the new file to MP4 with ADTS sound on a PC using vlc and then copy back to the Humax it will now play, but with the sound markedly out of sync.
The .ts file won't play on a Windows 10 PC using "MS Films & TV" - It gets the 1st frame and then freezes The MP4 file will play but with sound out of synch.
So more work needed I think :(
 
florca said:
The new .ts file reports "1" (encrypted I assume?) from stripts -E
Click to expand...
Curious. ffmpeg can't have been working on an encrypted file (can it?!?), and yet
florca said:
Channel encrypted or not available
Click to expand...
...implies that the HDR-FOX also thinks it is encrypted (or at least that the .hmt is out of sync).

Try this:
  1. Run the process on a test recording;
  2. Delete the original .ts;
  3. Rename the process output .ts to match the existing .hmt.
Are you quite sure the ffmpeg output is exactly the same format, ie M2TS? Might be useful to post the output of Media Info (or similar).
 
florca said:
don't know why I didn't try to play it from the Humax before...
Click to expand...
One reason all of the current processes that modify the .ts file leave the original in an /originals folder or move it to the dustbin is so that you can repeat tests and investigate problems

Also deleting large files can actually cause picture breakup during recordings and playback so it is best to to use the dustbin so that actual deletion occurs at a quiet time. To observe this try manually deleting the dustbin during a recording (but not SWMBO's favourite)
 
Moving forward, but still some work to do...
Running the Cut/Concat output file through VideoReDo's "Quick Stream Fix" produces a message "57 Audio Resync Frames Removed" and a file which will now play quite happily with perfect audio synch on the Humax and indexes / displays properly after running Sidecar against it. So need to figure out exactly what ffmpeg options "Quick Stream Fix" is using (they seem to use ffmpeg under the covers for most operations) and replicate. The now playable Cut/Concat'd .ts file still shows "1" (encryted) from stripts -E, even after new HMT files were created.
Black Hole said:
Try this:
  1. Run the process on a test recording;
  2. Delete the original .ts;
  3. Rename the process output .ts to match the existing .hmt.
Click to expand...
In effect this is exactly what I was doing with the original file and resulted in the unplayable .ts . The now-playable and newly named "Quick Stream Fix'd" .ts, with a new .hmt generated by Sidecar seems to work 100% fine on the Humax although missing the Synopsis etc.
MymsMan said:
One reason all of the current processes that modify the .ts file leave the original in an /originals folder or move it to the dustbin is so that you can repeat tests and investigate problems
Click to expand...
I did take (and still have) a copy under a modified name before starting any tests for exactly that reason. But will definitely change the script to keep a copy...
xyz321 said:
What does hmt <rootfilename> | grep ^Flags return?
Click to expand...
The newly created HMT for the playable Cut/Concat'd file shows: Flags: HD,New,Unlimited Copies,Shrunk,
The HMT for the original file shows: Flags: HD,Unlimited Copies,ODEncrypted,Shrunk,Deduped,Addetection,
 
You must log in or register to reply here.
Back
Top