Migrate (to HDR FOX-T2)

[sceedy]

Vn 0.8 broke with the kernel update between debian 10.3.0 and 10.4.0 but nobody, other than my family, noticed. It complained about a security problem tidying up the memory after the core processing had completed successfully. Out of habit, I had written the memory management the way m$ recommended since windows 98. This results in the compiler/optimiser creating an automatous thread that “free”d memory and set the pointers to NULL. I agree with the security concern that an automatous thread can be added to a program and thus be malicious, so it’s setting of pointers to NULL can change the behaviour of a program after it has done validation of the buffer referenced by the pointer. This resulted in Vn 0.8.1. The archive being published is Vn 0.8.2. Attachment MigrateVn0.8.2sizes.txt contains links to my OneDrive where the archives as set up for read only download.

New in Vn 0.8.1 :-
Audio – If a recoded file had a mono audio stream, a pseudo stereo output is produced by setting the left and right channels to the input audio. If listening over headphones, the sound coming from only one side seems strange.
Video – Re-multiplexing of a VOB set (no processing of the menu stream is involved, so I can’t claim it to be a DVD title) can be initiated from the file selection by selecting any VOB in the set. If selected this way, the input stream(s) language code metadata is translated to the DVB equivalent in the output. Note that ffmpeg currently does not do this, it does not have any language metadata in it’s output.

New in Vn 0.8.2 :-
Humax native cli static executable.
Health warnings :-
It is “proof of concept” so contains a lot of instrumentation.
If it receives a trap it outputs an error message containing the execution logical address as minimum extra information over the standard linux error message.
It toggles between 2 files, load0.log & load1.log, to record the information that is monitored by the kernel that can cause a restart without producing a crash dump.
It creates a file Page9G, if it has not been created by itself previously, to use as a swap file during recoding of an input file. The extreme bloat is due to using ffmpeg’s version of libav as the recoding engine.
My confidence checks take 20 minutes on my laptop (was mid range 3 years ago). The remultiplexing check takes 25 minutes and the recoding check takes 1 hour 20 minutes on an HDR-Fox T2 so it is considerably slower than the pc (windows and linux) versions.
Release testing highlighted a bug that does not affect operation. The scar is to do with the cpu load displayed to the user. Elapsed time is calculated as time now - time at start of execution. If you are unlucky enough for the time to wrap during execution, the elapsed time calculation produces nonsensical results, resulting in loads of far greater than 100%.

 

[sceedy]

I have finally reached the point where Vn 0.9 passed it’s automated testing. Automating the testing is just a case of splitting into 18 elapsed hours on the target. Automated results checking has a lot more ‘manual’ in it. Where the bit patterns differ between targets (all first examples of outputs have been checked by watching on a Humax) it needs to be identified that the differences are either processing time or floating point values. This time I have checked that only the lsb 3 bits are different.

Attachment MigrateVn0.9sizes.txt shows the sizes of the various ‘zips’ with download links to my one drive.

Migrate and Sidecar both index transport streams for the Humax. AV2HDR-T2 is windows only GUI version of Sidecar that uses tsMuxer as it’s re-multiplexing engine. Migrate has a GUI version for windows, 64 bit linux and 32 bit linux. A cli only version is available for windows, 64 bit linux, 32 bit linux and Humax. GUI versions can also be run from the cli. All versions use FFmpeg as it’s re-multiplexing and/or transcoding engine.

Vn0.9 contains subtitle processing. It uses FFmpeg Vn 4.4’s libraries. This version was chosen because it contains the fixes for subtitle problems caused by subtitles not always being present. My code contains functionality not in FFmpeg. DVD language metadata has been offered to FFmpeg on ticket #9158. Automatic correction of input language metadata to the correct variant of ISO 693 for DVB (FFmpeg just copies the code from input to output). Transcoding of non-graphical subtitles (FFmpeg just gives re-multiplexing of graphical subtitles).

I now intend to monitor FFmpeg for when it understands new formats etc. and create updates for these. Obviously bug reports will be dealt with.

 

[sceedy]

Attachment MigrateVn0.9.0.1sizes.txt gives links to download from my OneDrive. This version was supposed to be an Easter present, but Ubuntu specific problems delayed this until orthodox Easter.

This version upgrades the transcoding/multiplexing engine to ffmpeg Release 5.0.1 libraries and wxWidgets to Release 3.1.6.
It supports no new input file formats over Vn0.9.
The Linux GUI versions no longer need to be invoked from Terminal.
The Windows GUI versions now have the Icon built from a source file.
The “Master Plan” is to rebuild for new ffmpeg releases (to be able to transcode new input file formats) and picking up any third party library bug/stability fixes along the way.

 

[sceedy]

Migrate Vn0.9.1.0 has been created as a result of ffmpeg version 5.1 being formalised.

New in this version of Migrate :-
Windows 64 bit versions (as a result of the Windows 11 roadmap saying that support for 32 bit executables being removed at some unspecified future time)

Internal changes :-
(As stated above) Transcoding engine updated to ffmpeg version 5.1 (i.e. all the input formats recognised by this ffmpeg version are available for transcoding)
wxWidgets updated to version 3.2.0
Build scripted (after review by former colleague into skill level required to reproduce the build. I considered it too high. The version jump reflects this.)

Attachment MigrateVn0.9.1.0sizes.txt contains various links to OneDrive containing this version.

 

[sceedy]

Migrate Vn 0.9.1.1 is now available i.e. it’s automated testing has all passed.

This Migrate version uses ffmpeg release version 5.1.1 libraries as it’s transcoding/multiplexing engine (i.e. bug fixes for ffmpeg version 5.1 libraries). It also contains updates to allow Xcode compilation.

This Migrate version was originally intended as a private release, to a single beta tester, but was overtaken by the release of a new ffmpeg version on Wednesday (31 August 2022) at around 10PM. As a result macOS support is a “work in progress”. Currently only Monterey is supported.

Attachment MigrateVn0.9.1.1sizes.txt contains a list of the sizes of compressed files. Links to the .7z files on my OneDrive are given.

Attachment “macOS Confidence Check.png” (shown bellow) shows the GUIs after the initial confidence check passed i.e. before automated testing was started. The QT5 GUI is on the left and the wxWidgets 3.2.0 GUI is on the right.

 

[sceedy]

Version 0.9.1.2 was released privately to my beta testers. It contains the functionality updates that were in progress when 0.9.1.1 was released (in response to the latest ffmpeg release). They are happy so I am publishing publicly.

Attachment “MigrateVn0.9.1.2sizes.txt” contains links to the updated executables (excluding those which only differ, at the binary level, due to the version number updating).

The MacOS version now runs on High Sierra (thumbnail too big to upload) and Big Sur in addition to Monterey. This is as far back as I can go without the app no longer running on Monterey.

The Windows32 version has lensfun functionality enabled.

 

[sceedy]

ffmpeg version 5.1.2 was released on Sunday (25 September 2022 12:46). The ffmpeg update was mainly bug fixes. As a result Migrate has been updated to version 0.9.1.3. Automated testing has now passed so attachment “MigrateVn0.9.1.3sizes.txt” contains links to my code in my OneDrive.

Note: My OneDrive is now more than 95% full so, to make room, versions 0.9 and earlier are now only available as source code.

 

[sceedy]

Ffmpeg Vn 6.0 was released on 27th February 2023 causing Migrate to update to Vn 0.9.1.4 (using the new version of ffmpeg as the transcoding/remultiplexing engine). The Windows XP version does not contain librsvg (rust for 32 bit Windows states clearly that Windows 7 is the minimum supported version in the documentation. I am finding it interesting, in the sense of the old Chinese curse “May you live uninteresting times”, to try and make libstd XP compatible so I have not succeeded ?yet?). The MacOS versions have not been tested on Ventura (My Intel hardware will not run it. My m1 tester does not want to upgrade). All versions now have support for the “Chinese” video standard (see updated user guide).

As usual, links to the new build are in attachment “MigrateVn0.9.1.4sizes.txt”.

I apologise for the time this update took. My first problem was getting consistent test results from all supported platforms (The ffmpeg external libraries now insist on a version of meson that is not present in the stable version of debian – backport version used. MacOS homebrew core made changes that needed to be replicated in all targets, also the cmake update caused the MacOS build to fail several times.) Then my beta testers could not reproduce the binaries for the target they were interested in from the sources provided (one file not updated in the archive provided to them).

 

[sceedy]

FFmpeg version 6.0.1 has been released prompting MigrateVn0.9.1.5.

As usual the download links are in attachment “MigrateVn0.9.1.5sizes.txt”.

The changes since the last version are minor.

All versions (including windows xp) have librsvg in the build.

I raised ticket #10651 against FFmpeg because, unlike the main development path (ffmpeg-git), the release is only compatible with an old API of libjxl, and the source for that version is no longer available, so libjxl has not been built into the version of the libraries I used (FFmpeg version 7.0 is in the pipeline which should not have this problem).

Debian 11 (Bullseye) has become “old stable” and the old linux version does not work in debian 12 (Bookworm) or debian 13 (Trixie). This version does.

MacOS has updated several times. I have tested against all the Intel versions. I do not own apple silicon hardware so testing has only been on Monterey for apple silicon.

The title bar of the GUI now displays the FFmpeg version used (by reading it from the object files of the library used).

The Humax executable folder now includes an FFmpeg executable. Off topic : I did actually succeed in producing a working executable, with a main using ulibc compiled on a Humax box, using shared objects produced on debian using libc. However, the notes ran to 20 pages on the use of shared object with Tread Local Storage (i.e. compiled on debian and using libc) with a main that does not. I tried to get a former college to produce something that works. The exercise led to the conclusion that it was not practically possible to follow the notes. Hence the completely static FFmpeg executable being included.

 

[sceedy]

FFmpeg Vn 7.0 release has prompted Migrate Vn 0.9.1.6. Libjxl is back in. The last version of Migrate was based on ffmpeg Vn 6.0.1 so all the non-hardware accelerated decoders introduced in versions 6.1 and 7.0 are available to use to transcode input files.

I was asked to add avisynth and vapoursynth into my ffmpeg library build. This is experimental i.e. no confidence check or automated testing. My plan going forward is for him to confirm that my CLI and GUI need no changes (or make them in my next release), get him to provide me with some use cases for testing (and also for some runs under the debugger to confirm no internal changes are needed in the ffmpeg “plugin” or make them, on their own, in a release) and then work out patches so the default avisynth plugin’s functionality is available without shared objects. Note that vapousynth does this via configuration options.

Attachment “MigrateVn0.9.1.6Sizes.txt” contains links to the usual archives.

 

[sceedy]

FFmpeg Vn 7.0.1 release has prompted Migrate Vn 0.9.1.7. It took longer than usual because the bug fixes to the cairo library broke my build and it tookme a week to get the functionality consistent across all targets. The last version of Migrate was based on ffmpeg Vn 7.0 so this library version contains bug fix updates. This solves all known transcoding crashes.

I was asked to add avisynth and vapoursynth into my last ffmpeg library build. This is still experimental i.e. no confidence check or automated testing because I have still do not have use cases for testing. My plan going forward is for him to confirm that my CLI and GUI need no changes (or make them in my next release), get him to provide me with some use cases for testing (and also for some runs under the debugger to confirm no internal changes are needed in the ffmpeg “plugin” or make them, on their own, in a release). The default avisynth plugin’s functionality is now available without shared objects.

Attachment “MigrateVn0.9.1.7Sizes.txt” contains links to the usual archives.

 

[sceedy]

FFmpeg Vn 7.0.2 release has prompted Migrate Vn 0.9.1.8. The last version of Migrate was based on ffmpeg Vn 7.0.1 so this library version contains bug fix updates. I now have less hardware to run automated testing and less time in general hence the long publishing delay, but nobody seems bothered.

I was asked to change the 64 bit windows python version to 3.8.10 as this was the latest version for Windows 7 with an installer. If your VapourSynth script sticks to video processing the underlying python version is irrelevant and should work cross-platform. However, there is a full python interpreter underlying VapourSynth so if you want to compromise your security by doing other things, you can.

Attachment “MigrateVn0.9.1.8Sizes.txt” contains links to the usual archives.

The publishing delay was so long that FFmpeg Vn 7.1 release has prompted Migrate Vn 0.9.1.9. This has now started automated testing. I had hoped to make the internal changes for Migrate Vn 0.9.2.x series but FFmpeg Vn 7.1 release happened before I had time.