Cross compiling (anything) from Debian Wheezy to the HDR

njm

Member
I'm having trouble getting anything that I've cross compiled for mipsel on a Debian Wheezy x86_64 host to run on the HDR. I'm following the instructions on the Debian wiki here.
Code:
# add the embedian archive keyring
apt-get install emdebian-archive-keyring
 
# embedian toolchain packages
echo "deb http://www.emdebian.org/debian/ squeeze main" >> /etc/apt/sources.list.d/emdebian.list
 
# Squeeze mirror for embedian toolchain dependencies
echo "deb http://ftp.uk.debian.org/debian/ squeeze main" >> /etc/apt/sources.list.d/emdebian.list
 
apt-get update
apt-get install gcc-4.4-mipsel-linux-gnu
 
# ...which installed these packages:
#
# binutils-mipsel-linux-gnu (2.20.1-16)
# gcc-4.4-mipsel-linux-gnu-base (4.4.5-8)
# cpp-4.4-mipsel-linux-gnu (4.4.5-8)
# libc-bin-mipsel-cross (2.11.2-10)
# linux-libc-dev-mipsel-cross (2.6.32-30)
# libgcc1-mipsel-cross (1:4.4.5-8)
# libc6-mipsel-cross (2.11.2-10)
# libgomp1-mipsel-cross (4.4.5-8)
# gcc-4.4-mipsel-linux-gnu (4.4.5-8)
# libc-dev-bin-mipsel-cross (2.11.2-10)
# libc6-dev-mipsel-cross (2.11.2-10)
 
mipsel-linux-gnu-gcc helloworld.c -o helloworld

Later, on the HDR...
Code:
humax# ./helloworld
/bin/sh: ./helloworld: not found
Any ideas why this might not be working? Am I using the wrong C library? If so, which should I be using.

If I can cross compile anything successfully I'll update the notes on this page of the wiki:
http://wiki.hummy.tv/wiki/Compile_Software_for_the_Humax

EDIT: I must be using the wrong C library:
Code:
humax# ldd helloworld
checking sub-depends for 'not found'
        libc.so.6 => not found (0x00000000)
        /lib/ld.so.1 => /lib/ld.so.1 (0x00000000)
 
Yes, wrong C library. The Humax uses uclibc.

My main development environment is a Solaris x86 machine running Debian in a branded zone. It's so long since I built it I can't remember everything but I'm using the toolchain that Humax provide on their open-source downloads page (http://www.humaxdigital.com/uk/opensource.php - crosstools_hf-linux-2.6.18.0_gcc-4.2-11ts_uclibc-nptl-0.9.29-20070423_20090508.tar.bz2 should give you everything you need. )

Extract under /opt/toolchains/ and set up your path appropriately.
 
One more thing: did you have to do anything special to get autoconf configure scripts to use this toolchain? I'm trying to configure libsndfile like so:
Code:
./configure --build=mipsel-linux-uclibc --host=x86_64-linux-gnu --enable-largefile --prefix=/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/
...but configure still seems to use my host compiler
Code:
...
checking for version of x86_64-linux-gnu-gcc... 4.7
checking if x86_64-linux-gnu-gcc accepts -Wextra... yes
checking if x86_64-linux-gnu-g++ accepts -Wextra... yes
checking if x86_64-linux-gnu-gcc accepts -Wdeclaration-after-statement... yes
checking if x86_64-linux-gnu-gcc accepts -Wpointer-arith... yes
checking if x86_64-linux-gnu-gcc accepts -funsigned-char... yes
...
 
Code:
humax-dev# (105) echo $PATH
/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/toolchains/crosstools_hf-linux-2.6.18.0_gcc-4.2-11ts_uclibc-nptl-0.9.29-20070423_20090508/bin
Code:
CC='mipsel-linux-gcc' \
CFLAGS="-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64" \
./configure \
    --prefix=/mod \
    --host=mipsel-linux \
    --target=mipsel-linux
 
I've run into problems a little further down the line building libsndfile. Configure worked ok, and I can see in the Makefile that the correct compiler is being used:
Code:
CC = mipsel-linux-gcc
However, make falls over when it tries to build some of the examples:
Code:
$ make
  CCLD  make_sine
../src/.libs/libsndfile.so: undefined reference to `lrintf'
../src/.libs/libsndfile.so: undefined reference to `lrint'
collect2: ld returned 1 exit status
make: *** [make_sine] Error 1
As far as I know these functions are part of libm, which is in one of the directories the compiler looks in for libraries:
Code:
$ mipsel-linux-gcc -print-search-dirs
install: /opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//lib/gcc/mipsel-linux-uclibc/3.4.6/
programs: =/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../libexec/gcc/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../libexec/gcc/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//libexec/gcc/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//libexec/gcc/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//libexec/gcc/mipsel-linux-uclibc/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//lib/gcc/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//lib/gcc/mipsel-linux-uclibc/:/usr/libexec/gcc/mipsel-linux-uclibc/3.4.6/:/usr/libexec/gcc/mipsel-linux-uclibc/:/usr/lib/gcc/mipsel-linux-uclibc/3.4.6/:/usr/lib/gcc/mipsel-linux-uclibc/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/bin/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/bin/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/bin/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/bin/
libraries: =/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../lib/gcc/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../lib/gcc/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//lib/gcc/mipsel-linux-uclibc/3.4.6/:/usr/lib/gcc/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/lib/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/lib/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/lib/mipsel-linux-uclibc/3.4.6/:/opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131//lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/lib/
Code:
$ ls /opt/toolchains/crosstools_sf-linux-2.6.12.0_gcc-3.4.6-20_uclibc-0.9.28-20050817-20070131/bin/../lib/gcc/mipsel-linux-uclibc/3.4.6/../../../../mipsel-linux-uclibc/lib/ | grep libm
libm-0.9.28.so
libm.a
libm_pic.a
libm.so
libm.so.0
I added -Wl,--verbose to CFLAGS to get verbose output from ld (attached). Looks like ld found libm but libm doesn't contain lrint and lrintf.

When you compiled libsndfile did you compile it on the HDR or did you cross compile it?
 

Attachments

  • ld_output.txt
    15 KB · Views: 4
After a little digging I realised that the version of libm on the HDR is different to the version of libm that comes with Humax's cross compilation toolchain. The HDR has libm-0.9.29.so and the toolchain comes with libm-0.9.28.so. The toolchain's version of libm doesn't export lrint or lrintf whereas the HDR's version of libm does (see attached). I'll try copying the HDR's libraries over to my build host.
 

Attachments

  • libm-0.9.28_symbols.txt
    9.8 KB · Views: 3
  • libm-0.9.29_symbols.txt
    8 KB · Views: 3
Over the years I've ended up doing just that for lots of libraries and header files. My development environment isn't exactly clean any more! I built libsndfileon my development server with the cross compiler. I tend to do that for larger things but for small code that I'm developing I'm as likely to build it on the Humax. Getting the compiler working on the Humax was too much pain to stop using it : )
 
I've managed to cross-compile libsndfile now, but what a headache! After copying across libm I was still getting a similar error - no lrint or lrintf. It turned out the linker was using the static libraries in libm.a, which comes with the Humax toolchain and doesn't include lrint or lrintf, because the libm.so files from the HDR had no read permissions set for group or other users! Once I fixed that I also had to copy across libc, libdl, libpthread, and libsqlite3.
 
If you read the references above in this thread, there is a box native compiler (this community) and a cross-compilation toolset (OEM) available. For almost all situations these will satisfy your needs/wants. You only need to read on if you are insane(?) enough to still want to cross-compile using other tools.

The first question you need to ask yourself is which environment you wish to build. Most new machines are windows but running a linux virtual machine with VirtualBox, QEMU … is relatively simple. The HDR-Fox T2 is a linux machine so you will probably want to do linux specific stuff. Choosing linux means you can try stuff on your host machine (the one with the build chain on) to decouple the linux specific from the target (the one you run the executable binary on) specific part of any problem you encounter. The next choice is which “flavour” to use, which is the subject of lots of discussion elsewhere. My belief is that Ubuntu is better for people who just want to use it, debian is more useful for developers and red hat is for linux fundamentalists.

I chose debian. I wanted to use gcc version 9.2 or greater which is “pencilled in” on debian 11. Lockdown happened so I made more progress than I anticipated, so I prototyped on debian 10.

Debian provides cross tools with a triplex prefix on the standard tool names for the cross versions. The current mipsel-linux-gnu-gcc(g++) version is 8.3.0 which has compiler libraries compiled for a MIPS32 revision 2 processor. However the HDR-Fox T2 has a MIPS32 revision 1 processor so you need to recompile the default compiler libraries (gcc8.3.0compilerlibs.zip). This will still not create an executable that will run because the default glibc invoked by the compiler is also compiled for the wrong processor but, more importantly, needs a much younger kernel.

The next choice you need to make is libc. The choices are ulibc and glibc. Humax chose ulibc for valid reasons so it has to be present on the target. If you want a different libc then you can only build static because it is not possible to replace the ulibc shared library on the target without affecting the normal operation of the box.

The ulibc vs glibc debate rages elsewhere. The ethos of ulibc is to be as thin (as little code between the application and the kernel) as possible. The ethos of glibc is to provide as much functionality as possible, which necessitates it being more bloated than ulibc. If you read the linux kernel man pages you often come across entries like “a user space … was provided in glibc version … which became a kernel call in version …” but the wording is not consistent enough for a definitive goggle search to find them all. I have found no mention of ulibc providing compatibility code (but I am willing to be educated) so I took a closer look at glibc.

The glibc documentation clearly states that each version supports a minimum kernel version and a little digging finds the rationale. I believe this holds true for the 64 bit machine’s kernel ABI but is questionable for a 32 bit machine’s kernel ABI e.g. the HDR-Fox T2, so I decided to experiment with the glibc version pencilled in for debian 11.

The first problem is that purely static glibc builds are no longer supported despite configure still recognising the option even though it is removed from the documentation. The “trick” found on the internet is to do a “vanilla” build (purely to create the includes… in the build folder), remove all the shared objects and libraries, and then perform a second configure with the options that you actually want (Note: if you do a build clean after the second configure subsequent build will fail big time). Once you can do a static build it is then necessary to hack the configure output (BuildFromSource.pdf) to bypass the primary minimum kernel version enforcement mechanism and source changes (HumaxlibcSource.zip) to include the compatibility code I have found and bypass the 2 secondary enforcement mechanisms I have found before producing a version I am happy with (HumaxlibcBinary.zip).

This is still not binary compatible with the other libraries available from debian as libm has a different ABI in the default glibc (I was working towards debian 11) so you need a compatible version (HumaxlibmBinary.zip).

This gets you to the point where you can produce executables that only need standard c(++) functions. There is not much difference in the instruction set between Mips32 version 1 and version 2 but you are still playing Russian roulette with “Illegal instruction” when you include other pre-built libraries. I played “whack a mole” by always producing a link map and then using mipsel-linux-gnu-objdump -d to find the actual instructions in the executable. Document Number: MD00086 "MIPS® Architecture for Programmers Volume II-A: The MIPS32® Instruction Set Manual" "Appendix A: Instruction Bit Encodings" figures identify the instructions that cause grief. These can be searched for in the objdump output that gives you and address. The address can be searched for in the map which gives the archive and the object file within the archive.

If you want to debug on the target and not use “dwarf 4” as an expletive you will also need an up to date gdb (gdb.7z or gdb.zip). My version provides the native and server executables but not the host executable for remote debugging. My experience is that there is no advantage in remote debugging over native debugging via telnet with putty as the telnet client.

Attachment sizes.txt gives the sizes of the archives referenced above together with a link to my OneDrive with read only access where you can find them.

Edited 29th November 2020

I have updated sizes.txt. It now contains links to arm, i686 and m68k binary archives compatible with linux kernel version 2.6.22 produced for the communities that helped me with generic linux information whilst I produced the HDR-Fox T2 (Humax – kernel version 2.6.18) version. In the process of generating the binaries for other architectures I found 2 more areas of compatibility code in older libc versions that have been reinstated in my sources. An updated Humax binary archive has been produced based on these source updates. The <architecture>libcSourceExtras.zip files are the build folders containing the extra files necessary to bypass the secondary minimum kernel version enforcement mechanism(s).

WhackAMole.c.txt is the source for a program that helps automate the activity described as such above.
 

Attachments

  • BuildFromSource.pdf
    109.1 KB · Views: 4
  • WhackAMole.c.txt
    11.9 KB · Views: 0
  • sizes.txt
    1.2 KB · Views: 0
Last edited:
At the time of writing debian 11’s freeze was a few days ago. This means it is in testing prior to release so is still “flaky”. I have installed the 64 and 32 bit versions on a VirtualBox virtual machine where a timeout returns control to the login, however, increasing the physical memory allocation to 8G or above appears to stop this. This gives me access to gcc version 10.2 (see announcements.txt for full details).

As libc links in part of the compiler libraries, this necessitates an updated version for the new compiler in debian 11. Attachment sizes.txt contains links to anything referenced in my previous post that needed updating. Attachment BuildFromSource.pdf contains tweaks to the build from debian 10 to debian 11.

Off topic : I am looking forward to not having to disassemble linked binaries to confirm that intended function has not been optimised out when non-windows versions have test failures not in the windows version.
 

Attachments

  • BuildFromSource.pdf
    110 KB · Views: 0
  • announcements.txt
    3.7 KB · Views: 3
  • sizes.txt
    1 KB · Views: 0
Debian testing is, at the time of this post, Trixie (version 13). This is using GCC 13. The latest version of glibc is 2.38. My experience with Bullseye (version 11) and Bookworm (version 12), is that once a version of debian moves from testing to stable, you start to get build errors building the compiler. I have created an initial version of Humax compatible glibc 2.38 with cross and fully static Humax native versions of GCC 13.

To use the native compiler you need to update path e.g.
PATH=/…/mipsel-linux-gnu/bin:"$PATH"
Where … is the full path to where you put the folder extracted from the tarball.

Attachment GCC13Sizes.txt gives download links to tarballs.

If testing changes anything, I will post again.

Edited 12 Dec 2023 : One of the links in the origional GCC13Sizes.txt did not work so the attachment has been updated!
 

Attachments

  • Native Compiler.pdf
    20 KB · Views: 2
  • Mipsel13.sh.pdf
    9.7 KB · Views: 1
  • do_gold.sh.pdf
    9.7 KB · Views: 1
  • GCC13Sizes.txt
    726 bytes · Views: 0
Last edited:
The forum that is helping me with compiler testing tells me that the 11 Dec 2023 post does not follow their idea of best practice. Hence version 2 in this post, despite identical functionality, to keep the people with more knowledge and experience than me happy.
 

Attachments

  • Prefixglibc.sh.pdf
    9.3 KB · Views: 0
  • Native13.sh.pdf
    9.3 KB · Views: 0
  • Native Compiler V2.pdf
    20 KB · Views: 0
  • Mipsel13.sh.pdf
    9.7 KB · Views: 0
  • GCC13SizesV2.txt
    826 bytes · Views: 1
  • Cross13.sh.pdf
    9.3 KB · Views: 0
The compiler testing went well, the glibc testing not so much.

Attachment GCC13SizesV3.txt contains links to the results. In addition to the previous archives, a tarball of the sources and a copy of the executables of a gdb version 14.1 build (so you don’t have to use dwarf 5 as an expleative).

Attachment GCC13Trixie.txt contains links to files to be patched over a “vanilla” 64 bit debian Trixie to get my cross compiler. File mipsel-linux-gnu(patches).* (2 compression methods) contains all the rest. Their filename is derived from the pathname e.g. usr_bin.tar.xz contains the files to patch over /usr/bin. The exception is /usr/mipsel-linux-gnu/bin which has an X64 and mips32 variant. The X86 variant is for use on Trixie, the mips32 variant is for use on the Humax or a mips32 debian (e.g. the simulator I use for testing).

The intermediate (buggy) versions will be removed, i.e. the links will no longer work, when I start to run out of cloud storage.
 

Attachments

  • GCC13SizesV3.txt
    989 bytes · Views: 1
  • GCC13Trixie.txt
    1,009 bytes · Views: 0
Back
Top