cross compiling with sysroot from device

Question

I mounted target device's (Beaglebone Black, ARM arch) root with sshfs, to a folder on host, and trying to cross compile using --sysroot option of GCC.

But there's a problem - some include header files are located on target device not just at /usr/include folder, but in its subfolder arm-linux-gnueabihf. The same there's /usr/lib/arm-linux-gnueabihf folder too.

"Hello world" sample fails to compile

[gmixaz:/work] $ $CC --sysroot=/work/sysroot h.c
In file included from /work/sysroot/usr/include/stdio.h:27:0,
                 from h.c:1:
/work/sysroot/usr/include/features.h:364:25: fatal error: sys/cdefs.h: No such file or directory
 #  include <sys/cdefs.h>

because sys/cdefs.h is located in /usr/include/arm-linux-gnueabihf, while compiler expects it in /usr/include

My question is why do I have that subfolder arm-linux-gnueabihf in /usr/include? What was the rationale to put part of include files (and .so files in /usr/lib/arm-linux-gnueabihf) to that subfolder?

And how shall I address this issue when cross compiling with --sysroot option - do I need to specify the subfolder with -I and -L compiler options, or there's some better solution? I though only --sysroot should be enough but it isn't. Is that 'by design'?

After adding the folders to command line I got another problem:

[gmixaz:/work] 1 $ $CC --sysroot /work/sysroot3 -I /work/sysroot3/usr/include/arm-linux-gnueabihf -L /work/sysroot3/usr/lib/arm-linux-gnueabihf h.c
/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/lib/gcc/arm-cortexa9_neon-linux-gnueabihf/6.3.1/../../../../arm-cortexa9_neon-linux-gnueabihf/bin/ld.bfd: cannot find crt1.o: No such file or directory
/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/lib/gcc/arm-cortexa9_neon-linux-gnueabihf/6.3.1/../../../../arm-cortexa9_neon-linux-gnueabihf/bin/ld.bfd: cannot find crti.o: No such file or directory
collect2: error: ld returned 1 exit status

What's going on? When building crosstool-NG I tried to match target arch info. Cross-compiling GCC on host:

[gmixaz:/work] 1 $ $CC -v
Using built-in specs.
COLLECT_GCC=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/bin/arm-cortexa9_neon-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/libexec/gcc/arm-cortexa9_neon-linux-gnueabihf/6.3.1/lto-wrapper
Target: arm-cortexa9_neon-linux-gnueabihf
Configured with: /dockcross/crosstool/toolchain/.build/src/gcc-linaro-6.3-2017.02/configure --build=x86_64-build_pc-linux-gnu --host=x86_64-build_pc-linux-gnu --target=arm-cortexa9_neon-linux-gnueabihf --prefix=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf --with-sysroot=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/arm-cortexa9_neon-linux-gnueabihf/sysroot --enable-languages=c,c++ --with-cpu=cortex-a8 --with-fpu=neon --with-float=hard --with-pkgversion='crosstool-NG ' --enable-__cxa_atexit --disable-libmudflap --disable-libgomp --disable-libssp --disable-libquadmath --disable-libquadmath-support --disable-libsanitizer --disable-libmpx --with-gmp=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --with-mpfr=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --with-mpc=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --with-isl=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --enable-lto --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --enable-threads=posix --enable-plugin --enable-gold --with-libintl-prefix=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --disable-multilib --with-local-prefix=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/arm-cortexa9_neon-linux-gnueabihf/sysroot --enable-long-long
Thread model: posix
gcc version 6.3.1 20170109 (crosstool-NG ) 

GCC on target:

debian@beaglebone:~$ cc -v
Using built-in specs.
COLLECT_GCC=cc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/6/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Debian 6.3.0-18+deb9u1' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) 

What I'm missing in the toolchain configuration?

ADDED: I have found that following file is responsible for libraries being placed to that subfolder on target:

debian@beaglebone:~$ cat /etc/ld.so.conf.d/arm-linux-gnueabihf.conf
# Multiarch support
/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf

Now I'm trying to understand what is correct way to attach that folders to cross compiler toolchain.

Also, I guess that something similar is set for include header files, just interesting where's that setting for GCC on target?

ADDED: Similar issue discussed for RPi cross compilation: https://github.com/raspberrypi/tools/issues/42

Answer 1

I'm posting results of my investigation as an answer, but will accept if it solves the problem - I haven't got a solution yet )

It appears that the target's GCC was configured with --enable-multiarch, and it adds arm-linux-gnueabihf subfolder for that target name.

Good reading about the subject is https://wiki.debian.org/Multiarch/LibraryPathOverview

So probably my problem is that target name in the cross compiling GCC differs from that on the target's GCC (arm-cortexa9_neon-linux-gnueabihf vs arm-linux-gnueabihf). I'll check if I can change it in crosstool-NG options, and see if it fixes the issue.

ADDED: Unfortunately I was not able to configure crosstool-ng toolchain to use target sysroot. Similar issue was reported here: http://answers.opencv.org/question/180037/cmake-cross-compiler-problem-with-pkg_check_modules-for-some-packages/

I have finally decided to use dockcross/linux-armv6 image which installs linaro toolchain customized for RPi, at least it compiles "hello world" from box. Though the toolchain seems to be derived also from crosstool-ng, it is downloaded as binary from RPi tools repo, and its configuration significantly differs from one for dockcross/linux-armv7.

Now I'm trying to cross compile OpenCV per this guide: http://courses.engr.uky.edu/ideawiki/doku.php?id=resources:sop:cross_compiling_opencv_for_raspberry_pi and after specifying paths for pkg-config, OpenCV cmake script started to see installed libraries in target's sysroot.

Answer 2

Actually, I don't want to dig into your way because I think it is not a common way to cross compiling. The way you are cross-compiling is not suggested. There are two common ways to cross-compile an app.

First, creating a virtual environment just like your embedded board with some virtualization apps like Qemu. You can find a very good tutorial here. Then you can bring your app there and compile it natively. A drawback for this solution is you are creating a whole filesystem and you should bring all of your app dependencies (which may become confusing) into your main filesystem.

Second, you can use Linaro toolchains to cross-compile your app. Here is another tutorial and I think you can find more about it.

In my opinion, I prefer using Qemu when I need to build the app natively which have a lot of dependencies. On the other hand, cross-compiling apps that don't natively support it can become very confusing.