This is Gentoo's testing wiki. It is a non-operational environment and its textual content is outdated.

Please visit our production wiki at https://wiki.gentoo.org

Cross build environment

From Gentoo Wiki (test)
Jump to:navigation Jump to:search

This article provides instructions on creating a cross build environment using crossdev.

Cross build environments are needed for different situations:

  • To cross build software for slow target hosts on a fast build host.
  • To build software with a different toolchain (i.e. different libc versions).
  • When a specialized system environment is needed:
    • i.e. a separate multilib system for binaries with abnormal dependencies which you don't want to mix with your main system (like the steam platform).
    • i.e. a base image for docker containers.

Create the cross toolchain

  • Install crossdev:
root #emerge --ask crossdev
Note
It is recommended to create an overlay for crossdev, otherwise it may use any existing overlay to save generated ebuilds. More info: Custom repository#Crossdev.
  • Create the toolchain. For the raspberry-pi it would look like
root #crossdev --stable -t armv6j-hardfloat-linux-gnueabi
  • For a amd64-multilib environment it would look like:
root #crossdev --stable -t x86_64-multilib-linux-gnu
  • Target (-t) takes a tuple ARCH-VENDOR-OS-LIBC; see crossdev -t help
Note
If you experience a problem creating the gcc stage 2 (for gcc v5), see Known bugs and limitations below (you may need to add --genv 'EXTRA_ECONF="--disable-libstdc++-v3"' to the crossdev command line).

Update the target build configuration

  • The target make.conf should be changed according to the installation handbook. For the base system at least these options should be checked, the rest can be configured later:
FILE /usr/<TARGET>/etc/portage/make.conf
ARCH=arm #check your target architecture
FEATURES=buildpkg #remove buildpkg if you don't want all binary packages in $ROOT/packages
USE="${ARCH} -pam -acl" # disable acl to build the base system (essential packages will fail cross compilation otherwise),enable it later if it's needed
MAKEOPTS="-j5" # set -j1 for debugging failed compilation if necessary, otherwise set the build jobs appropriate to the number of cpu cores
CFLAGS="..." # you can't use -march=native here if the target has a different CPU. See the following subsections for useful adaptions.
  • Set the appropriate make profile. See below for target architecture specific examples.
  • If you build on amd64 see the lib64-bug at #Known_bugs_and_limitations

Raspberry Pi specific

FILE /usr/<TARGET>/etc/portage/make.conf
...
CFLAGS="-Ofast -mfpu=vfp -mfloat-abi=hard -march=armv6zk -mtune=arm1176jzf-s -fomit-frame-pointer -pipe -fno-stack-protector -U_FORTIFY_SOURCE"
...
  • Set the appropriate make profile. For the raspberry-pi it would be:
pi #ARCH=arm PORTAGE_CONFIGROOT=/usr/armv6j-hardfloat-linux-gnueabi eselect profile set default/linux/arm/13.0/armv6j

Allwinner A20 specific

In addition to the auto-generated file content, the following modifications are necessary for successful cross-compilation:

FILE /usr/armv7a-hardfloat-linux-gnueabi/etc/portage/make.conf
...
# motivation for "-native-extensions" https://bugs.gentoo.org/show_bug.cgi?id=628440
USE="${ARCH} -pam -acl -ncurses -xattr -vtv -native-extensions" # these use flags are required for successful cross-compilation (Aug. 2017)
# CFLAGS="-Ofast -fomit-frame-pointer -pipe -fno-stack-protector -U_FORTIFY_SOURCE -march=armv7ve -mtune=cortex-a7 -mfloat-abi=hard -mfpu=neon-vfpv4 -funsafe-math-optimizations"
CFLAGS="-O2 -pipe -fomit-frame-pointer" # above CFLAGS were not helpful (Aug. 2017)
...
  • Set the appropriate make profile. For Allwinner A20 based boards it would be:
sunxi #cd /usr/armv7a-hardfloat-linux-gnueabi/etc/portage && rm make.profile && ln -s /usr/portage/profiles/default/linux/arm/13.0/armv7a make.profile

Build the base system

It can be built from scratch or a stage3 tarball can be unpacked into /usr/<TARGET>. To build it from scratch:

  • Build the system packages
    root #<TARGET>-emerge -uva --keep-going @system
    For the raspberry-pi it would be:
    pi #armv6j-hardfloat-linux-gnueabi-emerge -uva --keep-going @system
    For the Allwinner A20 it would be:
    sunxi #armv7a-hardfloat-linux-gnueabi-emerge -uva --keep-going @system
    sunxi #armv7a-hardfloat-linux-gnueabi-emerge -uva --keep-going dev-util/systemtap # in case the previous command fails due to missing sys/sdt.h

(Do not care about failed packages, this will be fixed later)

  • Build other essential packages:
root #<TARGET>-emerge -uva1 --keep-going $(egrep '^[a-z]+' /usr/portage/profiles/default/linux/packages.build)
  • To build the failed packages it is needed to compile them "natively", which means in this case, that the packages need to be compiled in the target chroot environment. If the target host has a different architecture a qemu-chroot is needed. For targets that the build host cpu can handle directly the following steps can be skipped and you can chroot directly into the target environment.
    • install qemu on the host
      root #emerge --ask app-emulation/qemu
    • prepare qemu for the target (in this case for the arm architecture)
      root #QEMU_USER_TARGETS="arm" QEMU_SOFTMMU_TARGETS="arm" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu
    • install qemu to the build environment
      root #cd /usr/<TARGET> && ROOT=$PWD/ emerge --ask --usepkgonly --oneshot --nodeps qemu
    • A first test:
      root #/etc/init.d/qemu-binfmt start && cd /usr/<TARGET> && chroot . /bin/bash --login
      If it works, leave the chroot and go on with the next steps.
    • Optional: This step is not necessary in most cases. To make the target environment emulation more complete, a wrapper can be used, that passes the correct cpu option to qemu. The following would be an example for the raspberry-pi cpu option for qemu (-cpu arm1176). Please check if the command at the end (qemu-arm) is present on your build host.
      CODE 
      #include <string.h>
#include <unistd.h>
 
int main(int argc, char **argv, char **envp) {
    char *newargv[argc + 3];
 
    newargv[0] = argv[0];
    newargv[1] = "-cpu";
    newargv[2] = "arm1176";
 
    memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc - 1));
    newargv[argc + 2] = NULL;
    return execve("/usr/bin/qemu-arm", newargv, envp);
}
      • Build it with
        pi #gcc -static qemu-wrapper.c -Ofast -s -o /usr/armv6j-hardfloat-linux-gnueabi/usr/local/bin/qemu-wrapper

Chroot into the target environment

  • Create a chroot script
FILE /usr/local/bin/chroot-armv6jExample for raspberry-pi
/etc/init.d/qemu-binfmt start

#Next two lines are optional. Activate if the qemu-wrapper is used. Check that the wrapper location corresponds with the call at the end of line two
#echo '-1' > /proc/sys/fs/binfmt_misc/arm #deregister wrong arm
#echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\x00\xff\xfe\xff\xff\xff:/usr/local/bin/qemu-wrapper:' > /proc/sys/fs/binfmt_misc/register

cd /usr/armv6j-hardfloat-linux-gnueabi
mount -t proc none proc
mount -o bind /dev dev
mount -o bind /usr/portage usr/portage
mount -o bind /usr/local/portage usr/local/portage
mount -o bind /usr/src/raspberrypi-sources usr/src/linux
mount -o bind /lib/modules lib/modules
mount -o bind /sys sys
cp /etc/resolv.conf etc/resolv.conf
#mount -o bind /tmp tmp
#mount -o bind /dev/pts dev/pts #only for X
 
chroot . /bin/bash --login
 
#umount dev/pts
#umount tmp
umount sys
umount lib/modules
umount usr/src/linux
umount usr/local/portage
umount usr/portage
umount dev
umount proc
  • To chroot into the new environment run the script
    pi #chroot-armv6j
    and complete the set up of your build environment
    • Create the portage temp directory:
    root #ln -s /tmp /usr/armv6j-hardfloat-linux-gnueabi/tmp
    • Update /etc/locale.gen and /etc/env.d/02locale and run:
    root #locale-gen
    • Check/Reload config:
    root #gcc-config -l;ldconfig -v;ROOT=/ env-update; source /etc/profile
    • To run emerge inside the chroot it is needed to pass other config variables to portage, it can be done with an alias:
    root #echo 'alias emerge-chroot="ROOT=/ CBUILD=$(portageq envvar CHOST) HOSTCC=$CBUILD emerge"' > /etc/bash/bashrc.d/emerge-chroot && source /etc/profile
    • Packages, that refused the cross compilation can now be built with:
    root #emerge-chroot --ask --update -v --keep-going @system
  • After installation of the base system the target environment can be finished according to the installation handbook

Known bugs and limitations

  • On amd64 build hosts, some cross compiled packages end up in the target environment in /usr/lib64 even if it is not a 64bit target, so set a symlink
    root #cd /usr/<TARGET>/usr && ln -s lib lib64
    .
  • Some packages, that create new system users fail to create them in the target environment and create them in build host instead (i.e. openssh). You need to create the user manually or emerge the package in the chroot again.
  • If the build host is no-multilib and target environment is multilib: Emerge of sys-apps/sandbox fails because of missing 32bit support of the cross compiler. To solve this remove temporarily the dependency to sys-apps/sandbox in sys-apps/portage. Emerge sys-devel/gcc after that in the chroot environment. Now it is possible to emerge sys-apps/sandbox in the chroot.

Cross building static binaries for closed systems

Static binaries are not needed often, but there are some occasions where they are useful.

  • When creating (Docker) container images. According to the container philosophy it is recommended to run only one process per container. It is also recommended to put as little as possible into a container. In this case one statically linked binary is desirable.
  • When a program will run on a closed system like an ARM device with Android. In the case of Android it is possible to either use the Android's NDK and/or its clib implementation "bionic" or build a statically linked binary, that depends on no system libs and can run standalone.

Cross build toolchain for static binaries

It is pretty much the same as above beside that it is not needed to emerge a full @system. The build essentials are enough.

  • If the target is an Android device the architechture ist probably armv7a or arvm8a, so the tuple ARCH-VENDOR-OS-LIBC could be armv7a-android_hardfloat-linux-gnueabi
  • Compiling the build essentials for an exemplary android toolchain could look like
    android #armv7a-android_hardfloat-linux-gnueabi-emerge -uva1 --keep-going $(egrep '^[a-z]+' /usr/portage/profiles/default/linux/packages.build) portage
  • USE=static-libs should be switched on after installation of the base system (e.g. if the target program needs openssl)

customized glibc

There are some issues with glibc. Probably this does not affect other libc's like µclibc or musl. If you use glibc you should pay attention to the following.

  • Even when a program was built with -static, the resulting binaries aren't necessary really static. Because of design decisions of the glibc, at least the /lib/libnss_*.so are looked up dynamically. To force nss linked statically the flag --enable-static-nss can be used for compiling glibc.
  • When a program is linked statically and makes use of glibc's NSS features like getpwnam() the lookup of user names fails when nsswitch.conf is set to "compat". Set it to files in this case:
    FILE nsswitch.confnsswitch.conf for the resulting binaries
    #passwd:      compat
#shadow:      compat
#group:       compat

passwd:      files
shadow:      files
group:       files
  • The glibc has hardcoded absolute pathes for some configuration files like /etc/resolv.conf. On an closed system (like Android) these files doesn't necessarily exist and without them DNS lookups will fail. Normally the files can't be written without root priviledges. If becoming root is not an option, glibc must be customized to look at a different location for these files. Keep in mind, that his is only necessary if the program makes use of glibc functions that need these files. But virtually every program, that connects to the internet uses sorts of gethostbyname and therefor needs a resolv.conf.

(optionally) create an customized glibc ebuild

This step is only necessary, if the glibc config files don't reside in /etc and the target program makes use of glibc's lookup functions (probably when the program does dns or user name lookups)

  1. Copy /usr/portage/sys-libs/glibc to your local ebuild repository and create a custom glibc ebuild
  2. Make the following changes
    1. Remove the KEYWORDS line (to prevent a accidential use in other environments)
    2. If you want to change the path to the config files add to the eblit-src_prepare-post section
      CODE adding to the glibc ebuild
      #change the location for config files from /etc to /sdcard/etc (fits to an android environment, because it is writable by users)
sed -i -e 's:/etc/:/sdcard/etc/:' \
    resolv/{netdb.h,resolv.h,res_hconf.c} \
    nss/{nss_files/files-init.c,db-Makefile,files-netgrp.c,files-alias.c} \
    nis/nss_compat/{compat-pwd.c,compat-spwd.c,compat-grp.c,compat-initgroups.c} \
    sysdeps/{generic/paths.h,unix/sysv/linux/paths.h} || die

Rebuild a customized glibc

Rebuild glibc with static options. When the default path was changed in the previous step remember to change --sysconfdir here appropriate.

root #EXTRA_ECONF='--enable-static --enable-static-nss --sysconfdir=/etc' <TARGET>-emerge -va1 --keep-going sys-libs/glibc

Build the desired Software

  • Chroot into the target environment (e.g. chroot-armv7a-android)
  • If the default path for glibc config files was changed symlink it mv -i /sdcard/etc/* /etc && rmdir /sdcard/etc && ln -s /etc /sdcard/etc
  • Build a statically linked package
    • an example for android is privoxy
      android #CFLAGS="$(portageq envvar CFLAGS) -static" CXXFLAGS=$CFLAGS LDFLAGS="$(portageq envvar LDFLAGS) -static" PKGDIR=/tmp/ emerge-chroot privoxy --buildpkgonly -va1
    • an example for a statically linked nginx is
      container #NGINX_MODULES_HTTP="gzip" CFLAGS="$(portageq envvar CFLAGS) -static" CXXFLAGS=$CFLAGS LDFLAGS="$(portageq envvar LDFLAGS) -static" PKGDIR=/tmp/ emerge-chroot -va1 --buildpkgonly nginx:mainline

Example: Use a statically linked privoxy on Android

  • unzip the binary tarball from above to /sdcard/ on the android device (e.g. trough the ssh server "sshelper" via Google Play or "ftpserver" via F-Droid)
    • change in the privoxy config /sdcard/etc/privoxy/config at least all entries with /etc/ and /var/ in /sdcard/etc and /sdcard/var/
  • put a resolv.conf in /sdcard/etc/ (e.g. with nameservers from www.opennicproject.org or the ad blocking nameservers fom www.alternate-dns.com nameserver 8.8.8.8)
  • put a privoxy startscript in /sdcard/
    FILE /sdcard/privoxy.start
    #sdcard is mounted non-executable, so copy privoxy to the homedir of the used app                                                          
#Homedir of connectbot (change if a different app is used)                                                                                 
CB_HOME=/data/data/org.connectbot                                                                                                          
#copy if nonexistent or newer                                                                                                              
[ /sdcard/usr/sbin/privoxy -nt $CB_HOME/privoxy ]&& cat /sdcard/usr/sbin/privoxy > $CB_HOME/privoxy && chmod 755 $CB_HOME/privoxy          
#run privoxy in foreground                                                                                                                 
$CB_HOME/privoxy --no-daemon /sdcard/etc/privoxy/config
  • install the android app "connectbot" (available via F-Droid)
    • open a local connection named "privoxy" and close it again
    • hold long the "privoxy" connection to open the context menu and choose "edit host"
    • insert as "automation"-task: sh /sdcard/privoxy.start (a newline is needed at the end)
    • to start privoxy open the connection in connectbot
  • To browse trough privoxy, add as proxy localhost/8118 to the mobile data APN's (in the android control panel at "mobile networks" → "APNs") and WiFi's
Retrieved from "https://wikitest.gentoo.org/index.php?title=Cross_build_environment&oldid=727328"