-
Notifications
You must be signed in to change notification settings - Fork 377
FAQ
Documentation for answers to common questions and examples of how to extend cross.
Table of Contents
- Container Engines
- Container Engine Issues
- External Dependencies
- CI Workflows
- Missing Intrinsics
- Other
You can place a Cross.toml file in the root of your Cargo project or use a CROSS_CONFIG environment variable to tweak cross's behavior.
cross provides default Docker images for the targets listed below. However, it
can't cover every single use case out there. For other targets, or when the
default image is not enough, you can use the target.{{TARGET}}.image field in
Cross.toml to use custom Docker image for a specific target:
[target.aarch64-unknown-linux-gnu]
image = "my/image:tag"In the example above, cross will use a image named my/image:tag instead of
the default one. Normal Docker behavior applies, so:
-
Docker will first look for a local image named
my/image:tag -
If it doesn't find a local image, then it will look in Docker Hub.
-
If only
image:tagis specified, then Docker won't look in Docker Hub. -
If only
tagis omitted, then Docker will use thelatesttag.
It's recommended to base your custom image on the default Docker image that
cross uses: ghcr.io/cross-rs/{{TARGET}}:{{VERSION}} (where {{VERSION}} is cross's version).
This way you won't have to figure out how to install a cross C toolchain in your
custom image. Example below:
Dockerfile
FROM ghcr.io/cross-rs/aarch64-unknown-linux-gnu:latest
RUN dpkg --add-architecture arm64 && \
apt-get update && \
apt-get install --assume-yes libfoo:arm64Building
$ docker build -t my/image:tag path/to/where/the/Dockerfile/resides
Cross.toml
[target.aarch64-unknown-linux-gnu]
image = "my/image:tag"When running cross from inside a docker container, cross needs access to
the hosts docker daemon itself. This is normally achieved by mounting the
docker daemons socket /var/run/docker.sock. For example:
$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v .:/project \
-w /project my/development-image:tag cross build --target mips64-unknown-linux-gnuabi64
The image running cross requires the rust development tools to be installed.
With this setup cross must find and mount the correct host paths into the
container used for cross compilation. This includes the original project directory as
well as the root path of the parent container to give access to the rust build
tools.
To inform cross that it is running inside a container set CROSS_DOCKER_IN_DOCKER=true.
A development or CI container can be created like this:
FROM rust:1
# set CROSS_DOCKER_IN_DOCKER to inform `cross` that it is executed from within a container
ENV CROSS_DOCKER_IN_DOCKER=true
# install `cross`
RUN cargo install cross
...
Limitations: Finding the mount point for the containers root directory is
currently only available for the overlayfs or overlayfs2 storage driver. In order to access
the parent containers rust setup, the child container mounts the parents
overlayfs. The parent must not be stopped before the child container, as the
overlayfs can not be unmounted correctly by Docker if the child container still
accesses it. cross currently cannot find the mount point if it is using
a container engine running on top of the Windows Subsystem for Linux (WSL2),
since WSL2 uses atypical bind-mount paths for the overlayfs2 driver (see #728).
By default, cross tries to use Docker or Podman, in that order.
If you want to choose a container engine explicitly, you can set the
binary name (or path) using the CROSS_CONTAINER_ENGINE
environment variable.
For example in case you want use Podman, you can set CROSS_CONTAINER_ENGINE=podman.
Whenever debugging any container engine issues, please ensure you are using a recent version of docker or podman. Using an out-of-date version, particularly if the container engine is provided by a package manager, is a likely source of the issue.
There are a few known issues running cross on macOS. The simplest solution is to use Docker itself, which supports all desired features and has no known, macOS-specific issues. If you wish to use another container engine, such as Podman or Lima, the following issues are present:
- Lima and Podman both cannot mount directories in
/tmpby default. - Podman cannot use bind mounts, and must use remote cross since data volumes work.
- Podman and Lima do not support SELinux labels (the former errors, the latter warns), which we use with bind mounts.
By default Podman only makes the $HOME directory available to the virtual machine when running on macOS or Windows. You must initialize your virtual machine with additional mount points for every directory you may need in your build.
# Add any other mount points required
$ podman machine init -v /tmp:/tmp -v /private:/privateLima has a few specific issues due to a lack of compatibility with Docker's CLI interface. First, Lima by default only makes the /tmp/lima directory writable, meaning you must modify its configurations prior to running cross to ensure it works. Due to the beta nature of Lima and the risk of deleting your files, only make the specific directories you require for your project writable. To edit the Lima configurations, run:
$ limactl editInside, ensure the following values are set:
mounts:
# this should be the directory provided by `CARGO_HOME`, or `~/.cargo`
- location: "~/.cargo"
writable: true
# this should be the directory provided by `XARGO_HOME`, or `~/.xargo`
- location: "~/.xargo"
writable: true
# Also, the project directory (the workspace root), and additional
# mounted volumes that need write access, and the target directory
# must all be writable. If not, `cross` will fail.Next, Lima does not support user namespaces, and therefore CROSS_CONTAINER_USER_NAMESPACE=none must be set to disable user namespace remapping. Using Lima (as of nerdctl 0.21) is non-trivial and therefore not recommended.
Currently, when using docker-in-docker, we can only support the overlay, overlay2, or fuse-overlayfs storage driver, so we can find our mount points.
Docker
Docker uses overlay2 by default and can be configured by stopping the daemon, changing the overlay type, and restarting it. For example, on a system using systemd, first stop the daemon:
$ sudo systemctl stop dockerThen edit the storage driver config file (/etc/docker/daemon.json):
{
"storage-driver": "overlay2"
}Valid values for cross are overlay, overlay2, or fuse-overlayfs. Finally, restart the Docker daemon:
$ sudo systemctl start dockerTo check the changes have been registered, run:
$ docker info | grep 'Storage Driver'
Storage Driver: overlay2Podman
By default, podman uses the overlay driver and can be configured as described here. This might require changing the root directory data via the --root flag, or manually deleting existing storage drivers.
For example, first create a custom directory for our fuse-overlayfs driver:
$ mkdir -p $HOME/.local/share/containers/fuse-overlayfs-storage/Next, export our storage driver type via environment variables or use CROSS_CONTAINER_OPTS environment variable to ensure we use the correct storage driver. For example:
# Using environment variables
$ export STORAGE_DRIVER=fuse-overlayfs
$ CROSS_CONTAINER_OPTS="--root=\"$HOME/.local/share/containers/fuse-overlayfs-storage/\"" \
cross build ...
# Using `CROSS_CONTAINER_OPTS`
$ CROSS_CONTAINER_OPTS="--storage-driver=fuse-overlayfs --root=\"$HOME/.local/share/containers/fuse-overlayfs-storage/\"" \
cross build ...You can check podman is using the correct storage driver via:
$ podman --storage-driver=fuse-overlayfs --root $HOME/.local/share/containers/fuse-overlayfs-storage/ info | grep graphDriverName
graphDriverName: fuse-overlayfsSome rust projects depend on external libraries, which are not provided in the pre-built containers. For information on how to create, build, and use custom images, see Custom Images. For example, a Dockerfile containing ALSA Development files for an armv7-unknown-linux-gnueabihf target would be as follows:
# base pre-built cross image
FROM ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf:latest
# add our foreign architecture and install our dependencies
RUN apt-get update && apt-get install -y --no-install-recommends apt-utils
RUN dpkg --add-architecture armhf
RUN apt-get update && apt-get -y install libasound2-dev:armhf
# add our linker search paths and link arguments
ENV CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_RUSTFLAGS="-L /usr/lib/arm-linux-gnueabihf -C link-args=-Wl,-rpath-link,/usr/lib/arm-linux-gnueabihf $CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_RUSTFLAGS"A complete project linking to libdbus can be found here.
More packages for other architectures can be found in Debian repositories rather than Ubuntu. To install packages from Debian repositories (buster in the example below), you can extend the pre-built Docker images as follows. First, save this file to install_deb.sh, and make sure it's executable (chmod +x install_deb.sh).
Next, extend our Dockerfile with the custom logic (this uses aarch64-unknown-linux-gnu, but most architectures work):
FROM ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main
COPY install_deb.sh /
# Change the packages to your dependencies.
RUN /install_deb.sh arm64 libgstreamer1.0-dev \
libgstreamer-plugins-base1.0-dev \
libssl-dev
# Update any environment variables required with `ENV`.
# ENV MYVAR=MYVALUEWe can then build our image and use it for our target as shown in Custom Images. For certain architectures, you may need to use other Debian repositories or more complex logic. See linux-image.sh for code to handle more complex cases.
In order to use bindgen with cross, you must extend the Dockerfile to install clang-3.9 and libclang-3.9-dev (or a later version of clang). An example Dockerfile is:
FROM ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf:main
RUN apt-get update && apt-get install --assume-yes --no-install-recommends libclang-3.9-dev clang-3.9When installing clang for a custom image, you must install it for the host and not for the target architecture. Doing so will uninstall the GCC cross-compiler, and therefore prevent cross from compiling for that architecture. For example, the following code is wrong:
FROM ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf:main
RUN dpkg --add-architecture armhf
RUN apt-get update && apt-get install --assume-yes --no-install-recommends libclang-3.9-dev:armhf clang-3.9:armhfBy default, newer images will error if you try to install packages conflicting with our toolchains, saying there is no valid installation candidate. If you are using bindgen, see the Using Bindgen section.
Why isn't OpenSSL installed? Maintaining images with OpenSSL proved to be a source of numerous bugs (see #229 and #332), and only some images provide OpenSSL to begin with. Since the openssl crate provides a vendored copy, there's good ways of installed an OpenSSL dependency for rust packages, and we provide a recipe for doing so. If OpenSSL is needed as a dependency for other C/C++ libraries, we document how to install and link to external libraries.
For some targets, certain system packages can conflict with packages required to use the toolchain. For example, installing libclang may be required to use bindgen. However, libclang should be installed for the image architecture and not the target architecture, and many users assume clang is required as well. If users attempt to install clang for the target architecture (say, clang:armel for target arm-unknown-linux-gnueabi), it will uninstall the cross-compiler, rendering the Docker image useless.
To avoid users from installing packages that will negatively interfere with the build system, we use package pinning to prevent apt from finding valid installation candidates for these packages. Each pin will be found in /etc/apt/preferences.d/ with the pin under the package name. In order to override our package pins, you can either delete our custom pin or provide your own pin with a higher priority. Package pinning is superior to package holding, since it works for packages that are not yet installed.
We currently use package pinning to prevent the removal of binutils for any image that uses a cross-compiler installed via apt. In addition, we block the use of any package from armhf foreign architecture for the arm-unknown-linux-gnueabihf target, since the target is an ARMv6 hard-float target, but armhf packages are built for ARMv7-A. This means that any installed armhf system packages will successfully link to the Rust package, however, the generated code will be unable to run on an ARMv6 CPU.
actions-rs/cargo provides built-in support for using cross with the use-cross key. For example, to test your crate on aarch64-unknown-linux-gnu and arm-unknown-linux-gnueabi when pushing new commits, you can use the following sample workflow:
on: [push]
name: Cross CI
jobs:
cross:
name: Rust ${{matrix.target}}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
target:
- aarch64-unknown-linux-gnu
- arm-unknown-linux-gnueabi
steps:
- uses: actions/checkout@v2
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
target: ${{matrix.target}}
override: true
- uses: actions-rs/cargo@v1
with:
use-cross: true
command: test
args: --target ${{matrix.target}}Gitlab CI uses a remote docker build, which requires the use of cross remote. A sample .gitlab-ci.yml file as as follows:
variables:
# the host where the docker instance is running
DOCKER_HOST: tcp://docker:2375/
# use for much faster builds
DOCKER_DRIVER: overlay2
# ensure cross knows it's running remotely
CROSS_REMOTE: 1
services:
- docker:18.09-dind
armv6:
script:
- cross test --target arm-unknown-linux-gnueabihfWhen using cross's build-std configuration or -Z build-std, cross can fail with numerous error messages such as:
undefined reference to `__addtf3'
undefined reference to `__netf2'
undefined reference to `__subtf3'
undefined reference to `__addtf3'
undefined reference to `__fixtfsi'
undefined reference to `__floatsitf'
Symbols starting with __ are reserved for compiler vendors, and suggests a missing symbol in a compiler intrinsic. For some missing intrinsics, this can be fixed by linking to libgcc to your build command (only for *-linux-gnu and *-linux-musl targets):
# fails
$ cross +nightly build --target aarch64-unknown-linux-musl -Z build-std
# now this succeeds
$ export RUSTFLAGS="-C link-arg=-lgcc -Clink-arg=-static-libgcc"
$ cross +nightly build --target aarch64-unknown-linux-musl -Z build-stdThis is because certain missing intrinsics are provided by libgcc, but are not present without using the c feature of compiler-builtins. Not every target will have every intrinsic provided by libgcc, however:
$ cross +nightly build --target aarch64-unknown-linux-gnu -Z build-std
= note: libstd-3d482a16e52503f2.rlib(std-3d482a16e52503f2.std.18753d0b-cgu.3.rcgu.o): In function `core::sync::atomic::atomic_add::h038154eda15b0d27':
atomic.rs:3036: undefined reference to `__aarch64_ldadd8_relax'
atomic.rs:3038: undefined reference to `__aarch64_ldadd8_rel'
atomic.rs:3037: undefined reference to `__aarch64_ldadd8_acq'
atomic.rs:3039: undefined reference to `__aarch64_ldadd8_acq_rel'
atomic.rs:3040: undefined reference to `__aarch64_ldadd8_acq_rel'To fix this, you must provide the C sources from the LLVM tree. First, add the following to your Cargo.toml:
[dependencies.compiler_builtins]
git = "https://github.com/rust-lang/compiler-builtins"
features = ["c"]Next, clone to LLVM project and add the compiler-rt sources to your shared volumes and/or project:
$ git clone https://github.com/llvm/llvm-project --branch llvmorg-14.0.6 --depth 1Next, add the compiler-rt subdirectory to your project or mount the volume, and add the RUST_COMPILER_RT_ROOT environment variable so compiler-builtins knows where to find these sources.
For example, using a mounted volume, in Cross.toml
[build.env]
volumes = ["RUST_COMPILER_RT_ROOT=/path/to/compiler-rt"]Or, adding the sources to your project subdirectory, in Cross.toml
[build.env]
passthrough = ["RUST_COMPILER_RT_ROOT=/path/to/project/compiler-rt"]This will allow the project to build the all compiler-builtins intrinsics from source.
If you get an error saying missing symbols because libgcc was not provided (the error message below), you've likely found a bug in cross or compiler-builtins. Please file an issue and if applicable, we can patch this upstream.
= note: ld: libc.a(strtod.lo): undefined reference to symbol '__trunctfsf2@@GCC_3.0'
libgcc_s.so.1: error adding symbols: DSO missing from command line
Due to the large storage requirements of cross images, we provide utilities to list and remove images associated with cross. Images can be listed and removed with cross-util:
# list all images created by cross
$ cross-util images list
# a dry-run of removing all images created by cross
$ cross-util images remove
# remove all images created by cross
$ cross-util images remove --execute
# remove all images for the given target created by cross
$ cross-util images remove arm-unknown-linux-gnueabihf --executeBy default, cross runs native binaries without emulation, and non-native binaries using Qemu, WINE, or some other runner. However, it may not be desirable to run the binary natively in all cases: for example, code that runs on x86_64 for an i586 may not be valid on a real CPU. Therefore, we support 2 different ways of customizing the runners:
Cross Target Runner
See configuration for more details. A custom runner for cross can be provided via target.(...).runner, and can be qemu-system, qemu-user, or native. This can also be provided via the CROSS_TARGET_${TARGET}_RUNNER environment variable.
[target.aarch64-unknown-linux-gnu]
runner = "qemu-user"Cargo Target Runner
However, this may not be enough flexibility, and you may wish to provide your own wrapper, or test running on a specific CPU. In this case, you can provide CARGO_TARGET_${TARGET}_RUNNER (which will allow any command, or sequence of arguments, and override CROSS_TARGET_${TARGET}_RUNNER).
# run binaries on the cortex-a72 when targeting `aarch64-unknown-linux-gnu`.
$ export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUNNER="qemu-aarch64 -cpu cortex-a72"
$ cross run --target aarch64-unknown-linux-gnuRunning FreeBSD tests cannot be done on Linux without full system emulation via Qemu, or using FreeBSD cloud services (such as CI images or virtual private servers). The best solution is using Cirrus CI, as is done by Rust in testing libc. You can also use recommended VPSs by the FreeBSD project. Any VPS hosting provider supporting custom images will work for other BSD distros.
cross does not ship with a full Android emulator, and therefore some tests on Android can fail. The best solution is to install Anbox on a Linux system (possibly a cross Docker image using custom images), and following all the post-install instructions. This has not been tested to work in a container, and may only work on the host machine.
If using a container image with a different architecture than the host container engine (such as using a linux/amd64 image on a linux/arm64 host), the container engine may not have proper Qemu emulators to run the image with the foreign architecture.
To do so, install the binfmt support for the host kernel, do:
docker run --privileged --rm tonistiigi/binfmt --install allThis requires --privileged because it modifies the host kernel to register the executable file formats with the correct Qemu emulator.