Cross-compile builds

[DaAwesomeP]

Sorry if this is a duplicate of anything that has been requested or noted before. Please close this if that is the case.

Manylinux makes compiling for many linuxes very simple. However, it does not make it super simple to make wheels available for aarch64 and other non-x86 platforms. To build for another platform currently requires either:

1. A build service that supports it
   • Not all architectures supported by all platforms (GitHub Actions doesn't yet have hosted ARM runners, probably will never have i686)
   • Travis is no longer completely free
   • Splitting between multiple build platforms is tedious
2. Your own device
   • Not everyone has a Raspberry Pi (or other) to build on
   • People submitting pull requests can't easily ensure everything still builds
   • Setting up self-hosted runner is tedious and requires maintaining your own build platform
   • If you are trying to support devices like the older Pis, building can take a very very long time

Cross-compiling is difficult, but manylinux could provide a very simple end-user solution to developers who just want to make working wheels without the hassle. If users could compile for i686, ARM, and PPC from additional x86_64 Docker images, then it would be much easier to support these platforms. And as noted before, in most cases the readily available x86_64 runners are much more powerful than the ARM runners, so they build faster. Additionally, whenever a new platform becomes popular (inevitable with ARM) it will be trivial for an end-user to begin producing wheels for it.

The only caveat is that running tests on the cross-compiled wheels would still not be possible, but this still seems like a net-positive.

[wtfsck]

i686: use the linux32 command in docker, eg. docker ... image linux32 bash ... and you should get i686 wheels.

For the other archs, you can use qemu and let it emulate the archs. It will be slower but at least you can still use GitHub Actions.

I just tested this:

      - name: docker aarch64
        run: |
          sudo apt-get update
          sudo apt-get install qemu-user-static
          uname -m
          docker run --rm -i quay.io/pypa/manylinux2014_aarch64 uname -m

The output of the last two uname -m commands were:

x86_64
aarch64

Instead of using apt-get to install qemu yourself, there seems to be an official docker action that can do it for you, see https://github.com/docker/setup-qemu-action

[DaAwesomeP]

Since opening this issue I discovered cibuildwheel and how to use QEMU with manylinux, but as you mentioned it is very very slow. I think cross-compilation would still be more ideal.

[henryiii]

There are some discussions in cibuildwheel's issues. One problem is setuptools isn't designed to do cross compilation. At some point, probably shortly after the 2.0 release of cibuildwheel, someone will likely look into some sort of feature or recipe for it - cross compiling then testing with QEMU would be fantastic. I agree, it's 6x slower or something like that to build in emulation.

[henryiii]

Splitting between multiple build platforms is tedious

Cibuildwheel's new config mode (in the latest alpha, 2.0.0a3) makes splitting between CI providers much simpler, you can put most of the config in pyproject.toml now. ;)

[messense]

I've built https://github.com/messense/manylinux-cross , it's been used in https://github.com/messense/maturin-action , although it only has very old GCC 4.8.5 and 6.5.0.

[snnn]

FYI: I wrote a document about how to use the manylinux ARM image doing cross-compiling:
https://www.onnxruntime.ai/docs/how-to/build/inferencing.html#cross-compiling-on-linux

And an Azure Devops pipeline: https://github.com/microsoft/onnxruntime/blob/rel-1.7.0/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml#L970

It has one major restriction that I must static link to GCC libraries, because I used a different version of GCC other than the default one(GCC 4.8) in manylinux2014.