Extend support for more architectures on GitLab

[wbarnha]

Description

I really like using cibuildwheel on GitHub for building wheels for various architectures, but unfortunately I need to also support projects on GitLab that require wheels for aarch64 and x86-64 architectures on Linux. From what I understand, cibuildwheel can only build wheels for x86-64 architectures on GitLab. I have personally had success building wheels for multiple architectures without cibuildwheel, but I'd like to move away from my current methods. Taken from https://github.com/Mizux/python-native/blob/main/ci/README.md for example, I would run this to create wheels for a Python project to support aarch64:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx build -f path/to/Dockerfile --platform linux/arm64 .

I originally used https://github.com/dockcross/dockcross to cross-compile my C/C++ bindings for various architectures, however auditwheel indicated they were incompatible similar to what was noted in dockcross/dockcross#475. Unfortunately it seems that running docker buildx is the the most foolproof way as of this post to support GitLab builds. I'd like to get away from docker buildx since it's quite slow.

As noted in #951, it seems GitLab supports macOS runners so functionality could be added to build macOS wheels using that runner, so it could be worth exploring that category at some point.

TL;DR: I really want to build aarch64 wheels on GitLab for Linux and macOS. Any plans to support that?

[Czaki]

TL;DR: I really want to build aarch64 wheels on GitLab for Linux and macOS. Any plans to support that?

The core thing with lack of official support for other runners than Linux is lack of availability of such runner when we setup it.

@joerick it looks like an advanced open source support program for OSS is available https://about.gitlab.com/solutions/open-source. Did cibuildwheel is already a member?

But in general, the list of supported platforms and systems is the list where we run tests. There are no limitations in code that prevent running cibuildwheel on an unsupported platform.

Did you try to run it before opening the issue (on macOS runner)?

[wbarnha]

Did you try to run it before opening the issue (on macOS runner)?

Not yet, I'm primarily focused on getting Linux wheels built for aarch64, macOS is a secondary objective for now. Will follow up when I have the opportunity.

[Czaki]

In general, if you have a method to build wheels with cross-compilation that passes auditwheel please share (we are open for contributions). It will significantly improve the life of many people.

[joerick]

You might have already tried this, but there might be a way to emulate. The way that our example configs do emulation of aarch64 on Github Actions is using the github action docker/setup-qemu-action, which internally just invokes https://hub.docker.com/r/tonistiigi/binfmt . That sets up hooks in the OS (something called binfmt_misc) that enable the emulation. You could try invoking tonistiigi/binfmt on Gitlab and then running a cibuildwheel --archs aarch64.

[wbarnha]

I have been able to emulate on GitLab using multiarch/qemu-user-static. An example intended for GitLab + Linux aarch64 can be found in https://github.com/wbarnha/pybind_cmake_example_crossbuild.

[joerick]

Nice! I'd accept a PR with an example config for this.