Build wheels using QEMU emulation #469
Conversation
russkel
changed the title
|
Can this be run on GHA, or do they not support this? I can test it if it can. |
I really hope they support it. I have only run it locally at this point. Would be awesome to get someone else to test this as well, thanks @henryiii edit: Pretty sure there are github actions like "run-on-architecture" that use qemu so I would be surprised if it didn't work. |
|
There's an issue I need to fix not related to cibuildwheel, and I have to build NumPy because NumPy is really behind on providing all possible wheels - but it seems to work!!! See https://github.com/scikit-hep/boost-histogram/pull/474/checks?check_run_id=1495728863 (and ignore the one failing test) |
|
@joerick We should add tests in cibuildwheel for this, and is the naming, etc fine? I'm really happy to finally be able to build alternative arch wheels on GHA, especially with Travis clamping down / killing open-source support. :) |
Great to hear. I looked at the diff and seems really straightforward as intended.
I didn't hear about travis restricting open source access. but I always preferred to use GitHub actions because the infrastructure seemed better and I don't have to write as much boiler plate due to actions. I think adding tests would be wise. Not sure about naming, I was waiting for input from the maintainer, but he hasn't commented on this yet. |
https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing or https://www.jeffgeerling.com/blog/2020/travis-cis-new-pricing-plan-threw-wrench-my-open-source-works |
|
Wow! Thank you @russkel for putting this together! Apologies for not checking this out sooner - I didn't realise it was in a working state already with so little changes :) I'm looking at it again, but I'm still not sure I understand how it works! Apologies if I'm being a bit slow. Here's what I'm seeing:
Corrections/comments on the above would be super helpful. I have some ideas on interface too, and of course we'll have to add tests, but it's probably best to start with the basics! |
|
Hi @joerick , This technique was originally found on this Docker blog post: https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/ - Also, I am no expert in any of this. I use this technique to cross-compile all sorts of things because it's just so damn straightforward and always seems to work.
From wiki: binfmt_misc (Miscellaneous Binary Format) is a capability of the Linux kernel which allows arbitrary executable file formats to be recognized and passed to certain user space applications, such as emulators and virtual machines.[1] It is one of a number of binary format handlers in the kernel that are involved in preparing a user-space program to run.[2] It looks like binfmt registers some binaries with the kernel. I can't see if they are copied into the file system from the docker image. I don't believe the image continually runs. Upon restart of the machine you need to run the container again to re-register.
Looks like you're right. Should be able to substitute with https://github.com/linuxkit/linuxkit/tree/master/pkg/binfmt instead.
The kernel creates that sysfs file as a result of binfmt registration that occurs when running that image. It indicates what architectures have been registered.
They might get it for free, because all that docker does is execute things on the host kernel, right? The kernel, through registration knows what userland program to pass foreign architecture binaries to, and throws it over to qemu-static to deal with. |
I see, so perhaps it installs qemu on the host machine when it runs, and tells the kernel to run other binary formats through qemu? |
|
As for interface, my instinct was to expand the list of buildable architectures, as seen here. But that might be a little too much magic for a potentially confusing feature. So my current thinking would be add an Any thoughts @henryiii @YannickJadoul ? |
|
Yeah, I agree this is definitely very interesting, but I find it hard to wrap my head around how it works and how Not sure if I'm a huge fan of
I agree here, but if we want to take a step backwards: is this what we'd want? I had the feeling that the only way this is the case, is because we didn't know how to emulate things. Also, if I understand correctly, this is a different approach from #364, right? In #364, you're running |
|
I agree with the "use-binfmt" style, even if the name is not ideal (though it does tell the user what is happening?). Controlling the build using CIBW_BUILD and CIBW_SKIP and using these identifiers seems very natural. If there's no reason to communicate the arch's to enable, I don't think we should introduce it. As seen in my example, you'll probably still split this up since it's slow (does Travis actually have hardware for this? Didn't realize they did, but speed indicates they might), but you'd split the same way you can split now using those variables. |
The actual magic seems to be QEMU, right? So if I understand correctly, the Docker image only installs QEMU + registers the So, in order to find the best integration with cibuildwheel, it might be interesting to think about this as two sequential steps: first the installation of QEMU and registering it (which could in principle be done independently from |
|
|
Hmmm, what if (for some reason) only some architectures can be found? I think I was trying to argue that
Yeah, I didn't mean we can't provide a way of doing this, btw. Maybe we can add something that says |
|
Enjoying the discussion here. A few more thoughts-
|
OK, yes, I agree we should be careful here; I hadn't fully considered that. Maybe if we'd build this from scratch, In that case, I'd go for a flag that says "expect all architectures in That's basically @russkel's original suggestion, though, which I thought I didn't really like :-)
This should already be possible, though.
|
|
How about I wonder if this could be used the other direction, say to build x86 wheels on spare Raspberry pi's? |
It could indeed if qemu is available and binfmt is configured/registered correctly on the ARM platforms. As for the discussion on selecting builds: my thoughts was there was all this effort put into BUILD/SKIP filtering I didn't see the need to further expand that when sufficient filtering already exists. I simply added to the list of what the machine was capable building wheels for and the end user can select what they want to compile for with pre-existing functionality I didn't need to touch for the sake of the PR. |
|
I think the thing that's tripping me up with using the existing BUILD/SKIP is that they're kinda at a different level to the native machine filtering. Because really cibuildwheel's build selector logic is- ... that is, before BUILD/SKIP come into play, there is an implicit step where you can only build the wheels of the machine/OS you're on. This new emulate functionality changes the Let's say you already have a CI config like: env:
CIBW_BUILD: "cp3*"
CIBW_SKIP: "*i686"
linux_x86_64_worker:
steps:
run:
- cibuildwheel --output-dir wheelhouse .
macos_worker:
steps:
run:
- cibuildwheel --output-dir wheelhouse .Then you want to add just This is what I mean about the
No, we had a discussion a while back and I decided not to fail on this condition. Admittedly we weren't all in agreement and I was quite strongly in arguing this shouldn't be a failure though! 😛 I did have another idea, which might be a compromise - I'm quite keen on the specific emulation archs being in the hands of the user, and that we can raise an error if emulation isn't possible on the system. Previously I said we could have |
| @@ -92,8 +104,12 @@ def build(options: BuildOptions) -> None: | |||
| file=sys.stderr) | |||
| exit(2) | |||
|
|
|||
| if options.use_binfmt: | |||
| log.step('Installing binfmt...') | |||
| subprocess.check_output(['docker', 'run', '--rm', '--privileged', 'docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64']) | |||
There was a problem hiding this comment.
Judging by the about section and archival of the docker/binfmt repo, maybe this should be replaced with linuxkit/binfmt as per its suggestion?
EDIT: Whoops, just realized this was already discussed in the thread.
There was a problem hiding this comment.
Yeah absolutely. I will push up a patch.
|
Can QEMU be detected? Maybe the simplest thing would be to let users enable it using the method of their choice, then expand the set, possibly even automatically (or maybe not). |
Yeah, the way I detect the emulated architectures == detection of QEMU. Maybe that's the way to go, rather than forcing the use of linuxkit/binfmt solution. People can set up qemu and binfmt manually on their machines in a persistent manner. I was going for the all-in-one-batteries-included-just-press-go kind of solution so people could build wheels for everything easily, but maybe that's not ideal? |
|
Expanding on that now that I have a keyboard. :) I'm thinking of one of two things - the first before reading @joerick's comment above.
Or, at the very least, have a I feel like 1) is a bit more idealistic, and avoids new flags, but 2) might be favored and more explicit?
But doesn't this already cross compile 32 bit and 64 bit? You can build a 32 bit wheel on 64 bit host? Doesn't this just expand that logic to new systems? In the example, you should either expand |
|
We also should be thinking a bit about how this will expand to building macOS wheels, as we soon will need to have ways to select Universal2/AS/Intel wheels. |
I think (correct me if I'm wrong, please!) that @joerick was mostly worried about silently ignoring errors, with 1), when I suggested something similar? I.e., by explicitly requiring something to be built, you can also detect an error if it then is not available. |
This is a probably slower, but more generic way of detecting it, if Docker is available:
|
|
To keep everyone posted - I'm leaning in favour of the approach in #482 - that is, to keep the binfmt/qemu setup out of cibuildwheel, and a simple |
|
Cool. As long as the general functionality gets merged I am happy! |
From the discussion in #364 I had an attempt at getting builds working using qemu/binfmt which is pretty seamless with Docker.
Running on a Core i7 x86_64 machine: