Build aarch64 wheels #639
Build aarch64 wheels #639
Conversation
|
Love that feature! @janaknat , I see some duplicated code from |
|
@misl6 Done. Moved the code to |
|
@misl6 Ping. |
.github/workflows/create.yml
Outdated
| - name: Set up QEMU for aarch64 wheels | ||
| if: runner.os == 'Linux' | ||
| uses: docker/setup-qemu-action@v1 | ||
| with: | ||
| platforms: arm64 |
There was a problem hiding this comment.
QEMU only needs to be set up for aarch64 builds, not also x86_64 ones.
We're doing something similar here: https://github.com/kivy/kivy/blob/master/.github/workflows/manylinux_wheels.yml
(Same applies for the Build and test aarch64 wheels) step.
|
@misl6 Added checks. |
.github/workflows/create.yml
Outdated
| CIBW_TEST_REQUIRES: 'pytest' | ||
| CIBW_BEFORE_TEST: 'yum install ant -y && cd {project} && ant all' | ||
| CIBW_TEST_COMMAND: 'cd {project}/tests/ && CLASSPATH=../build/test-classes:../build/classes python -m pytest -v' |
There was a problem hiding this comment.
Optimization: we can expand the tests also to the x86_64 version, so there's no need to keep a separate step for aarch64 wheels.
|
pp39-manylinux_aarch64 passes. Required the extras [dev,ci] to be installed for testing. |
|
@misl6 Both x86 and aarch64 Linux wheels are built and tested in a single step. |
|
@misl6 Ping! |
1 similar comment
|
@misl6 Ping! |
.github/workflows/create.yml
Outdated
| cibw_skip: '*-manylinux_i686' | ||
| cibw_archs: 'x86_64' | ||
| - os: ubuntu-latest | ||
| architecture: 'x86' | ||
| cibw_archs: 'i686' |
There was a problem hiding this comment.
We want to skip the build for *-manylinux_i686 wheels, as we removed support in a previous version.
Can you please change the config to only build x86_64 and aarch64 wheels?
There was a problem hiding this comment.
Sure. Will remove i686 (x86).
.github/workflows/create.yml
Outdated
| - name: Build and test Linux wheels | ||
| if: ${{ matrix.os == 'ubuntu-latest' }} | ||
| env: | ||
| CIBW_SKIP: '*musllinux* ${{ matrix.cibw_skip }}' | ||
| CIBW_ARCHS: '${{ matrix.cibw_archs }}' | ||
| CIBW_TEST_REQUIRES: 'pytest' | ||
| CIBW_TEST_EXTRAS: 'dev,ci' | ||
| CIBW_BEFORE_TEST: 'yum install ant -y && cd {project} && ant all' | ||
| CIBW_TEST_COMMAND: 'cd {project}/tests/ && CLASSPATH=../build/test-classes:../build/classes python -m pytest -v' | ||
| run: python -m cibuildwheel --output-dir dist |
There was a problem hiding this comment.
Can we keep the tests separate from the build phase?
There was a problem hiding this comment.
Previously, it was built in this step and tested later on in a separate step. We want to use that approach? This current approach is to build the wheels and if successful, the CIBW_TEST_COMMAND is run to test the wheels.
There was a problem hiding this comment.
Yep, but unfortunately (as shown here: https://github.com/pypa/cibuildwheel/blob/main/README.md#how-it-works), tests are run into the same container / runner session.
Keeping it separate allows us to be sure that the build scripts/installations are not influencing the tests.
(A new and clean virtualenv is created and used for tests, but it's not enough sometime)
There was a problem hiding this comment.
We can test the wheels in a separate step for x86_64. For aarch64, that won't be possible because github natively does not support arm64 architecture.
There was a problem hiding this comment.
For aarch64 tests, qemu will help 😀
When setting up the test for aarch64, we can add:
- name: Set up QEMU for aarch64 tests
if: ${{ matrix.test_arch == 'aarch64' }} # here we need to add `test_arch` in the matrix.
uses: docker/setup-qemu-action@v1
with:
platforms: arm64
There was a problem hiding this comment.
@misl6 True, but that only sets up the QEMU for aarch64 binary. Any commands run further will be on the host architecture. To run the tests, we will have to use a docker image (manylinux2014_aarch64), setup python, pytest, ant. Then install the generated wheels, run 'ant all' and then the unit tests. cibuildwheels sets all of that up for us when we specify what we need using CIBW_TEST_REQUIRES. Unfortunately, cibuildwheel does not have a 'skip build and test only' option. We'll probably end up re-implementing the testing logic of cibuildwheel ourselves, unless there is another option.
There was a problem hiding this comment.
@misl6 Any suggestions?
There was a problem hiding this comment.
Hi @janaknat !
We're doing something similar here (that is actually used to build dependencies, as we can cache them, but the same approach could be used to process tests): https://github.com/kivy/kivy/blob/a94505ef854acae0046ab8d822a4c96c011bd269/.github/workflows/manylinux_wheels.yml#L73-L77
Also, build and test x86 wheels in one step, similar to aarch64.
|
Hi @janaknat ! This PR has been superseded by #653 (introduces a self-hosted runner for Everything started on top of your PR 😀. I squashed the two commits by not removing you as a co-author. (Hope is appreciated) Thank you! |
|
@misl6 Thank you for helping with this! |
Successful builds: https://github.com/janaknat/pyjnius/actions/runs/3305376283