Add support for m1 wheel builds in CI. #15738
Conversation
benjyw
force-pushed
the
m1_ci
branch
16 times, most recently
from
8d200c2
to
c4c4881
Compare
|
Reviewers: I recommend reviewing the generate_github_workflows.py changes first, to get a sense, then carefully examine the generated file diffs to verify that the impacts are correct on the existing workflows, and then review the new m1 workflow. |
| path: .pants.d/pants.log | ||
| - name: Upload native binaries | ||
| uses: actions/upload-artifact@v3 | ||
| with: | ||
| name: native_binaries.${{ matrix.python-version }}.${{ runner.os }} | ||
| name: native_binaries.${ matrix.python-version }.Linux-x86_64 |
There was a problem hiding this comment.
We can't use "${{ runner.os }}-${{ runner.arch }}" because the M1 runner is an X86 binary running under Rosetta, and it sees its arch as x86_64... But since we know the platform we're targeting, we can just inject that into the string directly.
| @@ -100,10 +102,6 @@ jobs: | |||
| - name: Bootstrap Pants | |||
| run: './pants --version | |||
|
|
|||
| ' | |||
| - name: Validate CI config | |||
There was a problem hiding this comment.
Moved below, after the bootstrapping is complete.
| steps: | ||
| - env: |
There was a problem hiding this comment.
This was here by mistake. The dependency on check_labels has already run this check, we just neglected to remove these steps from this job when we broke that out into its own job.
| @@ -253,18 +246,34 @@ jobs: | |||
| run: './pants --version | |||
|
|
|||
| ' | |||
| - name: Run smoke tests | |||
There was a problem hiding this comment.
No harm in running smoke tests as part of bootstrapping, as they should be really fast, and it saved me having to make this conditional. But we can do so if it turns out to be a problem.
|
|
||
| ' | ||
| - if: always() | ||
| name: Upload pants.log |
There was a problem hiding this comment.
We weren't previously uploading the pants.log in all cases, now we do.
| @@ -251,32 +268,17 @@ def bootstrap_caches() -> Sequence[Step]: | |||
| ] | |||
|
|
|||
|
|
|||
| def native_binaries_upload() -> Step: | |||
There was a problem hiding this comment.
These were just moved into Helper
| "needs": "check_labels", | ||
| "strategy": {"matrix": {"python-version": python_versions}}, | ||
| "env": DISABLE_REMOTE_CACHE_ENV, | ||
| "timeout-minutes": 40, | ||
| "if": IS_PANTS_OWNER, | ||
| "steps": [ | ||
| *checkout(), |
There was a problem hiding this comment.
All moved into Helper.bootstrap_pants()
| @@ -376,21 +475,7 @@ def test_python_linux(shard: str) -> dict[str, Any]: | |||
| ), | |||
| }, | |||
| { | |||
| "name": "Run smoke tests", | |||
There was a problem hiding this comment.
Moved into Helper.bootstrap_pants()
| @@ -527,15 +605,6 @@ def build_steps(*, is_macos: bool) -> list[Step]: | |||
| "env": DISABLE_REMOTE_CACHE_ENV, | |||
| "if": IS_PANTS_OWNER, | |||
| } | |||
| deploy_to_s3_step = { | |||
There was a problem hiding this comment.
Moved to a global helper function above
benjyw
force-pushed
the
m1_ci
branch
3 times, most recently
from
7b7f8aa
to
d00f199
Compare
There was a problem hiding this comment.
Yay!!! You also need to update here:
pants/build-support/bin/_release_helper.py
Lines 1240 to 1249 in dac89c4
Since we only have one self-hosted M1 runner, and we don't want it to become a bottleneck, we don't (yet?) run tests on M1, just bootstrap and build wheels. Refactors generate_github_workflows.py to break out the reusable bootstrapping steps. This involves creating a Helper class that knows about the target platform and can affect generation appropriately. Various cache keys that were previously only qualfied by os are now qualified by platform (os+arch). This refactoring also regularizes some of the previously haphazard environment settings, log uploading and smoke test running. It also moves the CI config validation after the bootstrapping, previously they were intermingled. Also removes a superfluous invocation of the label check steps inside an unrelated job which already depends on the "proper" label check job. [ci skip-rust] [ci skip-build-wheels]
|
Yeah, I'll change that in a follow up, when I know I have real wheels in S3 to test on. |
Since we only have one self-hosted M1 runner, and we don't want it to become a bottleneck,
we don't (yet?) run tests on M1, just bootstrap and build wheels.
Refactors generate_github_workflows.py to break out the reusable bootstrapping steps.
This involves creating a Helper class that knows about the target platform and can
affect generation appropriately.
Various cache keys that were previously only qualfied by os are now qualified
by platform (os+arch).
This refactoring also regularizes some of the previously haphazard environment settings,
log uploading and smoke test running.
It also moves the CI config validation after the bootstrapping, previously they
were intermingled.
Also removes a superfluous invocation of the label check steps inside an unrelated job
which already depends on the "proper" label check job.