Skip to content

Commit

Permalink
Add support for m1 wheel builds in CI.
Browse files Browse the repository at this point in the history
Since we only have one self-hosted M1 runner, and we
don't want it to become a bottleneck, we don't (yet?)
run tests on M1, just bootstrap and build wheels.

Refactors generate_github_workflows.py to break out the
reusable bootstrapping steps.

This involves creating a Helper class that knows about
the target platform and can affect generation appropriately.

Various cache keys that were previously only qualfied by os
are now qualified by platform (os+arch).

This refactoring also regularizes some of the previously
haphazard environment settings, log uploading and smoke test
running.

It also moves the CI config validation after the
bootstrapping, previously they were intermingled.

Also removes a superfluous invocation of the label check
steps inside an unrelated job which already depends on
the "proper" label check job.

[ci skip-rust]

[ci skip-build-wheels]
  • Loading branch information
benjyw committed Jun 5, 2022
1 parent a42d451 commit baa8431
Show file tree
Hide file tree
Showing 5 changed files with 499 additions and 197 deletions.
178 changes: 154 additions & 24 deletions .github/workflows/m1.yaml
Original file line number Diff line number Diff line change
@@ -1,27 +1,157 @@
# Hand-edited for now, while we experiment with self-hosted M1 runners.
#
# TODO: Integrate into workflow yaml generator.
#
# NB: The runner is an X86_64 binary, and runs under Rosetta.
# Therefore, its subprocesses default to X86_64 as well. To force subprocesses,
# such wheel building, to run as ARM64, they must be invoked via `arch -arm64e`.
# This also means that ${{runner.arch }} reports "X64" rather than "ARM64", and so
# cannot be usefully used in jobs to disambiguate cache keys.

name: M1 Wheel Build
'on':
push:
# TODO: Once we're satisfied with the job, change m1_test* to release_*, so we
# only build M1 wheels on release tags.
tags:
- m1test_*
# GENERATED, DO NOT EDIT!
# To change, edit `build-support/bin/generate_github_workflows.py` and run:
# ./pants run build-support/bin/generate_github_workflows.py


env:
ARCHFLAGS: -arch arm64e
PANTS_CONFIG_FILES: +['pants.ci.toml']
RUST_BACKTRACE: all
TMPDIR: ${{ runner.temp }}
jobs:
dummy:
build_wheels_macos_arm64:
env:
PANTS_REMOTE_CACHE_READ: 'false'
PANTS_REMOTE_CACHE_WRITE: 'false'
if: ${{ github.repository_owner == 'pantsbuild' }}
name: "Dummy (macOS ARM64)"
runs-on: [self-hosted, macOS, ARM64]
name: Bootstrap Pants and build wheels (macOS-ARM64)
needs: check_labels
runs-on:
- self-hosted
- macOS
- ARM64
steps:
- name: Echo something
run: |
arch -arm64e echo "Job was triggered on a runner that thinks it is "\
"${{ runner.os }} ${{ runner.arch }}, but can actually be $(arch -arm64e uname -p)."
- name: Check out code
uses: actions/checkout@v3
with:
fetch-depth: 10
- if: github.event_name == 'push'
name: Get commit message for branch builds
run: 'echo "COMMIT_MESSAGE<<EOF" >> $GITHUB_ENV
echo "$(git log --format=%B -n 1 HEAD)" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
'
- if: github.event_name == 'pull_request'
name: Get commit message for PR builds
run: 'echo "COMMIT_MESSAGE<<EOF" >> $GITHUB_ENV
echo "$(git log --format=%B -n 1 HEAD^2)" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
'
- name: Tell Pants to use Python ${{ matrix.python-version }}
run: 'echo "PY=python${{ matrix.python-version }}" >> $GITHUB_ENV
echo "PANTS_PYTHON_INTERPRETER_CONSTRAINTS=[''==${{ matrix.python-version
}}.*'']" >> $GITHUB_ENV
'
- name: Cache Rust toolchain
uses: actions/cache@v3
with:
key: ${{ runner.os }}-rustup-${{ hashFiles('rust-toolchain') }}-v1
path: '~/.rustup/toolchains/1.61.0-*
~/.rustup/update-hashes
~/.rustup/settings.toml
'
- name: Cache Cargo
uses: actions/cache@v3
with:
key: '${{ runner.os }}-cargo-${{ hashFiles(''rust-toolchain'') }}-${{ hashFiles(''src/rust/engine/Cargo.*'')
}}-v1
'
path: '~/.cargo/registry
~/.cargo/git
'
restore-keys: '${{ runner.os }}-cargo-${{ hashFiles(''rust-toolchain'') }}-
'
- id: get-engine-hash
name: Get native engine hash
run: 'echo "::set-output name=hash::$(./build-support/bin/rust/print_engine_hash.sh)"
'
shell: bash
- name: Cache native engine
uses: actions/cache@v3
with:
key: '${{ runner.os }}-engine-${{ steps.get-engine-hash.outputs.hash }}-v1
'
path: '.pants
src/python/pants/engine/internals/native_engine.so
src/python/pants/engine/internals/native_engine.so.metadata'
- if: github.event_name != 'pull_request'
name: Setup toolchain auth
run: 'echo TOOLCHAIN_AUTH_TOKEN="${{ secrets.TOOLCHAIN_AUTH_TOKEN }}" >> $GITHUB_ENV
'
- name: Bootstrap Pants
run: 'arch -arm64e ./pants --version
'
- name: Run smoke tests
run: 'arch -arm64e ./pants list ::
arch -arm64e ./pants roots
arch -arm64e ./pants help goals
arch -arm64e ./pants help targets
arch -arm64e ./pants help subsystems
'
- if: always()
name: Upload pants.log
uses: actions/upload-artifact@v3
with:
name: pants-log-bootstrap-macOS-ARM64
path: .pants.d/pants.log
- name: Upload native binaries
uses: actions/upload-artifact@v3
with:
name: native_binaries.${ matrix.python-version }.macOS-ARM64
path: '.pants
src/python/pants/engine/internals/native_engine.so
src/python/pants/engine/internals/native_engine.so.metadata'
- if: (github.event_name == 'push' || !contains(env.COMMIT_MESSAGE, '[ci skip-build-wheels]'))
&& (${{ github.repository_owner == 'pantsbuild' }})
name: Build wheels
run: '[[ "${GITHUB_EVENT_NAME}" == "pull_request" ]] && export MODE=debug
# TODO: Build local pex? Will require some changes to _release_helper.py to
qualify
# the .pex file name with the architecture, intead of just "darwin".
#USE_PY39=true arch -arm64e ./build-support/bin/release.sh build-local-pex
USE_PY39=true arch -arm64e ./build-support/bin/release.sh build-wheels
'
strategy:
matrix:
python-version:
- '3.9'
timeout-minutes: 60
name: Build wheels (macOS arm64)
'on':
pull_request: {}
push:
branches-ignore:
- dependabot/**
Loading

0 comments on commit baa8431

Please sign in to comment.