Skip to content

Commit 8ce2778

Browse files
XuehaiPanLeiWang1999Alex4210987
authored
[CI][Refactor] Merge test CI workflow files into one (#973)
* refactor: merge test CI workflow files into one * chore: set `UV_INDEX_STRATEGY=unsafe-best-match` * feat: add AST test with Python 3.8 * feat: implement manual caching mechanism for self-hosted runners * refactor: simplify cache logic for self-hosted runners * chore: clear uv cache on failure * chore: print format.sh output to logs * chore: improve uv caching * chore: disable parallel test * chore: use `PYTHONDEVMODE=1` in CI * feat: enable coredump generation * fix: fix perfbench condition * Revert "feat: enable coredump generation" This reverts commit c52da65. * chore: move example CI down * Revert "chore: move example CI down" This reverts commit 9d8e650. * chore: skip example `test_example_mha_sink_bwd_bhsd` * chore: skip example `test_example_gqa_sink_bwd_bhsd` * fix: fix example argument passing * fix: loosen test criteria * chore: rename `CMAKE_CONFIGURE_OPTIONS` -> `CLANG_TIDY_CMAKE_OPTIONS` for clarity * feat: enable parallel testings * chore: update pytest options * remove skipped test as now been resolved * chore: empty commit to re-trigger ci * test for n 1 * chore: remove ` --numprocesses=1` option in example * chore: disable failfast * chore: update cibw selection * fix: fix git submodule clone * chore: update cibw commands * fix: fix yapf multiprocessing * chore: setup ccache for CIBW on macOS only * chore: update comments * chore: update artifact listing * fix: do not fail if not found nvcc in PATH * fix: fix flash-attn installation * chore: update dist workflow trigger * chore: remove outdated comments * chore(workflows/dist): simplify build matrix strategy * fix: fix CUDA path finding * fix: fix CUDA path finding * chore: imcrease CI timeout * ci: disable failfast * fix: hide path prefix * chore: more verbose * chore: disable PR trigger for dist workflow * fix: seed for tests * fix: use nightly torch for ROCm tests * chore: enable PR trigger for dist workflow * chore: stop uploading debug wheels as artifacts in PR * chore: do not run workflows in forks * chore: housekeep requirements * chore: use Nightly-ROCm-6.3 for CI * chore: use Nightly-ROCm-6.4 for CI * Update ROCm toolkit version to 7.0 * chore: restore previous rocm-ci.yml for test * fix: cleanup PYTHONPATH * chore: remove previous rocm-ci.yml * ci fix * chore: remove previous rocm-ci.yml * chore: enable parallel example run --------- Co-authored-by: LeiWang1999 <leiwang1999@outlook.com> Co-authored-by: alex_xiao <xinyuxiao2024@gmail.com>
1 parent 80665cd commit 8ce2778

24 files changed

+655
-476
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 388 additions & 0 deletions
Large diffs are not rendered by default.

‎.github/workflows/cuda-ci.yml‎

Lines changed: 0 additions & 96 deletions
This file was deleted.

‎.github/workflows/dist.yml‎

Lines changed: 109 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -2,60 +2,125 @@ name: Dist
22
on:
33
schedule:
44
# gemini said this is 6:00 china time
5-
- cron: '0 22 * * *'
5+
- cron: "0 22 * * *"
6+
pull_request:
7+
types:
8+
- opened
9+
- synchronize
10+
- reopened
11+
- ready_for_review
12+
paths:
13+
- setup.py
14+
- setup.cfg
15+
- pyproject.toml
16+
- MANIFEST.in
17+
- CMakeLists.txt
18+
- version_provider.py
19+
- .github/workflows/dist.yml
620
release:
7-
types: [ published ]
21+
types:
22+
- published
823

9-
env:
10-
PYTHON_VERSION: '3.12'
24+
permissions:
25+
contents: read
1126

1227
concurrency:
13-
group: ${{ github.workflow }}-${{ github.ref }}
28+
group: "${{ github.workflow }}-${{ github.ref }}"
1429
cancel-in-progress: true
1530

1631
jobs:
1732
build-wheels:
33+
name: Build wheels for Python ${{ matrix.python-version }} on ${{ matrix.target.runner }} with ${{ matrix.target.toolkit }}
34+
if: |
35+
github.repository_owner == 'tile-ai' &&
36+
(github.event_name != 'pull_request' || !github.event.pull_request.draft)
1837
strategy:
1938
matrix:
20-
os: [ubuntu-22.04, ubuntu-22.04-arm, macos-16]
21-
include:
22-
- os: ubuntu-22.04
23-
cuda_version: "12.1"
24-
- os: ubuntu-22.04-arm
25-
cuda_version: "12.8"
26-
fail-fast: true
27-
runs-on: ${{ matrix.os }}
39+
target:
40+
- { runner: ubuntu-latest, toolkit: "CUDA-12.1" }
41+
- { runner: ubuntu-24.04-arm, toolkit: "CUDA-12.8" }
42+
- { runner: macos-latest, toolkit: "Metal" }
43+
python-version:
44+
- "3.8"
45+
# TVM is built with Python 3.8 Limited API, it should work with all Python >= 3.8.
46+
# - "3.9"
47+
# - "3.10"
48+
# - "3.11"
49+
# - "3.12"
50+
# - "3.13"
51+
# - "3.14"
52+
fail-fast: false
53+
timeout-minutes: 120
54+
runs-on: ${{ matrix.target.runner }}
2855
env:
29-
CUDA_VERSION: ${{ matrix.cuda_version }}
30-
NO_VERSION_LABEL: ${{ github.event_name != 'release' }}
56+
NO_VERSION_LABEL: ${{ github.event_name == 'release' && 'OFF' || 'ON' }}
3157

3258
steps:
33-
- name: Checkout repository
34-
uses: actions/checkout@v4
35-
with:
36-
fetch-depth: 1
37-
submodules: recursive
38-
39-
- name: ccache
40-
uses: hendrikmuhs/ccache-action@v1.2
41-
if: startsWith(matrix.os, 'macos')
42-
with:
43-
create-symlink: true
44-
key: ${{ github.job }}-${{ matrix.os }}
45-
46-
- name: Build wheels
47-
uses: pypa/cibuildwheel@v3.2
48-
with:
49-
output-dir: wheelhouse
50-
config-file: "{package}/pyproject.toml"
51-
52-
# just for now to list all files
53-
- name: List wheels
54-
id: ls-whl
55-
run: echo "whl_name=$(ls wheelhouse | head -n1)" >> $GITHUB_OUTPUT
56-
57-
- uses: actions/upload-artifact@v4
58-
with:
59-
name: ${{ steps.ls-whl.outputs.whl_name }}.zip
60-
path: wheelhouse/${{ steps.ls-whl.outputs.whl_name }}
61-
compression-level: 0
59+
- name: Checkout repository
60+
uses: actions/checkout@v4
61+
with:
62+
fetch-depth: 1
63+
submodules: recursive
64+
65+
# NB: CIBW builds wheels in containers on Linux
66+
- name: Setup ccache (macOS only)
67+
if: runner.os == 'macOS'
68+
uses: hendrikmuhs/ccache-action@v1
69+
with:
70+
create-symlink: true
71+
key: ccache-${{ runner.os }}-${{ runner.arch }}-${{ matrix.python-version }}-${{ matrix.target.toolkit }}
72+
evict-old-files: "7d"
73+
74+
- name: Set CIBW_BUILD
75+
run: |
76+
PYTHON_VERSION="${{ matrix.python-version }}"
77+
PYTHON_VERSION_MAJMIN="$(echo "${PYTHON_VERSION}" | cut -d '.' -f-2)"
78+
PYTHON_VERSION_MAJMIN_NODOT="${PYTHON_VERSION_MAJMIN//./}"
79+
echo "CIBW_BUILD=cp${PYTHON_VERSION_MAJMIN_NODOT}-*" | tee -a "${GITHUB_ENV}"
80+
81+
if [[ "${{ matrix.target.toolkit }}" == *"CUDA"* ]]; then
82+
CUDA_VERSION="${{ matrix.target.toolkit }}"
83+
CUDA_VERSION="${CUDA_VERSION#CUDA-}"
84+
echo "CUDA_VERSION=${CUDA_VERSION}" | tee -a "${GITHUB_ENV}"
85+
fi
86+
87+
- name: Build wheels
88+
uses: pypa/cibuildwheel@v3.2
89+
with:
90+
package-dir: .
91+
output-dir: wheelhouse
92+
config-file: "{package}/pyproject.toml"
93+
94+
- name: Upload wheels
95+
# Not PR to save artifact storage, as wheels are only needed for releases.
96+
if: github.event_name != 'pull_request'
97+
uses: actions/upload-artifact@v4
98+
with:
99+
name: wheels-${{ matrix.python-version }}-${{ runner.os }}-${{ runner.arch }}-${{ matrix.target.toolkit }}
100+
path: wheelhouse/*.whl
101+
if-no-files-found: error
102+
103+
list-artifacts:
104+
name: List artifacts
105+
# Not PR to save artifact storage, as wheels are only needed for releases.
106+
if: github.event_name != 'pull_request'
107+
runs-on: ubuntu-latest
108+
needs: [build-wheels]
109+
timeout-minutes: 15
110+
steps:
111+
- name: Download built wheels
112+
uses: actions/download-artifact@v5
113+
with:
114+
pattern: wheels-*
115+
path: dist
116+
merge-multiple: true
117+
118+
- name: List distributions
119+
run: ls -lh dist/*
120+
121+
- name: Upload artifacts
122+
uses: actions/upload-artifact@v4
123+
with:
124+
name: artifacts
125+
path: dist/*
126+
if-no-files-found: error

‎.github/workflows/metal-ci.yml‎

Lines changed: 0 additions & 95 deletions
This file was deleted.

‎.github/workflows/pr-perfbench-bot.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ jobs:
1616
perfbench:
1717
name: Benchmark between PR and main
1818
if: |
19-
github.event_name == 'pull_request' &&
19+
github.repository_owner == 'tile-ai' &&
20+
github.event.issue.pull_request &&
2021
(contains(github.event.comment.body, '/performance-report') || contains(github.event.comment.body, '/perf'))
2122
runs-on: [self-hosted, nvidia]
2223
steps:
@@ -27,7 +28,7 @@ jobs:
2728
fetch-depth: 0
2829
submodules: recursive
2930

30-
- name: Set up Python
31+
- name: Setup Python
3132
uses: actions/setup-python@v6
3233
with:
3334
python-version: "3.9"

‎.github/workflows/pr-reminder-bot.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ on:
88
jobs:
99
remind:
1010
runs-on: ubuntu-latest
11+
if: github.repository_owner == 'tile-ai'
1112
steps:
1213
- name: Remind
1314
uses: actions/github-script@v8

0 commit comments

Comments
 (0)