Skip to content

Commit d89ba5b

Browse files
oralubenpre-commit-ci[bot]XuehaiPan
authored
[Build] Migrate to scikit-build-core (#939)
* cleanup * init * build first wheel that may not work * build cython ext * fix tvm build * use sabi * update rpath to support auditwheel * pass editible build * update ci * fix warnings * do not use ccache in self host runner * test local uv cache * test pip index * update lib search to respect new lib location * fix * update ci * enable cuda by default * update src map * fix * fix * fix * Generate version with backend and git information at build time * copy tvm_cython to wheels * fix tvm lib search * fmt * remove unused * auto detect ccache * add back backend-related files * remove jit cython adaptor to simplify code * fmt * fix ci * ci fix 2 * ci fix 3 * workaround metal * ci fix 4 * fmt * fmt * Revert "ci fix 4" This reverts commit d1de829. * tmp * fix metal * trivial cleanup * add detailed build-time version for cuda * add back mlc * Restore wheel info and other trivial updates * update * fix cuda * upd * fix metal ci * test for ga build * test for nvidia/cuda * test ubuntu 20 * fix * fix * Do not use `uv build` * fix * fix * log toolchain version * merge wheel * update * debug * fix * update * skip rocm * update artifacts each * fix * fix * add mac * fix cache * fix cache * fix cache * reset and add comment * upd * fix git version * update deps * trivial update * use in-tree build dir and install to src to speedup editable build * Revert "use in-tree build dir and install to src to speedup editable build" This reverts commit 6ab87b0. * add build-dir * update docs * remove old scrips * [1/n] cleanup scripts * [Lint]: [pre-commit.ci] auto fixes [...] * fix and update * wait for tvm fix * revert some tmp fix * fix * fix * spell * doc update * test cibuildwheel * fix and test macos on ci * Update .github/workflows/dist.yml Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> * fix * test ga event * cleanup * bump tvm to support api3 * test final version * add cron * Update .github/workflows/dist.yml Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> * fix * test ccache for metal cibuildwheel * test newer macos * finish --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com>
1 parent bab57f2 commit d89ba5b

35 files changed

+603
-2261
lines changed

‎.github/workflows/cuda-ci.yml‎

Lines changed: 27 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -12,49 +12,39 @@ env:
1212
jobs:
1313
format-check:
1414
runs-on: [self-hosted, nvidia]
15-
1615
permissions:
1716
contents: write
17+
env:
18+
UV_INDEX_URL: https://mirrors.bfsu.edu.cn/pypi/web/simple
1819

1920
steps:
2021
- name: Checkout repository
2122
uses: actions/checkout@v5
2223
with:
2324
fetch-depth: 0
25+
submodules: recursive
2426

25-
- name: Set up Python
26-
uses: actions/setup-python@v6
27+
- name: Install python via uv
28+
uses: astral-sh/setup-uv@v6
2729
with:
30+
enable-cache: false
31+
cache-local-path: ${{ runner.tool_cache }}/uv
32+
activate-environment: true
2833
python-version: ${{ env.PYTHON_VERSION }}
2934

3035
- name: Ensure venv (local & persistent)
3136
run: |
32-
set -e
33-
REQS_HASH=$(sha256sum requirements-test.txt 2>/dev/null | awk '{print $1}' || echo "no_requirements")
34-
MARKER="${{ runner.tool_cache }}/.venv_marker_${{ env.PYTHON_VERSION }}_${REQS_HASH:0:8}"
35-
36-
if [[ -f "$MARKER" ]] && [[ -f "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate" ]]; then
37-
echo "venv exists and hash matches – reuse it"
38-
else
39-
echo "venv stale or missing – recreating"
40-
rm -rf "${{ runner.tool_cache }}/${{ env.VENV_DIR }}" "$MARKER"
41-
python -m venv "${{ runner.tool_cache }}/${{ env.VENV_DIR }}"
42-
# shellcheck source=/dev/null
43-
source "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate"
44-
python -m pip install --upgrade pip --no-user
45-
[[ -f requirements-test.txt ]] && \
46-
PIP_NO_BUILD_ISOLATION=1 pip install -r requirements-test.txt --no-user
47-
pip install flash_attn==2.5.8 --no-user --no-build-isolation
48-
touch "$MARKER"
49-
fi
37+
[[ -f requirements-test.txt ]] && \
38+
uv pip install -r requirements-test.txt --no-build-isolation
39+
uv pip install flash_attn==2.5.8 --no-build-isolation
5040
5141
- name: Run format check
5242
run: |
53-
source "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate"
54-
git submodule update --init --recursive
43+
set -ex
5544
mkdir -p build
5645
# run cmake to create the build directory with compile_commands.json
57-
cd build; cmake .. -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DUSE_CUDA=ON; cd ..
46+
uv pip install cmake
47+
cd build; USE_CUDA=1 cmake .. -DCMAKE_EXPORT_COMPILE_COMMANDS=ON; cd ..
5848
if ! output=$(./format.sh 2>&1); then
5949
echo "------------------------------------"
6050
echo "message:"
@@ -70,56 +60,41 @@ jobs:
7060
needs: format-check
7161
permissions:
7262
contents: read
63+
env:
64+
UV_INDEX_URL: https://mirrors.bfsu.edu.cn/pypi/web/simple
7365
steps:
7466
- name: Checkout repository
7567
uses: actions/checkout@v5
7668
with:
7769
fetch-depth: 0
70+
submodules: recursive
7871
repository: ${{ github.event.pull_request.head.repo.full_name }}
7972
ref: ${{ github.event.pull_request.head.ref }}
8073

81-
- name: Set up Python
82-
uses: actions/setup-python@v6
74+
- name: Install python via uv
75+
uses: astral-sh/setup-uv@v6
8376
with:
77+
enable-cache: false
78+
cache-local-path: ${{ runner.tool_cache }}/uv
79+
activate-environment: true
8480
python-version: ${{ env.PYTHON_VERSION }}
8581

86-
- name: Ensure venv (local & persistent)
82+
- name: Setup venv
8783
run: |
88-
set -e
89-
REQS_HASH=$(cat requirements-test.txt 2>/dev/null || true)
90-
MARKER="${{ runner.tool_cache }}/.venv_marker_${{ env.PYTHON_VERSION }}_${REQS_HASH:0:8}"
91-
92-
if [[ -f "$MARKER" ]] && [[ -f "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate" ]]; then
93-
echo "venv exists and hash matches – reuse it"
94-
else
95-
echo "venv stale or missing – recreating"
96-
rm -rf "${{ runner.tool_cache }}/${{ env.VENV_DIR }}" "$MARKER"
97-
python -m venv "${{ runner.tool_cache }}/${{ env.VENV_DIR }}"
98-
source "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate"
99-
python -m pip install --upgrade pip --no-user
100-
[[ -f requirements-test.txt ]] && \
101-
PIP_NO_BUILD_ISOLATION=1 pip install -r requirements-test.txt --no-user
102-
# flash attention usually requires no isolation build
103-
pip install flash_attn==2.5.8 --no-user --no-build-isolation
104-
pip install . --no-user
105-
touch "$MARKER"
106-
fi
84+
[[ -f requirements-test.txt ]] && \
85+
uv pip install -r requirements-test.txt --no-build-isolation
86+
uv pip install flash_attn==2.5.8 --no-build-isolation
10787
10888
- name: Install project (wheel form)
10989
run: |
110-
source "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate"
111-
pip install . --no-user -v
90+
uv pip install .
11291
11392
- name: Run examples
11493
run: |
115-
source "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate"
11694
cd examples
117-
unset PYTHONPATH
11895
python -m pytest -n 4 **/test*.py -v -r fE --durations=0 --cache-clear
11996
12097
- name: Run tests
12198
run: |
122-
source "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate"
12399
cd testing/python
124-
unset PYTHONPATH
125100
python -m pytest -n 4 -v -r fE --durations=0 --cache-clear --timeout=3600

‎.github/workflows/dist.yml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
name: Dist
2+
on:
3+
schedule:
4+
# gemini said this is 6:00 china time
5+
- cron: '0 22 * * *'
6+
release:
7+
types: [ published ]
8+
9+
env:
10+
PYTHON_VERSION: '3.12'
11+
12+
concurrency:
13+
group: ${{ github.workflow }}-${{ github.ref }}
14+
cancel-in-progress: true
15+
16+
jobs:
17+
build-wheels:
18+
strategy:
19+
matrix:
20+
os: [ubuntu-22.04, ubuntu-22.04-arm, macos-16]
21+
include:
22+
- os: ubuntu-22.04
23+
cuda_version: "12.1"
24+
- os: ubuntu-22.04-arm
25+
cuda_version: "12.8"
26+
fail-fast: true
27+
runs-on: ${{ matrix.os }}
28+
env:
29+
CUDA_VERSION: ${{ matrix.cuda_version }}
30+
NO_VERSION_LABEL: ${{ github.event_name != 'release' }}
31+
32+
steps:
33+
- name: Checkout repository
34+
uses: actions/checkout@v4
35+
with:
36+
fetch-depth: 1
37+
submodules: recursive
38+
39+
- name: ccache
40+
uses: hendrikmuhs/ccache-action@v1.2
41+
if: startsWith(matrix.os, 'macos')
42+
with:
43+
create-symlink: true
44+
key: ${{ github.job }}-${{ matrix.os }}
45+
46+
- name: Build wheels
47+
uses: pypa/cibuildwheel@v3.2
48+
with:
49+
output-dir: wheelhouse
50+
config-file: "{package}/pyproject.toml"
51+
52+
# just for now to list all files
53+
- name: List wheels
54+
id: ls-whl
55+
run: echo "whl_name=$(ls wheelhouse | head -n1)" >> $GITHUB_OUTPUT
56+
57+
- uses: actions/upload-artifact@v4
58+
with:
59+
name: ${{ steps.ls-whl.outputs.whl_name }}.zip
60+
path: wheelhouse/${{ steps.ls-whl.outputs.whl_name }}
61+
compression-level: 0

‎.github/workflows/metal-ci.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,12 +81,12 @@ jobs:
8181
python-version: ${{ env.PYTHON_VERSION }}
8282

8383
- name: Ensure venv (local & persistent)
84-
run: uv pip install -r requirements-test.txt -r requirements-build.txt
84+
run: uv pip install -r requirements-test.txt
8585

8686
- name: Build wheel
8787
run: |
8888
source .venv/bin/activate
89-
uv pip install -v --no-build-isolation .
89+
uv pip install -v .
9090
9191
- name: Run metal test
9292
run: |

‎.github/workflows/rocm-ci.yml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,9 +51,9 @@ jobs:
5151
- name: Run format check
5252
run: |
5353
source "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate"
54-
git submodule update --init --recursive
54+
git submodule update --init --recursive --checkout
5555
mkdir -p build
56-
cd build; cmake .. -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DUSE_ROCM=ON; cd ..
56+
cd build; USE_ROCM=1 cmake .. -DCMAKE_EXPORT_COMPILE_COMMANDS=ON; cd ..
5757
if ! output=$(./format.sh 2>&1); then
5858
echo "------------------------------------"
5959
echo "message:"
@@ -73,7 +73,7 @@ jobs:
7373
- name: Checkout repository
7474
uses: actions/checkout@v5
7575
with:
76-
fetch-depth: 0
76+
fetch-depth: 1
7777
repository: ${{ github.event.pull_request.head.repo.full_name }}
7878
ref: ${{ github.event.pull_request.head.ref }}
7979

@@ -111,6 +111,7 @@ jobs:
111111
run: |
112112
echo "Installing project (wheel form)"
113113
source "${{ runner.tool_cache }}/${{ env.VENV_DIR }}/bin/activate"
114+
git submodule update --init --recursive --checkout --recommend-shallow
114115
USE_ROCM=True pip install . --no-user
115116
116117
- name: Run tests

‎.gitignore‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
*.slo
33
*.lo
44
*.o
5+
*.so
56
*.obj
67
*.pyc
78

@@ -18,7 +19,7 @@
1819

1920
debug/
2021
build/
21-
dist/
22+
*dist/
2223
wheelhouse/
2324
__pycache__
2425
nnfusion.tar.gz
@@ -82,18 +83,12 @@ models/frozenmodels/
8283
# .ruff_cache
8384
.ruff_cache
8485

85-
# build sdist
86-
build_sdist/
87-
8886
# exclude debug testing folder
8987
!testing/python/debug
9088

9189
# ignore lib with develop mode
9290
tilelang/lib
9391

94-
# tox
95-
.tox/
96-
9792
# cython
9893
tilelang/jit/adapter/cython/.cycache
9994

‎3rdparty/tvm‎

Submodule tvm updated from 883e96b to 5bf17a3

0 commit comments

Comments
 (0)