Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: more modern nox #1823

Merged
merged 4 commits into from
May 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,4 @@ build:
- asdf global uv latest
- uv venv
- uv pip install -e.[docs]
- . .venv/bin/activate && NO_COLOR=1 mkdocs build --strict --site-dir $READTHEDOCS_OUTPUT/html
- NO_COLOR=1 .venv/bin/mkdocs build --strict --site-dir $READTHEDOCS_OUTPUT/html
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,19 +165,19 @@ Here are some repos that use cibuildwheel.
| [NCNN][] | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | ncnn is a high-performance neural network inference framework optimized for the mobile platform |
| [Prophet][] | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth. |
| [MyPy][] | ![github icon][] | ![apple icon][] ![linux icon][] ![windows icon][] | The compiled version of MyPy using MyPyC. |
| [Kivy][] | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Open source UI framework written in Python, running on Windows, Linux, macOS, Android and iOS |
| [duckdb][] | ![github icon][] | ![apple icon][] ![linux icon][] ![windows icon][] | DuckDB is an in-process SQL OLAP Database Management System |
| [Kivy][] | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Open source UI framework written in Python, running on Windows, Linux, macOS, Android and iOS |

[scikit-learn]: https://github.com/scikit-learn/scikit-learn
[pytorch-fairseq]: https://github.com/pytorch/fairseq
[pytorch-fairseq]: https://github.com/facebookresearch/fairseq
[NumPy]: https://github.com/numpy/numpy
[Tornado]: https://github.com/tornadoweb/tornado
[Matplotlib]: https://github.com/matplotlib/matplotlib
[NCNN]: https://github.com/Tencent/ncnn
[Prophet]: https://github.com/facebook/prophet
[MyPy]: https://github.com/mypyc/mypy_mypyc-wheels
[Kivy]: https://github.com/kivy/kivy
[duckdb]: https://github.com/duckdb/duckdb
[Kivy]: https://github.com/kivy/kivy

[appveyor icon]: docs/data/readme_icons/appveyor.svg
[github icon]: docs/data/readme_icons/github.svg
Expand Down Expand Up @@ -214,11 +214,11 @@ Changelog

_12 May 2024_

- ✨ Adds CPython 3.13 support, under the prerelease flag [CIBW_PRERELEASE_PYTHONS](https://cibuildwheel.pypa.io/en/stable/options/#prerelease-pythons). This version of cibuildwheel uses 3.13.0b1. Free-threading mode is not available yet, waiting on official binaries (planned for beta 2) and pip support.
- ✨ Adds CPython 3.13 support, under the prerelease flag [CIBW_PRERELEASE_PYTHONS](https://cibuildwheel.pypa.io/en/stable/options/#prerelease-pythons). This version of cibuildwheel uses 3.13.0b1. Free-threading mode is not available yet (#1657), waiting on official binaries (planned for beta 2) and pip support.

_While CPython is in beta, the ABI can change, so your wheels might not be compatible with the final release. For this reason, we don't recommend distributing wheels until RC1, at which point 3.13 will be available in cibuildwheel without the flag._ (#1816)
_While CPython is in beta, the ABI can change, so your wheels might not be compatible with the final release. For this reason, we don't recommend distributing wheels until RC1, at which point 3.13 will be available in cibuildwheel without the flag._ (#1815)

- ✨ Musllinux now defaults to `musllinux_1_2`. You can set the older `manylinux_1_1` via config if needed. (#1817)
- ✨ Musllinux now defaults to `musllinux_1_2`. You can set the older `musllinux_1_1` via config if needed. (#1817)
- 🛠 No longer pre-seed setuptools/wheel in virtual environments (#1819)
- 🛠 Respect the constraints file when building with pip, matching build (#1818)
- 🛠 Use uv to compile our pinned dependencies, 10x faster and doesn't require special setup (#1778)
Expand Down
4 changes: 4 additions & 0 deletions bin/generate_schema.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
#!/usr/bin/env python

# /// script
# dependencies = ["pyyaml"]
# ///

import argparse
import copy
import json
Expand Down
8 changes: 8 additions & 0 deletions bin/projects.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
#!/usr/bin/env python3

# /// script
# dependencies = [
# "click",
# "pyyaml",
# "pygithub",
# ]
# ///

"""
Convert a yaml project list into a nice table.

Expand Down
6 changes: 3 additions & 3 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ title: Changelog

_12 May 2024_

- ✨ Adds CPython 3.13 support, under the prerelease flag [CIBW_PRERELEASE_PYTHONS](https://cibuildwheel.pypa.io/en/stable/options/#prerelease-pythons). This version of cibuildwheel uses 3.13.0b1. Free-threading mode is not available yet, waiting on official binaries (planned for beta 2) and pip support.
- ✨ Adds CPython 3.13 support, under the prerelease flag [CIBW_PRERELEASE_PYTHONS](https://cibuildwheel.pypa.io/en/stable/options/#prerelease-pythons). This version of cibuildwheel uses 3.13.0b1. Free-threading mode is not available yet (#1657), waiting on official binaries (planned for beta 2) and pip support.

_While CPython is in beta, the ABI can change, so your wheels might not be compatible with the final release. For this reason, we don't recommend distributing wheels until RC1, at which point 3.13 will be available in cibuildwheel without the flag._ (#1816)
_While CPython is in beta, the ABI can change, so your wheels might not be compatible with the final release. For this reason, we don't recommend distributing wheels until RC1, at which point 3.13 will be available in cibuildwheel without the flag._ (#1815)

- ✨ Musllinux now defaults to `musllinux_1_2`. You can set the older `manylinux_1_1` via config if needed. (#1817)
- ✨ Musllinux now defaults to `musllinux_1_2`. You can set the older `musllinux_1_1` via config if needed. (#1817)
- 🛠 No longer pre-seed setuptools/wheel in virtual environments (#1819)
- 🛠 Respect the constraints file when building with pip, matching build (#1818)
- 🛠 Use uv to compile our pinned dependencies, 10x faster and doesn't require special setup (#1778)
Expand Down
14 changes: 7 additions & 7 deletions docs/data/projects.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
notes: A simple C extension, without external dependencies

- name: websockets
gh: aaugustin/websockets
gh: python-websockets/websockets
ci: [travisci]
ci_config: .travis.yml
os: [apple, linux]
Expand Down Expand Up @@ -325,12 +325,12 @@
notes: Full range of wheels for setuptools rust, with auto release and PyPI deploy.

- name: python-snappy
gh: andrix/python-snappy
gh: intake/python-snappy
ci: [github]
os: [apple, linux, windows]

- name: sourmash
gh: dib-lab/sourmash
gh: sourmash-bio/sourmash
ci: [github]
os: [apple, linux, windows]

Expand Down Expand Up @@ -418,7 +418,7 @@
notes: Tornado is a Python web framework and asynchronous networking library. Uses stable ABI for a small C extension.

- name: pytorch-fairseq
gh: pytorch/fairseq
gh: facebookresearch/fairseq
ci: [github]
os: [apple, linux]

Expand Down Expand Up @@ -475,7 +475,7 @@
os: [apple, linux, windows]

- name: OpenSpiel
gh: deepmind/open_spiel
gh: google-deepmind/open_spiel
ci: [github]
os: [apple, linux]

Expand Down Expand Up @@ -505,7 +505,7 @@
os: [apple, linux, windows]

- name: OpenTimelineIO
gh: PixarAnimationStudios/OpenTimelineIO
gh: AcademySoftwareFoundation/OpenTimelineIO
ci: [github]
os: [apple, linux, windows]

Expand Down Expand Up @@ -584,7 +584,7 @@
notes: Includes GPU support for linux wheels

- name: power-grid-model
gh: alliander-opensource/power-grid-model
gh: PowerGridModel/power-grid-model
ci: [github]
os: [windows, apple, linux]
notes: Python/C++ library for distribution power system analysis
Expand Down
9 changes: 6 additions & 3 deletions docs/main.py
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
from __future__ import annotations

import os
import subprocess
import sysconfig
from typing import Any

# Requires Python 3.7+


def define_env(env: Any) -> None:
"Hook function for mkdocs-macros"

@env.macro
def subprocess_run(*args: str) -> str:
"Run a subprocess and return the stdout"
return subprocess.run(args, check=True, capture_output=True, text=True).stdout
env = os.environ.copy()
scripts = sysconfig.get_path("scripts")
env["PATH"] = f"{scripts}{os.pathsep}{env.get('PATH', '')}"
return subprocess.run(args, check=True, capture_output=True, text=True, env=env).stdout
18 changes: 9 additions & 9 deletions docs/working-examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ title: Working examples
| [NCNN][] | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | ncnn is a high-performance neural network inference framework optimized for the mobile platform |
| [Prophet][] | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth. |
| [MyPy][] | ![github icon][] | ![apple icon][] ![linux icon][] ![windows icon][] | The compiled version of MyPy using MyPyC. |
| [Kivy][] | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Open source UI framework written in Python, running on Windows, Linux, macOS, Android and iOS |
| [duckdb][] | ![github icon][] | ![apple icon][] ![linux icon][] ![windows icon][] | DuckDB is an in-process SQL OLAP Database Management System |
| [Kivy][] | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Open source UI framework written in Python, running on Windows, Linux, macOS, Android and iOS |
| [MemRay][] | ![github icon][] | ![linux icon][] | Memray is a memory profiler for Python |
| [Triton][] | ![github icon][] | ![linux icon][] | Self hosted runners |
| [uvloop][] | ![github icon][] | ![apple icon][] ![linux icon][] | Ultra fast asyncio event loop. |
Expand Down Expand Up @@ -118,15 +118,15 @@ title: Working examples
| [pybind11 cross build example][] | ![github icon][] ![gitlab icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Same as pybind11 cmake_example but used to demo Linux ARM + Windows + macOS builds on GitLab |

[scikit-learn]: https://github.com/scikit-learn/scikit-learn
[pytorch-fairseq]: https://github.com/pytorch/fairseq
[pytorch-fairseq]: https://github.com/facebookresearch/fairseq
[NumPy]: https://github.com/numpy/numpy
[Tornado]: https://github.com/tornadoweb/tornado
[Matplotlib]: https://github.com/matplotlib/matplotlib
[NCNN]: https://github.com/Tencent/ncnn
[Prophet]: https://github.com/facebook/prophet
[MyPy]: https://github.com/mypyc/mypy_mypyc-wheels
[Kivy]: https://github.com/kivy/kivy
[duckdb]: https://github.com/duckdb/duckdb
[Kivy]: https://github.com/kivy/kivy
[MemRay]: https://github.com/bloomberg/memray
[Triton]: https://github.com/openai/triton
[uvloop]: https://github.com/MagicStack/uvloop
Expand All @@ -141,11 +141,11 @@ title: Working examples
[twisted-iocpsupport]: https://github.com/twisted/twisted-iocpsupport
[PyOxidizer]: https://github.com/indygreg/PyOxidizer
[cvxpy]: https://github.com/cvxpy/cvxpy
[websockets]: https://github.com/aaugustin/websockets
[websockets]: https://github.com/python-websockets/websockets
[pedalboard]: https://github.com/spotify/pedalboard
[River]: https://github.com/online-ml/river
[UltraJSON]: https://github.com/ultrajson/ultrajson
[OpenSpiel]: https://github.com/deepmind/open_spiel
[OpenSpiel]: https://github.com/google-deepmind/open_spiel
[aiortc]: https://github.com/aiortc/aiortc
[Confluent client for Kafka]: https://github.com/confluentinc/confluent-kafka-python
[Dependency Injector]: https://github.com/ets-labs/python-dependency-injector
Expand All @@ -169,7 +169,7 @@ title: Working examples
[aioquic]: https://github.com/aiortc/aioquic
[Psycopg 3]: https://github.com/psycopg/psycopg
[ruptures]: https://github.com/deepcharles/ruptures
[OpenTimelineIO]: https://github.com/PixarAnimationStudios/OpenTimelineIO
[OpenTimelineIO]: https://github.com/AcademySoftwareFoundation/OpenTimelineIO
[PyTables]: https://github.com/PyTables/PyTables
[Parselmouth]: https://github.com/YannickJadoul/Parselmouth
[google neuroglancer]: https://github.com/google/neuroglancer
Expand All @@ -186,9 +186,9 @@ title: Working examples
[dd-trace-py]: https://github.com/DataDog/dd-trace-py
[python-rapidjson]: https://github.com/python-rapidjson/python-rapidjson
[tgcalls]: https://github.com/MarshalX/tgcalls
[python-snappy]: https://github.com/andrix/python-snappy
[python-snappy]: https://github.com/intake/python-snappy
[pybind11 python_example]: https://github.com/pybind/python_example
[sourmash]: https://github.com/dib-lab/sourmash
[sourmash]: https://github.com/sourmash-bio/sourmash
[abess]: https://github.com/abess-team/abess
[cyvcf2]: https://github.com/brentp/cyvcf2
[matrixprofile]: https://github.com/matrix-profile-foundation/matrixprofile
Expand All @@ -203,7 +203,7 @@ title: Working examples
[bx-python]: https://github.com/bxlab/bx-python
[boost-histogram]: https://github.com/scikit-hep/boost-histogram
[Python-WebRTC]: https://github.com/MarshalX/python-webrtc
[power-grid-model]: https://github.com/alliander-opensource/power-grid-model
[power-grid-model]: https://github.com/PowerGridModel/power-grid-model
[pybase64]: https://github.com/mayeut/pybase64
[fathon]: https://github.com/stfbnc/fathon
[Imagecodecs (fork)]: https://github.com/czaki/imagecodecs_build
Expand Down
39 changes: 25 additions & 14 deletions noxfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,33 @@
import shutil
import sys
from pathlib import Path
from typing import Any

import nox

nox.needs_version = ">=2024.4.15"
nox.options.sessions = ["lint", "pylint", "check_manifest", "tests"]
nox.options.default_venv_backend = "uv|virtualenv"

DIR = Path(__file__).parent.resolve()

if os.environ.get("CI", None):
nox.options.error_on_missing_interpreters = True

def install_and_run(session: nox.Session, script: str, *args: str, **kwargs: Any) -> str | None:
deps = nox.project.load_toml(script)["dependencies"]
session.install(*deps)
return session.run("python", script, *args, **kwargs)


@nox.session
def tests(session: nox.Session) -> None:
"""
Run the unit and regular tests.
"""
unit_test_args = ["--run-docker"] if sys.platform.startswith("linux") else []
session.install("-e.[test]")
if session.posargs:
session.run("pytest", *session.posargs)
else:
unit_test_args = ["--run-docker"] if sys.platform.startswith("linux") else []
session.run("pytest", "unit_test", *unit_test_args)
session.run("pytest", "test", "-x", "--durations", "0", "--timeout=2400", "test")

Expand All @@ -44,8 +50,7 @@ def pylint(session: nox.Session) -> None:
Run pylint.
"""

name = "cibuildwheel @ ." if getattr(session.virtualenv, "venv_backend", "") == "uv" else "."
session.install("pylint", name)
session.install("pylint", ".")
session.run("pylint", "cibuildwheel", *session.posargs)


Expand All @@ -65,7 +70,7 @@ def update_constraints(session: nox.Session) -> None:
Update the dependencies inplace.
"""

if getattr(session.virtualenv, "venv_backend", "") != "uv":
if session.venv_backend != "uv":
session.install("uv>=0.1.23")

for minor_version in range(7, 14):
Expand Down Expand Up @@ -96,20 +101,19 @@ def update_pins(session: nox.Session) -> None:
"""
Update the python, docker and virtualenv pins version inplace.
"""
session.install("-e", ".[bin]")
session.install("-e.[bin]")
session.run("python", "bin/update_pythons.py", "--force")
session.run("python", "bin/update_docker.py")
session.run("python", "bin/update_virtualenv.py", "--force")


@nox.session
@nox.session(reuse_venv=True)
def update_proj(session: nox.Session) -> None:
"""
Update the README inplace.
"""
session.install("-e.[bin]")
session.run(
"python",
install_and_run(
session,
"bin/projects.py",
"docs/data/projects.yml",
*session.posargs,
Expand All @@ -121,13 +125,20 @@ def generate_schema(session: nox.Session) -> None:
"""
Generate the cibuildwheel.schema.json file.
"""
session.install("pyyaml")
output = session.run("python", "bin/generate_schema.py", silent=True)
output = install_and_run(session, "bin/generate_schema.py", silent=True)
assert isinstance(output, str)
DIR.joinpath("cibuildwheel/resources/cibuildwheel.schema.json").write_text(output)


@nox.session(python="3.9")
@nox.session(reuse_venv=True)
def bump_version(session: nox.Session) -> None:
"""
Bump cibuildwheel's version. Interactive.
"""
install_and_run(session, "bin/bump_version.py")


@nox.session(python="3.12")
def docs(session: nox.Session) -> None:
"""
Build the docs. Will serve unless --non-interactive
Expand Down