Add publish and release workflows

[flying-sheep]

Fixes #46

Based on https://github.com/pydantic/pydantic-core/blob/6472887b3ad3e865e494b46efb66a71e87ceb1cf/.github/workflows/ci.yml#L399-L500

Release flow

Trigger a build

1. Trigger build by adding Full Build cause CI to do a full build to a PR (as seen here), pushing a tag to a repo (any tag will do it), or pushing to main
2. Wheel builds for all supported architectures happen
3. Some checks happen on the collected wheels

Create an actual release on PyPI and GitHub

1. push a tag matching semver (e.g. v1.2.3 or v1.2.4a1)
2. as above
3. as above
4. the wheels are uploaded to new PyPI release
5. a pretty spartan GitHub release is made

Alternatives:

• https://python-semantic-release.readthedocs.io/en/latest/automatic-releases/github-actions.html
• release-plz

Why not release-plz

The way release-plz works is as follows:

• it expects to be triggered by a push event to the main branch
  • if the last pushed commit is a merged Release PR (see below), it executes the command release which creates a GitHub release and a crates.io release
  • if the last pushed commit is something else, it executes the command release-pr which creates or updates a Release PR that bumps the versions and appends to a changelog

unfortunately a GitHub-only mode (release-plz/release-plz#1144) is not yet implemented, so unless we want to publish to crates.io, we need to implement the release command ourselves. Might be easy or not.

TODOs

• make sure the stub generator is built and check for the stubs in the wheels
• add check if wheels are importable and so on
• add check if tag matches version on release or change the flow
• research if we should change some parameters, e.g. musllinux_1_{1->2} or manylinux_{2_8->1_7} or so
• test the publish step (temporarily modify it to dry-run the PyPI publish step, then …)
  • create a semver tag to see if the GitHub release is created (we might we need to add some permissions or set GITHUB_TOKEN to make it work)
  • create a non-semver tab to see if the publish step is skipped
• use PEP 440 instead of semver parsing

TODO build errors:

• linux x86_64/i686: openssl-sys (docs). Seems like it wants to vendor it but fails? Maybe we need to install more perl stuff into the build env? (https://github.com/ilan-gold/zarrs-python/actions/runs/11895077573/job/33143798322?pr=47)

• linux aarch64/armv7/ppc64le/s390x: snappy-src. “aarch64-unknown-linux-gnu-g++: error: unrecognized command line option '-std=c++14'“ (https://github.com/ilan-gold/zarrs-python/actions/runs/11895077573/job/33143798957?pr=47)

  Seems to be solved if we use manylinux_2_28 instead of manylinux2014

[flying-sheep]

@LDeakin I’m not sure if installing OpenSSL is the right move. It might result in symlinking to a version that isn’t guaranteed to be there.

It’s unfortunate but I don’t think there’s a version of OpenSSL that can be readily installed in all Linux distributions, is there?

[LDeakin]

Yea not sure, I just went with sfackler/rust-openssl#2036 (comment), which seemed popular

[LDeakin]

I'm thinking of giving up on i686, also do we need non-musl x86 x86_64 as well?

[flying-sheep]

Yea not sure, I just went with sfackler/rust-openssl#2036 (comment), which seemed popular

Doesn‘t mean that it actually produces a usable result. I think there are two possible ways to do this:

1. link against a stable ABI. If OpenSSL has that and we can check if we‘re linking against it (e.g. openssl.so.1 and not say openssl.1.3.6), we’re good
2. statically link it. I’m pretty sure that’s what’s happening on the other platforms, that’s why they didn’t make problems. But it does have the disadvantage that we’re in charge of delivering security fixes – which amounts to frequent releases.

I’d much prefer 1., but I don’t know if it’s possible, i.e. if OpenSSL offers a stable ABI.

I'm thinking of giving up on i686, also do we need non-musl x86 x86_64 as well?

Ah yeah! I think I need to go over this and check if there are any other platforms we’re missing because I only realized halfway through that I copied the wrong include section

[LDeakin]

OpenSSL issue fixed properly with d4e8bd7

I'll take my hands off this for a bit if you want to have a crack at the remaining platforms.

[flying-sheep]

I’m good, I’m enjoying my evening. I just saw you working on this and wanted to share the tidbits I knew but hadn’t written down yet. I enabled linux x86_64 rn though haha!

I copied it from here and missed the entries they used for PGO builds. We also need to check that the wheels that each job makes is correct, e.g. it’s a bit suspect to me that only that windows i686 build has python-architecture: x86.

This workflow also needs quite some more parts like making a GH release and so on, and tying other things together.

Finally, we should merge #38 first, I really think there shouldn’t be two different rust crates called “zarrs” in the same build, and we should make sure the stubs end up in the wheels.

[LDeakin]

Sorry for all the CI spam tonight @flying-sheep.

The x86_64 wheels work on windows/linux 🚀.
Adding _internal.pyi to VC with 48ff131 was necessary to get it in the wheels.

Any other blocking TODOs?
@ilan-gold, is PyPi trusted publishing setup?

[flying-sheep]

The release PR job isn’t functional yet: I don’t think release-plz works without releasing to crates.

And it would try to release to PyPI for every tag, that needs to be fixed as well.

[ilan-gold]

@LDeakin no I needed the file name of the release-triggering yml first - I didn't want to just guess at it because I wasn't clear on the consequences of a "wrong guess." But once this merges, I can do that.

[flying-sheep]

@LDeakin adding the stub to VCS makes things harder in other ways: now pre-commit wants to format it, whereas build.rs regenerates the unformatted one.

And where do we want to run ruff from? pre-commit run --files=python/zarrs/_internal.pyi ruff-format? ruff format python/zarrs/_internal.pyi? Neither is in the PATH for my VS Code right now …

The only thing I can think of would be to

• commit it
• format it
• check during the tests that regenerating and formatting it doesn’t create a diff

[ilan-gold]

What is up with the submodule?

[flying-sheep]

What is up with the submodule?

no clue, it always pops up out of nowhere!

[LDeakin]

add check if tag matches version on release or change the flow

@flying-sheep, is 64cf651 roughly what you had in mind?

test the publish step (temporarily modify it to dry-run the PyPI publish step, then …)

Shall we try publish to test.pypi.org with 83fbb0c?

[flying-sheep]

OK let me specify what I want to do:

1. identify if we have a version tag (v and then a PEP 440 compliant version)
2. if no (i.e. no tag or a tag that’s no version), just build, but then exit (successfully) without publishing
3. if yes, get the version from both tag and package metadata, then
   1. check if the tag version is properly normalized or fail (e.g. note how this gets changed: str(packaging.version.Version("1.2.0.alpha1")) == '1.2.0a1')
   2. check if the tag version matches the metadata version or fail
4. publish