Roll out nightly and stable release packages

[ScottTodd]

Tasks

Target: this week (by Nov 1)

• Land new versioning scheme in sharktank and shortfin: Apply new versioning scheme #398
• Revamp sharktank packaging and releases #294 Build sharktank wheel and push to https://github.com/nod-ai/SHARK-Platform/releases/tag/dev-wheels
• Remove old version scheme packages from https://github.com/nod-ai/SHARK-Platform/releases/tag/dev-wheels
• Pick Draft a shark-platform meta package to organize Python packages. #244 back up and land the meta package
• Add instructions to README files for installing nightly releases Refresh user and developer guides for sharktank and shortfin #359

This will get us pip install shark-platform -f https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels

Target: next week (by Nov 8)

• Roll out new versioning scheme and renamed packages in IREE: [RFC] New versioning scheme for IREE Python packages iree-org/iree#18938
• Point sharktank and shortfin at the newly renamed packages
• Write scripts for promoting nightly releases to stable releases for pypi publishing (drop candidate / rc from names/versions)
• Publish stable releases for all packages
• User facing installation guide (repository root README, Refresh user and developer guides for sharktank and shortfin #359)

This will get us pip install shark-platform

Backlog

Wider support for Python versions

• Build and publish Python 3.13 and 3.13t wheels iree-org/iree#18652
• (sharktank) Python3.12+ not supported for torch.compile #349

[ScottTodd]

Researching how to promote versions from dev/rc --> stable, without needing to rebuild wheels:

• https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-versions.html
• https://hatch.pypa.io/latest/version/ (if we switch our build tool)
• https://pypi.org/project/change-wheel-version/ and https://github.com/hauntsaninja/change_wheel_version (could adapt/reference the source if the tool doesn't do everything we need)

[ScottTodd]

Nightly releases are now documented here: https://github.com/nod-ai/SHARK-Platform/blob/main/docs/nightly_releases.md

[ScottTodd]

We're making good progress on this.

Latest detailed task list (in order of rollout):

Publish iree-base-compiler and iree-base-runtime version 2.9.0 to nightly and PyPI

Today

• Merge Apply new versioning scheme iree-org/iree#19034
• Merge Rename base packages iree-org/iree#19061
• Trigger IREE release of iree-base-compiler and iree-base-runtime version 2.9.0 using https://github.com/iree-org/iree/actions/workflows/oneshot_candidate_release.yml or https://github.com/iree-org/iree/actions/workflows/schedule_candidate_release.yml
• Drop 'rcYYYYMMDD' suffix from packages when deploying to PyPI. iree-org/iree#19067
• Promote release from one-shot or nightly to pypi using new package name

Update downstream projects to new package names and versions

Late today / tomorrow

• Update docs, experimental and samples iree-org/iree#19065
• Switch to new IREE package names and versions. iree-org/iree-turbine#258
• Decrement sharktank and shortfin versions to 2.9.0. #448 then delete existing 3.0.0 dev wheels
• Switch to new IREE package names and versions. #449
• [shortfin] Get the version number from file #454

Complete the packaging for shark-platform

Late today / tomorrow

• Build shark-platform meta-package that pins explicit versions of iree-turbine, iree-base-compiler, and iree-base-runtime. Allow for patch versions to drift?
• Start publishing shark-platform meta-package to dev wheels (nightly)
• Set sharktank package deps during package build reliably. #440
• Publish sharktank, shortfin, and shark-platform 2.9.0 packages to pypi

Finish 3.0.0 release

Next week

• Update versions to 3.0.0 for candidate releases
• Wait for release branch cut
• Cut 3.0.0 releases for each package
• Push all packages to PyPI
• Finalize release notes and user documentation to use the PyPI packages

[ScottTodd]

I've been stuck all day trying to update iree-turbine to use the new iree-base-compiler and iree-base-runtime packages: iree-org/iree-turbine#258. There are new test failures that look meaningful.

Without that change, the iree-turbine package will still depend on the old IREE package names, so users will download both iree-compiler and iree-base-compiler packages. That is already happening today in some workflows here, due to #449.

We might actually be able to proceed with the redundant installs for a little bit, but I'm not sure what the implications will be.

[ScottTodd]

Figured out all the test failures and pushed iree-turbine 2.9.0:

• https://github.com/iree-org/iree-turbine/releases/tag/v2.9.0
• https://pypi.org/project/iree-turbine/2.9.0/

[ScottTodd]

I'm going to call it for the day. Next steps for Monday:

• Check https://github.com/nod-ai/SHARK-Platform/releases/tag/dev-wheels to see if the shortfin and sharktank rc wheels there are suitable for pushing to pypi
  • See if installing them pulls in iree-turbine, iree-base-compiler, and iree-base-runtime versions 2.9.0 from PyPI as expected
• Fork https://github.com/iree-org/iree/blob/main/build_tools/python_deploy/promote_whl_from_rc_to_final.py to https://github.com/nod-ai/SHARK-Platform/tree/main/build_tools
• Use that script to drop the rcYYYYMMDD part of the .whl files
• Upload the new wheel files to https://pypi.org/project/sharktank/ and https://pypi.org/project/shortfin/

Once we have that initially working, we can also add the shark-platform meta package that directly pins the versions of each package that we want, then do the same build -> publish to dev wheels -> push to pypi process for that package.

[ScottTodd]

Initial 2.9.0 releases of all packages have been published to PyPI. More work remains to get the process running smoothly, but we should have enough for testing using standard pip commands with no extra package index or source builds.