Include prereleases from downstream projects in pip release links page.

[ScottTodd]

This fixes #19193.

Nightly releases of packages from https://github.com/iree-org/iree-turbine will now be available from https://iree.dev/pip-release-links.html with (for example)

pip install --pre --find-links https://iree.dev/pip-release-links.html iree-turbine

Previously, this relied on its own index page at https://github.com/iree-org/iree-turbine/releases/expanded_assets/dev-wheels . While this link will continue to work, the consolidated page is more convenient for users.

Projects included

I'm starting with just iree-turbine. See the notes below about other packages.

We could still switch to a PEP 503-compliant package index (usable with --extra-index-url, not -f, --find-links), like what PyTorch provides at https://download.pytorch.org/whl/cpu. Note the 50+ packages listed there, included mirrors of a few versions of all of PyTorch's dependencies.

Testing

• Publish action: https://github.com/ScottTodd/iree/actions/runs/12188757072/job/34002585332

• Pushed to GitHub pages: https://scotttodd.github.io/iree/pip-release-links.html

• Installed using the release links:

  python3.11 -m venv 3.11.venv
  source 3.11.venv/bin/activate
  
  pip install --pre --find-links https://scotttodd.github.io/iree/pip-release-links.html iree-turbine --no-deps
  # Looking in links: https://scotttodd.github.io/iree/pip-release-links.html
  # Collecting iree-turbine
  # Downloading https://github.com/iree-org/iree-turbine/releases/download/dev-wheels/iree_turbine-3.1.0rc20241205-py3-none-any.whl (292 kB)
  #   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.3/292.3 kB 8.1 MB/s eta 0:00:00
  # Installing collected packages: iree-turbine
  # Successfully installed iree-turbine-3.1.0rc20241205

Other considerations

We may want to also include downstream packages like https://github.com/nod-ai/shark-ai (release index page: https://github.com/nod-ai/shark-ai/releases/expanded_assets/dev-wheels). I'm currently trying to simplify workflows for that project (example here).

I left some notes in the workflow file too, but I could see us extending this to also include torch-mlir and stablehlo packages. The extra release scraping doesn't cost us much CI time, git storage on the gh-pages branch, or network bandwidth on the website. I also don't foresee any issues with making it easier to install nightly versions of those packages. I would pause a bit if the list grows too much beyond that though.

Note on the shark-ai package

The shark-ai meta package (https://pypi.org/project/shark-ai/, https://github.com/nod-ai/shark-ai/tree/main/shark-ai) pins dependencies in ways that isn't currently compatible with this package installation flow. Specifically, the latest nightly releases pin shortfin==3.1.0 which can't be resolved by a release candidate like 3.1.0rc20241205, so this doesn't work as one might expect:

pip install --pre --find-links https://iree.dev/pip-release-links.html shark-ai
# Successfully installed shark-ai-3.0.0

# Test with an explicit version to demonstrate the issue:
pip install --pre --find-links https://scotttodd.github.io/iree/pip-release-links.html \
  shark-ai==3.1.0rc20241205
# ERROR: Could not find a version that satisfies the requirement shortfin==3.1.0 (from shark-ai) (from versions: 2.9.0rc20241108, 2.9.0rc20241109, 2.9.0rc20241110, 2.9.0rc20241111, 2.9.0, 2.9.1rc20241112, 2.9.1rc20241113, 2.9.1rc20241114, 2.9.1, 2.9.2rc20241115, 2.9.2rc20241116, 2.9.2rc20241117, 2.9.2rc20241118, 2.9.2, 3.0.0rc20241118, 3.0.0, 3.1.0rc20241119, 3.1.0rc20241120, 3.1.0rc20241121, 3.1.0rc20241122, 3.1.0rc20241123, 3.1.0rc20241124, 3.1.0rc20241125, 3.1.0rc20241126, 3.1.0rc20241127, 3.1.0rc20241128, 3.1.0rc20241129, 3.1.0rc20241130, 3.1.0rc20241201, 3.1.0rc20241202, 3.1.0rc20241203, 3.1.0rc20241204, 3.1.0rc20241205)
# ERROR: No matching distribution found for shortfin==3.1.0

[marbre]

I am not convinced that any non iree-org package should be included in the index. It might be reasonable to publish links for dependencies that are required but I don't know what those would be at the moment. The dependencies are available via PyPI and wheels like the ones provided via https://github.com/llvm/torch-mlir-release are not direct dependencies. Furthermore, it doesn't seem to be the right place to incclude downstream packages that make use of IREE. I would rather expect that those packages might include IREE in their index instead but here the other way around is proposed. I might miss the rational for this, so I am happy to discuss!

[ScottTodd]

I'm fine with just including iree-org/iree-turbine in addition to this iree-org/iree repository in this PR and seeing how far that gets us.

I am not convinced that any non iree-org package should be included in the index. It might be reasonable to publish links for dependencies that are required but I don't know what those would be at the moment. The dependencies are available via PyPI and wheels like the ones provided via https://github.com/llvm/torch-mlir-release are not direct dependencies.

Here's my thinking for including other packages:

• I consider the nightly releases in general to be a convenience we provide for developers and automated workflows, not a fully supported path that we want users to depend on. Most users should be using stable releases, and we need to keep the release train going on that. Our current SLO is a stable release every 6 weeks, but we could increase that frequency.
• Including more packages increases the convenience factor, at little technical cost. The larger costs are social - does including a package in the index imply some association or support status?

Furthermore, it doesn't seem to be the right place to incclude downstream packages that make use of IREE. I would rather expect that those packages might include IREE in their index instead but here the other way around is proposed.

Sure, we could have each downstream project provide its own package index too, but I'd rather avoid the scenario where the nightly package install instructions are different depending on which downstream project(s) you are using.

We've had #9532 open highlighting the issue with having multiple package indices:

pip install \
  nodai-shark \
  -f https://github.com/nod-ai/SHARK/releases \
  -f https://github.com/llvm/torch-mlir/releases \
  -f https://github.com/nod-ai/shark-runtime/releases \
  --extra-index-url https://download.pytorch.org/whl/nightly/cpu`

That was two years ago, and today it looks something like this:

pip install \
  shortfin \
  -f https://github.com/nod-ai/shark-ai/releases/expanded_assets/dev-wheels \
  -f https://iree.dev/pip-release-links.html

We could publish a downstream https://nod-ai.github.io/shark-ai/pip-release-links that includes IREE nightly packages, but then other downstream projects may want to do the same, and it could end up fragmenting documentation, scripts, reproducer instructions, etc.

A more "final" fix would be to push pre-releases to the official package index on https://pypi.org/, but then we run into storage and bandwidth limits more quickly and the project release history looks messier. For example, see how stable releases are rc0/rc1/etc. candidates are clearly visible on https://pypi.org/project/tensorflow/#history compared to the stream of nightly releases on https://pypi.org/project/tf-nightly/#history.

Some related discussions in other parts of the Python packaging ecosystem, mostly around having a warehouse (pypi) API to set retention policies on pre-releases pushed there, avoiding the need to maintain project-specific package indices:

• Geoffrey's analysis here: Improved package index for IREE pre-releases #9532 (comment)
• https://discuss.python.org/t/publishing-nightly-builds-on-test-pypi-org-with-a-time-based-retention-policy/3152/9
• Support for nightly releases in pypi pypi/warehouse#8792

---

All that being said, my gut feeling is "eh, why not create this upstream index for any projects that use IREE to use?". There's a similar precedent in https://anaconda.org/scientific-python-nightly-wheels hosting 29 packages, many from different organizations like:

• https://github.com/scipy/scipy
• https://github.com/statsmodels/statsmodels
• https://github.com/pandas-dev/pandas
• https://github.com/contourpy/contourpy
• etc.

If we were to do that, anchoring on IREE as the host makes sense to me, but I could see an argument for MLIR, LLVM, the Linux Foundation, or some other super-project / organization hosting.

[ScottTodd]

Switched to just --repos="iree-org/iree,iree-org/iree-turbine" and updated the PR description.

[marbre]

Thanks for doing this @ScottTodd. Let us follow up on the discussion we just started here next week.