[libshortfin] Add package build script for Linux using manylinux. #230
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ScottTodd
force-pushed
the
libshortfin-package-manylinux
branch
from
0ce1f57 to
9223811
Compare
7 tasks
marbre
reviewed
Sep 30, 2024
| } | ||
|
|
||
| function build_shortfin() { | ||
| export SHORTFIN_ENABLE_TRACING=ON |
There was a problem hiding this comment.
Do we always want to build _shortfin_tracy and have it in the same wheel?
There was a problem hiding this comment.
I think so, matching IREE: https://iree.dev/reference/bindings/python/#profiling
The lib.cpython-313-x86_64-linux-gnu.so is currently 450 KB in _shortfin_default and 851 KB in _shortfin_tracy. We could split them into separate packages or an "extra", like shortfin[tracing]: https://packaging.python.org/en/latest/specifications/dependency-specifiers/#extras
marbre
approved these changes
Sep 30, 2024
ScottTodd
added a commit
that referenced
this pull request
Sep 30, 2024
Progress on #130, follow-up to #230. ## Overview This gets us the general structure of a release pipeline that: 1. Computes some version information metadata based on build id and date 2. Calls `shortfin/build_tools/build_linux_package.sh`, which runs `python -m pip wheel` under a manylinux Docker container 3. Uploads wheels to both GitHub artifacts (for the workflow run itself) and GitHub releases (for archival and developer usage) This follows how stablehlo (https://github.com/openxla/stablehlo/releases/tag/dev-wheels) and torch-mlir (https://github.com/llvm/torch-mlir-release/releases/tag/dev-wheels) both publish to a constantly growing "dev-wheels" release rather than separate releases like IREE does. That makes it harder to directly "promote" a candidate release, but it avoids polluting the release history with nightly builds. ## Testing * Sample run: https://github.com/ScottTodd/SHARK-Platform/actions/runs/11078636913 * Dev wheels release: https://github.com/ScottTodd/SHARK-Platform/releases/tag/dev-wheels * Install with `python -m pip install shortfin -f https://github.com/ScottTodd/SHARK-Platform/releases/expanded_assets/dev-wheels` ## What's next In no particular order, - [ ] Verify that the Tracy build enabled by `SHORTFIN_ENABLE_TRACING` is functional - [ ] Set up dependencies and optional components such that a user can just `pip install shortfin` and have it bring along a compatible version of packages like `iree-runtime` and `iree-compiler` - [ ] Update the dockerfile version from manylinux2014 (that is independent of this workflow) - [x] Build for Python 3.13 free threaded and test that - [ ] Build for other platforms/architectures (macOS, Windows, arm64, etc.) - [ ] Set up a workflow like IREE's pkgci that installs the packages and runs some tests on them - [ ] Document how to install the developer packages --------- Co-authored-by: Marius Brehler <marius.brehler@gmail.com>
ScottTodd
commented
Oct 1, 2024
There was a problem hiding this comment.
Oh, from iree-org/iree#15856:
I think we can actually completely forgo most of the gunk in build_linux_packages.sh (all of the docker stuff). For other projects, I'm either using GHA action runners that are already the right OS or telling GHA to run in the context of the manylinux docker image. Makes it really, really simple.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Progress on #130
This is forked from
TODO (future PRs?):