-
-
Notifications
You must be signed in to change notification settings - Fork 640
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upload Wheels and Per-platform PEXs to the Github Release #19431
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So far I've got:
$ rm -rf dist/gh_upload/
$ pants run src/python/pants_release/release.py -- build-local-pex --dest dist/gh_upload
josh@cephandrius:~/work/pants$ NO_SCIE_WARNING=1 ./dist/gh_upload/pants.cp39-linux_x86_64.pex --pants-config-files=[] --version
2.18.0.dev2+202306261651.git073c224f
$ rm -rf dist/gh_upload/
$ pants run src/python/pants_release/release.py -- build-local-pex --dest dist/gh_upload --stable
$ NO_SCIE_WARNING=1 ./dist/gh_upload/pants.cp39-linux_x86_64.pex --pants-config-files=[] --version
2.18.0.dev2
ls
the directory (after removing wheels) gives me the pex and both pantsbuild wheels.
BUILD_LOCAL_PEX_ARGS=("--dest" "dist/gh_upload") | ||
if [[ "${{ needs.determine_ref.outputs.is-release }}" == "true" ]]; then | ||
BUILD_LOCAL_PEX_ARGS+=("--stable") | ||
fi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested this locally using act
@@ -1009,19 +1025,19 @@ def release_jobs_and_inputs() -> tuple[Jobs, dict[str, Any]]: | |||
pypi_release_dir = "dest/pypi_release" | |||
helper = Helper(Platform.LINUX_X86_64) | |||
wheels_jobs = build_wheels_jobs( | |||
needs=["determine_ref"], for_deploy_ref=gha_expr("needs.determine_ref.outputs.build-ref") | |||
needs=["release_info"], for_deploy_ref=gha_expr("needs.release_info.outputs.build-ref") | |||
) | |||
wheels_job_names = tuple(wheels_jobs.keys()) | |||
jobs = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The job and step got renamed to be more accurate.
Re-reading #19028 gives this quote from @stuhood
So alternatively, we could just publish the lockfile and not the PEX itself. It's hard to weigh all the pros/cons, realistically. Also, I'm wondering if we continue to not use the PEX, and instead the published wheel from That would be completely sufficient if we did #19282 So, maybe we adjust the PR to just upload the wheels to GitHub, and the rest still follows and we avoid building PEXs at all? |
....And while I think about it, another thing to consider is backporting. We can easily download wheels from S3 or PyPI and upload those to earlier releases. And (and I'm not sure I'm proposing this, I'm just documenting) if we stopped supporting |
OK I'm splitting this PR into several smaller ones |
…e assets (#19432) After #19431 is merged and we're releasing wheels as GitHub Release assets, if/when we stop publishing to PyPI we need an alternate solution for pants plugin requirements. This change switches the `pants_requirements` target generator in two ways: - Allows for a version to be specified (defaulting to the current running version) which is matched exactly - Populates a private `_find_links` field with a value pointing to the relevant assets list for the release Then, the necessary plumbing is done for the new `_find_links` field, and testing. Local testing: ```console josh@cephandrius:~/work/techlabs$ cat pants-plugins/BUILD.pants | head -n6 pants_requirements( name = "pants", resolve = "pants-plugins", version="2.16.0.dev5", testutil=False ) josh@cephandrius:~/work/techlabs$ PANTS_SOURCE=../pants pants generate-lockfiles --resolve=pants-plugins Pantsd has been turned off via Env. 21:13:13.20 [INFO] Completed: Generate lockfile for pants-plugins 21:13:13.21 [INFO] Wrote lockfile for the resolve `pants-plugins` to 3rdparty/python/lockfiles/pants-plugins.lock josh@cephandrius:~/work/techlabs$ grep -v '^//' 3rdparty/python/lockfiles/pants-plugins.lock | jq '.locked_resolves[0].locked_requirements[] | select(.project_name == "pantsbuild-pants")' { "artifacts": [ { "algorithm": "sha256", "hash": "ea66ac27d983d4c7b3252845fbea635f4ae49ebd5fd5d139a167eb173a5dc5e0", "url": "https://github.com/pantsbuild/pants/releases/download/release_2.16.0.dev5/pantsbuild.pants-2.16.0.dev5-cp39-cp39-manylinux2014_x86_64.whl" }, { "algorithm": "sha256", "hash": "5428b46f33d824a8066a6d5c44379bce678cd3c504a07e06d214a6665fa8604a", "url": "https://github.com/pantsbuild/pants/releases/download/release_2.16.0.dev5/pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_10_15_x86_64.whl" }, { "algorithm": "sha256", "hash": "f97f91ecf9913f0b1d2898f03326ac567e0e5e7df5bd1638536e1e87a566cd47", "url": "https://github.com/pantsbuild/pants/releases/download/release_2.16.0.dev5/pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_11_0_arm64.whl" }, { "algorithm": "sha256", "hash": "c7e4e9a1a88f6ee1fac2d62fc52caebcb66029458fe1c1978b730b7a5bc8823a", "url": "https://github.com/pantsbuild/pants/releases/download/release_2.16.0.dev5/pantsbuild.pants-2.16.0.dev5-cp39-cp39-manylinux2014_aarch64.whl" } ], "project_name": "pantsbuild-pants", "requires_dists": [ "PyYAML<7.0,>=6.0", "ansicolors==1.1.8", "chevron==0.14.0", "fasteners==0.16.3", "humbug==0.2.7", "ijson==3.1.4", "importlib-resources==5.0.*", "packaging==21.3", "pex==2.1.120", "psutil==5.9.0", "python-lsp-jsonrpc==1.0.0", "setproctitle==1.3.2", "setuptools<64.0,>=63.1.0", "toml==0.10.2", "types-PyYAML==6.0.3", "types-setuptools==62.6.1", "types-toml==0.10.8", "typing-extensions==4.3.0" ], "requires_python": "<3.10,>=3.7", "version": "2.16.0.dev5" } ```
This change does a few things:
dist/gh_upload
to the release (in each wheel job)release.py
to do what we want for the new worldNote that the only backwards-incompatible changes are we are no longer building one universal PEX, and instead building one per platform. We 're still uploading to S3, etc...
Tweaks the the release action
First, we change the release action to create the release, instead of a human doing it.
--prerelease
(we weren't consistent about this previously)dev_<SHA>
and it will always be--prerelease
Tweaks to
release.py
build-local-pex
now takes a destination and optionally a--stable
argument(s)gh_uploads
to make uploading easier--stable
is passed if this is a real releasex86_64
)pants.<VERSION>.pex
) the wheels are unstable/stable versionedTweaks to docs
We get to remove all of step 4 🎉
Next steps
scie-pants
to use the uploaded PEXs and avoid PyPI altogetherpants_requirements
to use the uploaded wheels from the release, and avoid PyPI altogetherpants_requirements
from direct wheel links to GitHub Release assets #19432Then,
Fixes #12397 (at least, the Pants side, we'd wanna update scie-pants)