Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload Wheels and Per-platform PEXs to the Github Release #19431

Closed
wants to merge 7 commits into from

Conversation

thejcannon
Copy link
Member

@thejcannon thejcannon commented Jul 7, 2023

This change does a few things:

  • Tweaks the release action to create a new release and upload everything in dist/gh_upload to the release (in each wheel job)
  • Tweaks release.py to do what we want for the new world
  • Tweaks the docs to reflect the changes

Note that the only backwards-incompatible changes are we are no longer building one universal PEX, and instead building one per platform. We 're still uploading to S3, etc...

Tweaks the the release action

First, we change the release action to create the release, instead of a human doing it.

  • If this is a real release, will set the title to the release tag, and if the version doesn't have any letters, tags as --prerelease (we weren't consistent about this previously)
  • If this is a "dev release" (those which people request to have wheels built) the title will be dev_<SHA> and it will always be --prerelease

Tweaks to release.py

  • The universal PEX command was removed, and the local PEX is the one-and-only option
  • build-local-pex now takes a destination and optionally a --stable argument(s)
    • the dest is set to gh_uploads to make uploading easier
    • --stable is passed if this is a real release
  • It now builds the local PEX, with the name tweaked to include the machine (e.g. x86_64)
  • The pex is always named simply (e.g. pants.<VERSION>.pex) the wheels are unstable/stable versioned

Tweaks to docs

We get to remove all of step 4 🎉

Next steps

Then,

  • Stop uploading to S3?
  • Stop uploading to PyPI?
  • Rip out a bunch of release code?

Fixes #12397 (at least, the Pants side, we'd wanna update scie-pants)

@thejcannon thejcannon added the category:internal CI, fixes for not-yet-released features, etc. label Jul 7, 2023
Copy link
Member Author

@thejcannon thejcannon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far I've got:

$ rm -rf dist/gh_upload/
$ pants run src/python/pants_release/release.py -- build-local-pex --dest dist/gh_upload
josh@cephandrius:~/work/pants$ NO_SCIE_WARNING=1 ./dist/gh_upload/pants.cp39-linux_x86_64.pex --pants-config-files=[] --version
2.18.0.dev2+202306261651.git073c224f

$ rm -rf dist/gh_upload/
$ pants run src/python/pants_release/release.py -- build-local-pex --dest dist/gh_upload --stable
$ NO_SCIE_WARNING=1 ./dist/gh_upload/pants.cp39-linux_x86_64.pex --pants-config-files=[] --version
2.18.0.dev2

ls the directory (after removing wheels) gives me the pex and both pantsbuild wheels.

Comment on lines 524 to 527
BUILD_LOCAL_PEX_ARGS=("--dest" "dist/gh_upload")
if [[ "${{ needs.determine_ref.outputs.is-release }}" == "true" ]]; then
BUILD_LOCAL_PEX_ARGS+=("--stable")
fi
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this locally using act

@@ -1009,19 +1025,19 @@ def release_jobs_and_inputs() -> tuple[Jobs, dict[str, Any]]:
pypi_release_dir = "dest/pypi_release"
helper = Helper(Platform.LINUX_X86_64)
wheels_jobs = build_wheels_jobs(
needs=["determine_ref"], for_deploy_ref=gha_expr("needs.determine_ref.outputs.build-ref")
needs=["release_info"], for_deploy_ref=gha_expr("needs.release_info.outputs.build-ref")
)
wheels_job_names = tuple(wheels_jobs.keys())
jobs = {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The job and step got renamed to be more accurate.

@thejcannon thejcannon marked this pull request as ready for review July 7, 2023 18:04
@thejcannon
Copy link
Member Author

Re-reading #19028 gives this quote from @stuhood

A PEX lockfile is effectively equivalent to a PEX. And so I think that if scie-pants were to begin consuming a released PEX lockfile to build venvs?

So alternatively, we could just publish the lockfile and not the PEX itself. It's hard to weigh all the pros/cons, realistically.

Also, I'm wondering if we continue to not use the PEX, and instead the published wheel from scie-pants? E.g. the find-links uses the release page.

That would be completely sufficient if we did #19282

So, maybe we adjust the PR to just upload the wheels to GitHub, and the rest still follows and we avoid building PEXs at all?

@thejcannon
Copy link
Member Author

thejcannon commented Jul 8, 2023

....And while I think about it, another thing to consider is backporting. We can easily download wheels from S3 or PyPI and upload those to earlier releases.

And (and I'm not sure I'm proposing this, I'm just documenting) if we stopped supporting PANTS_SHA (even for older releases) we could shut down per-sha wheel hosting on binaries.pantsbuild.org)

@thejcannon thejcannon marked this pull request as draft July 10, 2023 16:33
@thejcannon
Copy link
Member Author

OK I'm splitting this PR into several smaller ones

@thejcannon thejcannon closed this Jul 10, 2023
thejcannon added a commit that referenced this pull request Aug 6, 2023
…e assets (#19432)

After #19431 is merged and we're
releasing wheels as GitHub Release assets, if/when we stop publishing to
PyPI we need an alternate solution for pants plugin requirements.

This change switches the `pants_requirements` target generator in two
ways:
- Allows for a version to be specified (defaulting to the current
running version) which is matched exactly
- Populates a private `_find_links` field with a value pointing to the
relevant assets list for the release

Then, the necessary plumbing is done for the new `_find_links` field,
and testing.

Local testing:

```console
josh@cephandrius:~/work/techlabs$ cat pants-plugins/BUILD.pants | head -n6
pants_requirements(
    name = "pants",
    resolve = "pants-plugins",
    version="2.16.0.dev5",
    testutil=False
)
josh@cephandrius:~/work/techlabs$ PANTS_SOURCE=../pants pants generate-lockfiles --resolve=pants-plugins                                                           
Pantsd has been turned off via Env.
21:13:13.20 [INFO] Completed: Generate lockfile for pants-plugins                                                                                                  
21:13:13.21 [INFO] Wrote lockfile for the resolve `pants-plugins` to 3rdparty/python/lockfiles/pants-plugins.lock
josh@cephandrius:~/work/techlabs$ grep -v '^//'  3rdparty/python/lockfiles/pants-plugins.lock | jq '.locked_resolves[0].locked_requirements[] | select(.project_name == "pantsbuild-pants")'
{
  "artifacts": [
    {
      "algorithm": "sha256",
      "hash": "ea66ac27d983d4c7b3252845fbea635f4ae49ebd5fd5d139a167eb173a5dc5e0",
      "url": "https://github.com/pantsbuild/pants/releases/download/release_2.16.0.dev5/pantsbuild.pants-2.16.0.dev5-cp39-cp39-manylinux2014_x86_64.whl"
    },
    {
      "algorithm": "sha256",
      "hash": "5428b46f33d824a8066a6d5c44379bce678cd3c504a07e06d214a6665fa8604a",
      "url": "https://github.com/pantsbuild/pants/releases/download/release_2.16.0.dev5/pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_10_15_x86_64.whl"
    },
    {
      "algorithm": "sha256",
      "hash": "f97f91ecf9913f0b1d2898f03326ac567e0e5e7df5bd1638536e1e87a566cd47",
      "url": "https://github.com/pantsbuild/pants/releases/download/release_2.16.0.dev5/pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_11_0_arm64.whl"
    },
    {
      "algorithm": "sha256",
      "hash": "c7e4e9a1a88f6ee1fac2d62fc52caebcb66029458fe1c1978b730b7a5bc8823a",
      "url": "https://github.com/pantsbuild/pants/releases/download/release_2.16.0.dev5/pantsbuild.pants-2.16.0.dev5-cp39-cp39-manylinux2014_aarch64.whl"
    }
  ],
  "project_name": "pantsbuild-pants",
  "requires_dists": [
    "PyYAML<7.0,>=6.0",
    "ansicolors==1.1.8",
    "chevron==0.14.0",
    "fasteners==0.16.3",
    "humbug==0.2.7",
    "ijson==3.1.4",
    "importlib-resources==5.0.*",
    "packaging==21.3",
    "pex==2.1.120",
    "psutil==5.9.0",
    "python-lsp-jsonrpc==1.0.0",
    "setproctitle==1.3.2",
    "setuptools<64.0,>=63.1.0",
    "toml==0.10.2",
    "types-PyYAML==6.0.3",
    "types-setuptools==62.6.1",
    "types-toml==0.10.8",
    "typing-extensions==4.3.0"
  ],
  "requires_python": "<3.10,>=3.7",
  "version": "2.16.0.dev5"
}
```
@thejcannon thejcannon deleted the jcannon/gha_upload branch September 19, 2023 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:internal CI, fixes for not-yet-released features, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pants should be distributed on Github Releases
1 participant