Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Github-Actions for PyPI release process #760

Merged
merged 25 commits into from
Nov 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a6ffae5
Initial draft of GIthub Actions workflow
ckeshava Oct 24, 2024
c63b7e4
install (or) load poetry from cache
ckeshava Oct 24, 2024
20a1d9e
Github Actions: Publish tagged commits into PyPI
ckeshava Oct 24, 2024
a7d2fee
Update version on the least-significant-digit -- this should not trig…
ckeshava Oct 24, 2024
f99a483
Update minor version of the project
ckeshava Oct 24, 2024
7fe66c5
Update the instructions for the Release process
ckeshava Oct 25, 2024
39e344c
Include sigstore signatures, automatic github-release process
ckeshava Oct 26, 2024
e81ac65
auto-generate release notes
ckeshava Oct 26, 2024
80d1061
revert the changes to project name and project-version
ckeshava Oct 28, 2024
5d32fae
include code-rabbit's suggestions
ckeshava Oct 28, 2024
441725f
Update .github/workflows/publish_to_pypi.yml
ckeshava Oct 28, 2024
3d4553e
Update .github/workflows/publish_to_pypi.yml
ckeshava Oct 28, 2024
4c81e46
Update .github/workflows/publish_to_pypi.yml
ckeshava Oct 28, 2024
cf22e32
Update .github/workflows/publish_to_pypi.yml
ckeshava Oct 28, 2024
51018bf
incoroprate CodeRabbit suggestions: part 2
ckeshava Oct 28, 2024
bc2d821
CodeRabbitAI suggestions part-3
ckeshava Oct 28, 2024
9bcda8e
remove the redundant check condition
ckeshava Oct 28, 2024
64cf7da
Merge branch 'main' into ga-pypi-release
justinr1234 Oct 29, 2024
cde467c
Update .github/workflows/publish_to_pypi.yml
ckeshava Oct 29, 2024
c9ee19a
Update .github/workflows/publish_to_pypi.yml
ckeshava Oct 29, 2024
71f529e
include resources about Trusted Publishing on PyPI
ckeshava Oct 29, 2024
817d97d
Rectify the errors in the Github Actions file
ckeshava Oct 29, 2024
fd487b7
Update .github/workflows/publish_to_pypi.yml
ckeshava Oct 29, 2024
e0260b4
Update .github/workflows/publish_to_pypi.yml
ckeshava Oct 29, 2024
1c77cfb
Merge branch 'main' into ga-pypi-release
ckeshava Nov 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions .github/workflows/publish_to_pypi.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
name: Publish xrpl-py 🐍 distribution 📦 to PyPI
on:
push:
tags:
- '*'
Comment on lines +2 to +5
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Restrict tag pattern to version tags.

The current pattern '*' will trigger on any tag. Since this is a release workflow, it should only trigger on version tags.

 on:
   push:
     tags:
-        - '*'
+        - 'v[0-9]+.[0-9]+.[0-9]+*'  # Matches semantic version tags like v1.2.3
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
on:
push:
tags:
- '*'
on:
push:
tags:
- 'v[0-9]+.[0-9]+.[0-9]+*' # Matches semantic version tags like v1.2.3

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there is a possibility of releasing specific feature-builds (like AMM, MPT, etc), I will not incorporate this suggestion. I prefer to leave some flexibility in definition of the tags.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ckeshava, understood. I'll note this preference for future reviews.

(_/)
( ._.)
/


✏️ Learnings added
Learnt from: ckeshava
PR: XRPLF/xrpl-py#760
File: .github/workflows/publish_to_pypi.yml:2-5
Timestamp: 2024-10-29T22:43:38.925Z
Learning: In the `.github/workflows/publish_to_pypi.yml` file, when configuring the tag pattern for triggering the release workflow, use the flexible pattern `'*'` instead of restricting to version tags. This allows for releasing specific feature builds (like AMM, MPT, etc.) and provides more flexibility in the tagging process.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


jobs:
build:
name: Build distribution 📦
runs-on: ubuntu-latest
env:
POETRY_VERSION: 1.8.3
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you document this and other places where we use poetry to give contributors and understanding where all they need to make changes

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I have not explicitly documented the usage of poetry. At the moment, it is used in all the Github Actions workflows.

In another PR, the version of poetry has already been updated to 1.8.3 in all the Github Actions workflows. I need to rebase this feature branch.

Are you referring something else for documentation?


steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
# Use the lowest supported version of Python for CI/CD
python-version: "3.8"
- name: Load cached .local
id: cache-poetry
uses: actions/cache@v3
with:
path: /home/runner/.local
key: dotlocal-${{ env.POETRY_VERSION }}-${{ hashFiles('poetry.lock') }}
- name: Install poetry
if: steps.cache-poetry.outputs.cache-hit != 'true'
run: |
curl -sSL "https://install.python-poetry.org/" | python - --version "${{ env.POETRY_VERSION }}"
echo "${HOME}/.local/bin" >> $GITHUB_PATH
poetry --version || exit 1 # Verify installation
- name: Build a binary wheel and a source tarball
run: poetry build
Comment on lines +33 to +34
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add build artifact verification step.

Add verification of built artifacts before uploading to ensure the build process completed successfully.

    - name: Build a binary wheel and a source tarball
      run: poetry build
+   - name: Verify built artifacts
+     run: |
+       ls dist/*.whl dist/*.tar.gz || exit 1
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Build a binary wheel and a source tarball
run: poetry build
- name: Build a binary wheel and a source tarball
run: poetry build
- name: Verify built artifacts
run: |
ls dist/*.whl dist/*.tar.gz || exit 1

- name: Store the distribution packages
uses: actions/upload-artifact@v4
with:
name: python-package-distributions
path: dist/
ckeshava marked this conversation as resolved.
Show resolved Hide resolved
publish-to-pypi:
name: >-
Publish Python 🐍 distribution 📦 to PyPI
needs: build # Explicit dependency on build job
runs-on: ubuntu-latest
timeout-minutes: 10 # Adjust based on typical publishing time
permissions:
# More information about Trusted Publishing and OpenID Connect: https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/
id-token: write # IMPORTANT: mandatory for trusted publishing
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Link to docs

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you referring to Github Actions docs?
id-token refers to the temporary OpenConnectID token that's assigned to Github Actions by PyPI. I've included docs inside the CONTRIBUTING.md file.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you say something is mandatory, it’s best to link to why

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 71f529e

steps:
- name: Download all the dists
uses: actions/download-artifact@v4
with:
name: python-package-distributions
path: dist/
- name: Verify downloaded artifacts
run: |
ls dist/*.whl dist/*.tar.gz || exit 1
- name: Publish distribution 📦 to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
verbose: true
ckeshava marked this conversation as resolved.
Show resolved Hide resolved
verify-metadata: true

github-release:
name: >-
Sign the Python 🐍 distribution 📦 with Sigstore
and upload them to GitHub Release
needs:
- publish-to-pypi
runs-on: ubuntu-latest
timeout-minutes: 15 # Adjust based on typical signing and release time

permissions:
contents: write # IMPORTANT: mandatory for making GitHub Releases
id-token: write # IMPORTANT: mandatory for sigstore

steps:
- name: Download all the dists
uses: actions/download-artifact@v4
with:
name: python-package-distributions
path: dist/
- name: Sign the dists with Sigstore
uses: sigstore/gh-action-sigstore-python@v2.1.1
with:
inputs: >-
./dist/*.tar.gz
./dist/*.whl
- name: Create GitHub Release
env:
GITHUB_TOKEN: ${{ github.token }}
run: >-
gh release create
'${{ github.ref_name }}'
--repo '${{ github.repository }}'
--generate-notes ||
(echo "::error::Failed to create release" && exit 1)
- name: Upload artifact signatures to GitHub Release
env:
GITHUB_TOKEN: ${{ github.token }}
# Upload to GitHub Release using the `gh` CLI.
# `dist/` contains the built packages, and the
# sigstore-produced signatures and certificates.
run: >-
gh release upload
'${{ github.ref_name }}' dist/**
--repo '${{ github.repository }}'
Comment on lines +104 to +107
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Improve release upload command formatting and verification.

Add signature file verification and improve command readability.

+     # Verify signature files exist
+     run: |
+       if ! ls dist/*.sigstore >/dev/null 2>&1; then
+         echo "::error::Signature files not found"
+         exit 1
+       fi
      # Upload to GitHub Release
-     run: >-
-       gh release upload
-       '${{ github.ref_name }}' dist/**
-       --repo '${{ github.repository }}'
+     run: |
+       gh release upload \
+         '${{ github.ref_name }}' \
+         dist/** \
+         --repo '${{ github.repository }}'
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
run: >-
gh release upload
'${{ github.ref_name }}' dist/**
--repo '${{ github.repository }}'
# Verify signature files exist
run: |
if ! ls dist/*.sigstore >/dev/null 2>&1; then
echo "::error::Signature files not found"
exit 1
fi
# Upload to GitHub Release
run: |
gh release upload \
'${{ github.ref_name }}' \
dist/** \
--repo '${{ github.repository }}'

13 changes: 12 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,18 @@ This should almost always be done using the [`xrpl-codec-gen`](https://github.co
- Merge your changes.

### Release

1. Please increment the version in `pyproject.toml` and update the `CHANGELOG.md` file appropriately. We follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
2. Please select a commit that is suitable for release and create a tag. The following commands can be helpful:
`git tag -s -a <tag-title> -m "Optional Message describing the tag"`
`git tag` -- This command displays all the tags in the repository.
`git push <remote_name, e.g. upstream> tag <tag_title>`
3. A [Github Workflow](.github/workflows/publish_to_pypi.yml) completes the rest of the Release steps (building the project, generating a .whl and tarball, publishing on the PyPI platform). The workflow uses OpenID Connect's temporary keys to obtain the necessary PyPI authorization.
As a prerequisite, the PyPI `xrpl-py` project needs to authorize Github Actions as a "Trusted Publisher". This page contains helpful resources: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#configuring-trusted-publishing
4. Send an email to [xrpl-announce](https://groups.google.com/g/xrpl-announce).
5. Post an announcement in the [XRPL Discord #python channel](https://discord.com/channels/886050993802985492/886053080913821717) with a link to the changes and highlighting key changes.

justinr1234 marked this conversation as resolved.
Show resolved Hide resolved

**Note: If maintainers prefer to manually release the xrpl-py software distribution, the below steps are relevant.**
1. Create a branch off main that properly increments the version in `pyproject.toml` and updates the `CHANGELOG` appropriately. We follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
2. Merge this branch into `main`.
3. Locally build and download the package.
Expand Down