Automate the release guide - Take 2 - Github workflows

[pingtimeout]

This PR is the next iteration of release automation. It builds on top of #2156 and reuses the common bash libraries that were defined.

Differences from the initial PR

• Release automation can only be triggered via Github Workflows. It is not possible to perform a semi-automated release from a committer/PMC computer.
• It assumes that the following secrets are defined:
  • DOCKERHUB_USERNAME and DOCKERHUB_TOKEN - the credentials that can be used to push Docker images to Dockerhub
  • GPG_PRIVATE_KEY and GPG_PASSPHRASE - the ASCII armored private key that should be used to sign artifacts and its associated passphrase. The associated public key is assumed to be added to the KEYS file prior to being entered here.
  • APACHE_USERNAME and APACHE_PASSWORD - the credentials that can be used to connect to the ASF SVN server as well as to the ASF Nexus server.
• All code that was used to perform release steps locally has been removed to keep the PR as small as possible.

Similarities with the initial PR

This PR builds on the same assumptions from the initial PR:

• Release cannot be fully automated as of today as there are concerns that the release guide may not be comprehensive. Hence full automation is not desirable yet.
• Release can only be semi-automated given that certain operations must be manually performed by the release manager.

Remaining known-unknowns

I can see that apache/polaris has a Nightly build github workflow that publishes snapshots every night to the Apache Nexus repository. In this workflow's definition, I can find references two Nexus credentials (secrets.NEXUS_USER and secrets.NEXUS_PW). However I cannot find any such secret defined on https://github.com/apache/polaris/settings/secrets/actions, nor am I sure whether those credentials can be used to interact with the ASF SVN server. Some clarification is needed to ensure proper credentials configuration.

Example runs

I have used this PR to simulate the release of Polaris 99.98.97-incubating-rc1 on my own fork. No upload was performed, this is just to prove out that things should be working as expected. You can find links to the following workflow executions:

• This Create Release Branch workflow was used to cut the release branch with proper naming pattern: release/99.98.97-incubating. It was run with dry-run=0 so that the release branch was actually created.
• This Update version and Changelog for Release Candidate workflow was used to set the Polaris version to 99.98.97-incubating, update the changelog and push the RC1 tag. It was also run with dry-run=0 so that the modifications were actually performed (on my fork only)
• For comparison, this Update version and Changelog for Release Candidate workflow is what happens when we try to cut an RC from a commit where some Github checks have failed (e.g. CI)
• From there, all subsequent workflows were run with dry-run=1 so that no interaction with any ASF server happened.
• This Build and Publish Release Artifacts workflow was run with dry-run=1 to check the commands that would be executed, if an actual release was to be performed. We can see the binaries and Helm Charts would be built, signed, checksum'ed and publish to ASF SVN and Nexus repositories. Docker images would be built but not published.
• And finally, this Publish Release After Vote Success workflow shows how the artifacts would be moved from the dist dev space to the dist release space on ASF SVN, the final release tag would be created and the Nexus repository would be automatically released. This is also where the Github Release itself would be created.

Next steps

If this PR is approved, it should be possible to use the Create Release Branch and Update Release Candidate workflows to respectively cut the release branch and set the RC version as well as create the RC tag.

Recommendations for reviewers

• Please review the releasey/release-process-flowchart.png flowchart first. It shows how each workflow participates to the overall release process. It should substantially help make sense of the Github workflows for the rest of the review.

[snazy]

Created a PR against your branch.
It contains:

• the one suggestion
• fixes for GH actions
• proposal for the helm-version update

We have to use hash-references (see Actions Policy. And also only use action+version that are allowed by the policy, the currently approved patterns are here. Otherwise the workflows won't run.

I ran most of the commands in the workflows locally, leaving out the git pushes and svn commits and that worked, except for two things:

• Helm version update didn't work
• Helm GPG signing didn't work for me locally. Would you mind cross-checking?

Otherwise this looks really really good.
I'd appreciate if someone else could take a look at the workflows and scripts just to double-check that there are no oversights.

[snazy]

Could you cross-check this one?

It failed for me locally:

$ exec_process helm gpg sign polaris-2.3.0-incubating-SNAPSHOT.tgz 
DEBUG: Executing 'helm gpg sign polaris-2.3.0-incubating-SNAPSHOT.tgz'
Signing polaris-2.3.0-incubating-SNAPSHOT.tgz
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: */Chart.yaml: Not found in archive
tar: Exiting with failure status due to previous errors
Error: plugin "gpg" exited with error

The tarball contains these files (abbreviated list):

polaris/Chart.yaml
polaris/values.yaml
...

[pingtimeout]

I wonder if it is linked to a tar version? I do not have any warning on my end.

$ tar tzf polaris-99.98.97-incubating.tgz | head -n 5
polaris/Chart.yaml
polaris/values.yaml
polaris/templates/_helpers.tpl
polaris/templates/configmap.yaml
polaris/templates/deployment.yaml

$ helm gpg sign polaris-99.98.97-incubating.tgz
Signing polaris-99.98.97-incubating.tgz

$ echo $?
0

[snazy]

I think this could work with helm package --sign and we can get rid of the helm-gpg-plugin, which is unmaintained for 5 years now.

[snazy]

@jbonofre could you provide the details how you generated generated the gpg key? Then we can do another round of testing simulating the whole roundtrip (key -> secret -> workflow -> get artifacts -> verify) to be sure that it works.

[snazy]

Woot! Very close!
The only thing that worries me a bit is the helm gpg sign/tar thing.
Let me see if I can hack a dummy repo/workflow to see whether this works or not in GH.

[snazy]

This is really a big improvement!

The change now LGTM, so +1!

We just need to do a final safety cross-check with a dummy generated GPG key built in the same way as the "real" GPG key. But after that we should merge this PR.

Then we can test this (nearly) end-to-end and clean up the release-test artifacts. What we cannot test is the final release publication, because there is no CI for this use case but the good old try-n-error approach with some fingers-crossed.

[pingtimeout]

@snazy There were a couple of things that needed fixed, like the fact that svn is not installed by default on Github Runners. I pushed a few new commits. Here are the links to workflow runs that passed:

• Create release branch
• Update version and changelog for RC
• Build and publish release artifacts

That last one correctly created a staging repository on Nexus (see screenshot below). It also successfully pushed artifacts to dist dev. I verified that those artifacts were in SVN and deleted them. I will drop the staging repository next.

The only thing that is not tested is the last release workflow to publish artifacts after a release is successful. We will have to wait for the official 1.3.0 release to test it given its real-life impacts.

[snazy]

Nice work!

The GH WF UI is playing tricks...

Although all jobs depend on the Prerequisite ... job and those are ran in parallel, the UI gives the illusion that those are run sequentially.
Another (dummy) job that depends on the 3 parallel ones could help with that. But I don't insist on doing that in this PR.

[pingtimeout]

Run with the manual docker buildx setup: https://github.com/pingtimeout/polaris/actions/runs/19470515841, all good

[snazy]

Not much to comment anymore... +1 :)