Move release automation to GHA

[ekcasey]

Now that github actions support manually triggered workflows we should move the lifecycle release process from concourse to GHA

https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/

[natalieparellano]

As part of this work, we should change the way we're publishing lifecycle images. Instead of building a new image when releases are cut, we should build a new remote image with every push to main and re-tag the lifecycle image on release e.g., if the release SHA is 101f194 then the published lifecycle:101f194 would be re-tagged as lifecycle:0.X.0.

[natalieparellano]

@yaelharel and I are thinking of picking this up

[natalieparellano]

After some experimentation and discussion we think this is the way to proceed:

• pack already has automation around the release process with GHA. We want to copy most of what is being done, namely:
  ✅ Derive the release version from the branch name -> if branch is release/0.12.0, the release version will be 0.12.0
  ✅ We should therefore remove the hard coded VERSION file and the hard coded version in lifecycle.toml. We'll need to template lifecycle.toml and fill in the right version in the uploaded artifact.
  • Auto generated changelog. We'll create a separate issue for this.
    ✅ However with pack a new draft release is created every time there are new commits on the branch. We want to avoid this. We should look into manually triggered workflows: https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/

Other needed things:
✅ We'll need to ensure that tests are run against PRs to release/*
✅ release/* should probably be protected branches (we're pretty sure this is already the case)

• The workflow that creates the draft release should also build a new lifecycle image
• The existing lifecycle-image.yml workflow should be changed to re-tag an existing image with the release version
• Add logic to appropriately describe the release in the release notes as minor / patch
• See about remove v from the version in dev releases, it's confusing when smushed together with the tag
• Confirm that using the git tag is okay (this is what we currently use for dev lifecycle images, whatever we choose it should be consistent)

[ekcasey]

• Add logic to appropriately describe the release in the release notes as minor / patch

We could also just scrap this from the release notes text. Folks can already see from the version number whether it is a release or a patch.

[yaelharel]

In order to protect the release/* branches, an owner or an admin should follow the steps in this link.
@ekcasey, I guess that you'll be able to do so before we close this issue.

[ekcasey]

hey @yaelharel, I am pretty sure the release branches are already protected.

[natalieparellano]

Some context: right now pack determines the lifecycle version from reading the lifecycle.toml. This version currently does NOT have any additional metadata appended to it. Thus when pack looks for a lifecycle image with matching version, it does not expect metadata.

[natalieparellano]

Possible a/c:

local development w/ LIFECYCLE_VERSION set:

• LIFECYCLE_VERSION=0.99.0 make build package build-image-linux -> the tarball in the out directory should be named lifecycle-v0.99.0+linux.x86-64.tgz (same for windows)
• cd out/ && tar xzvf lifecycle-v0.99.0+linux.x86-64.tgz && cat lifecycle.toml -> the lifecycle version should be 0.99.0 (same for windows)
• docker run --rm -v $PWD/lifecycle:/lifecycle ubuntu:bionic /lifecycle/lifecycle -version -> should be 0.99.0+7b88d91 (where 7b88d91 is the git sha)
• docker run --rm lifecycle:0.99.0 /cnb/lifecycle/lifecycle -version -> should also be 0.99.0+7b88d91

local development w/o LIFECYCLE_VERSION set:

• make build package build-image-linux -> the tarball in the out directory should be named lifecycle-v0.9.1-38+7b88d91+linux.x86-64.tgz
• cd out/ && tar xzvf lifecycle-v0.9.1-38+7b88d91+linux.x86-64.tgz && cat lifecycle.toml -> the lifecycle version should be 0.9.1-38+7b88d91 (same for windows)
• docker run --rm -v $PWD/lifecycle:/lifecycle ubuntu:bionic /lifecycle/lifecycle -version -> should be 0.9.1-38+7b88d91 (where 7b88d91 is the git sha) - note it is NOT 0.9.1-38+7b88d91+7b88d91 (requires change to cmd/version.go)
• docker run --rm lifecycle:7b88d91 /cnb/lifecycle/lifecycle -version -> should also be 0.9.1-38+7b88d91

new commits to main:

• lifecycle-linux-x86-64 and lifecycle-windows-x86-64 artifacts are created
• Downloading and unzipping the artifact:
  • There should be a tgz with name lifecycle-v0.9.1-38+7b88d91+linux.x86-64.tgz
  • Extracting the tgz, cat lifecycle.toml -> the lifecycle version should be 0.9.1-38+7b88d91
  • docker run --rm -v $PWD/lifecycle:/lifecycle ubuntu:bionic /lifecycle/lifecycle -version -> should be 0.9.1-38+7b88d91 (where 7b88d91 is the git sha) - note it is NOT 0.9.1-38+7b88d91+7b88d91 (requires change to cmd/version.go)
• lifecycle image is published in Docker Hub with version: 7b88d91
  • There are actually two images and a manifest list: buildpacksio/lifecycle:7b88d91-linux, buildpacksio/lifecycle:7b88d91-windows, buildpacksio/lifecycle:7b88d91 (see: https://hub.docker.com/r/buildpacksio/lifecycle/tags)
  • docker run --rm buildpacksio/lifecycle:7b88d91 /cnb/lifecycle/lifecycle -version -> should be 0.9.1-38+7b88d91 (note NO -rc)

PRs to main and release/** branches:

Same as new commits to main, except no lifecycle image is created. Note that PRs to release branches will still have a "dev" version like 0.9.1-38+7b88d91 unless the FROM branch is also named release/*

new commits to release/** branches:

• Assume branch name is release/0.99.0
• lifecycle-linux-x86-64 and lifecycle-windows-x86-64 artifacts are created
• Downloading and unzipping the artifact:
  • There should be a tgz with name lifecycle-v0.99.0+linux.x86-64.tgz
  • Extracting the tgz, cat lifecycle.toml -> the lifecycle version should be 0.99.0
  • docker run --rm -v $PWD/lifecycle:/lifecycle ubuntu:bionic /lifecycle/lifecycle -version -> should be 0.99.0+7b88d91 (where 7b88d91 is the git sha)
• lifecycle image is published in Docker Hub with version: 7b88d91
  • There are actually two images and a manifest list: buildpacksio/lifecycle:7b88d91-linux, buildpacksio/lifecycle:7b88d91-windows, buildpacksio/lifecycle:7b88d91 (see: https://hub.docker.com/r/buildpacksio/lifecycle/tags)
  • docker run --rm buildpacksio/lifecycle:7b88d91 /cnb/lifecycle/lifecycle -version -> should be 0.99.0+7b88d91 (note NO -rc)

release candidate (push the button in GitHub):

• Assume branch name is release/0.99.0 and the current commit is 7b88d91
• No new artifacts are created
• A draft release is created in GitHub and the artifacts uploaded to the release are the lifecycle-linux-x86-64 and lifecycle-windows-x86-64 that were created when 7b88d91 was merged into the release branch (except they are appropriately named tarballs i.e. lifecycle-v0.99.0+linux.x86-64.tgz)
• Note that because the version in lifecycle.toml and the lifecycle image tag are different, pack will fail in untrusted builder workflows without further changes (we should create an issue for this and for improving the local testing workflow more generally)

release candidate failure case (push the button in GitHub when build of the last commit is still in progress or failed):

• We should see a failure in the determine download url step

release is published:

• Assume release commit sha is 7b88d91 (we publish the draft release that was created in the previous step)
• buildpacksio/lifecycle:7b88d91-linux is re-tagged as buildpacksio/lifecycle:0.99.0-linux
• buildpacksio/lifecycle:7b88d91-windows is re-tagged as buildpacksio/lifecycle:0.99.0-windows
• buildpacksio/lifecycle:7b88d91 is re-tagged as buildpacksio/lifecycle:0.99.0

[ekcasey]

Love this overall. Just a few questions:

1. I like retagging to release the lifecycle images rather than building them after we release the lifecycle, but why wait until we create a draft release rather than building one for every commit to main or release?
2. Should we just do these steps (from release candidate) For all commits to a release/** branch?

• Assume branch name is release/0.99.0
• lifecycle-linux-x86-64 and lifecycle-windows-x86-64 artifacts are created
• Downloading and unzipping the artifact:
  • cat lifecycle.toml -> the lifecycle version should be 0.99.0
  • docker run --rm -v $PWD/lifecycle:/lifecycle ubuntu:bionic /lifecycle/lifecycle -version -> should be >0.99.0+7b88d91 (where 7b88d91 is the git sha)
• lifecycle image is published in Docker Hub with version: 0.99.0-rc+7b88d91
  • There are actually two images and a manifest list: buildpacksio/lifecycle:0.99.0-rc+7b88d91-linux, >buildpacksio/lifecycle:0.99.0-rc+7b88d91-windows, buildpacksio/lifecycle:0.99.0-rc+7b88d91 (see: https://hub.docker.com/r/buildpacksio/lifecycle/tags)
  • docker run --rm buildpacksio/lifecycle:0.99.0-rc+7b88d91 /cnb/lifecycle/lifecycle -version -> should be 0.99.0+7b88d91 (note NO -rc)

Then we can publish to a draft without rebuilding and validate the candidate images before publishing the draft. Essentially, why not treat every commit to a release branch like a release candidate?

[natalieparellano]

@ekcasey sounds good. I'll update the previous comment. We want to hold off on publishing lifecycle images on new commits to main, but otherwise we are in agreement! (main poses additional challenges determining what the right version should be... but let us know if you want us to pursue it further...)

Edit: it ended up being just as easy to build lifecycle images for new commits to main, so we are doing that as well.

[ekcasey]

Implemented in #415