Release Automation: Add cloud build target for release builds

[roberthbailey]

Is your feature request related to a problem? Please describe.
We currently do release builds on a local machine. This means that the releaser must configure their local environment very specifically to ensure that the builds are successful.

Moving the builds to cloud build will significantly reduce the amount of work necessary to create a build and reduce a number of steps that are prone to human error.

We can also move the release note generation into the cloud build target as part of the release process.

The step to check that all example images are properly published can be part of the release target as well.

The release build needs to be installed and smoke tested. We should determine whether we run the full e2e suite (which can at times be flaky) or a subset of it or maybe even a single test.

Finally, part of the make do-release target pushes a new github branch. This may be tricky from cloud build, so we could move to using github actions to create a new branch each time we create a release tag.

Right now there are a number of steps in the release process related to building the release locally:

• Run make shell and run gcloud config configurations activate agones-images.
• git checkout main && git pull --rebase upstream main
• Run make gen-changelog to generate the CHANGELOG.md (if release candidate
  make gen-changelog RELEASE_VERSION={version}-rc). You will need your
  Github Personal Access Token for this.
• Run make test-examples-on-gcr to ensure all example images exist on gcr.io/agones-images-
• Confirm local git remote upstream points at git@github.com:googleforgames/agones.git
• Run git remote update && git checkout main && git reset --hard upstream/main to ensure your code is in line
  with upstream (unless this is a hotfix, then do the same, but for the release branch)
• Run make do-release. (if release candidate: make do-release RELEASE_VERSION={version}-rc) to create and push the docker images and helm chart.
• Do a helm repo add agones https://agones.dev/chart/stable / helm repo update and verify that the new
  version is available via the command helm search repo agones --versions --devel.
• Do a helm install --namespace=agones-system agones agones/agones
  (helm install --namespace=agones-system agones agones/agones --version={version}-rc if release candidate) and a smoke test to confirm everything is working.

and two relate pre-requisites:

• Have a gcloud config configurations configuration called agones-images that points to the same project.
• Have a Github Personal Access Token with repo permissions.

Describe the solution you'd like
Move the manual release builds into cloud build.

[roberthbailey]

We may be able to take some inspiration from the Quilkin release process, since they do release builds using cloud build.

[markmandel]

As a thought - make gen-changelog might be the easiest place to start.

You can see an example of us doing this in the Quilkin project:

https://github.com/googleforgames/quilkin/blob/f7745c10c158e78eabc50b1953bb63b5fce3c8bc/build/release/cloudbuild.yaml#L44-L53

[markmandel]

Reopen, since not complete.

[markmandel]

@Kalaiselvi84 last time you ran a release, did you have to wait an hour before seeing the release via helm search ?

[Kalaiselvi84]

If I remember correctly, I didn't wait for an hour. The post-release was completed in less than 30 minutes.

[markmandel]

Wondering if you could add the

Do a helm install --namespace=agones-system agones agones/agones
(helm install --namespace=agones-system agones agones/agones --version={version}-rc if release candidate) and a smoke test to confirm everything is working.

🤔 not sure how easy it is to do this, since the only place you could run it is on the e2e cluster -- which we could do.

What are your thoughts? Is it easier to do the test locally / on a dev cluster?

[Kalaiselvi84]

I will ensure the following checks are completed before proceeding with the "helm repo update" task,

1. Check if older version of Agones installed - helm ls -n agones-system
2. If so uninstall agones from agones-system - helm uninstall -n agones-system agones
3. If the cluster is not authenticated: make gcloud-auth-cluster GCP_CLUSTER_ZONE=us-west1-c
4. Retrieve the pods running info in the agones-system - kubectl get pods -n agones-system
5. Delete a namespace agones-system from the cluster - kubectl delete ns agones-system

After helm repo update, helm search repo agones --versions --devel and helm install --create-namespace --namespace=agones-system agones agones/agones:

1. Retrieve information about the pods running in the agones-system - kubectl get pods --namespace agones-system
2. Smoke Test to test anyone of the existing test from test/e2e - example: TestCreateFleetGameServerAllocate(t *testing.T) -> If the test is not succeeded with error failed to cleanup e2e namespace -> kubectl get gs

This will take few minutes but not an hour👍🏻

[markmandel]

3. If the cluster is not authenticated: make gcloud-auth-cluster GCP_CLUSTER_ZONE=us-west1-c

The only thing here that might be tricky, is that you will need to authenticate against one of the e2e cluster (since I can't think of a better place to run these tests during the release on the Agones project) - as the release process is running in the Agones project, not your own, and there isn't a developer test cluster in that project.

@gongmax do you have any thoughts here?

[Kalaiselvi84]

So far, I haven't utilized make gcloud-auth-cluster GCP_CLUSTER_ZONE=us-west1-c command in any of the previous releases. I've taken note that all these steps are from the release recording where you have had authentication issue.

[markmandel]

• Do a helm install --namespace=agones-system agones agones/agones
  (helm install --namespace=agones-system agones agones/agones --version={version}-rc if release candidate) and a smoke test to confirm everything is working.

So I think for this ticket we skip automating this step -- otherwise we might have to provision test infra etc, and that seems like too much of a investment that could be solved through installing on a local dev cluster.

TL;DR - close the ticket at the level of completion we have now.

@Kalaiselvi84 @roberthbailey @zmerlynn WDYT?

[Kalaiselvi84]

 Do a helm install --namespace=agones-system agones agones/agones or helm install --create-namespace --namespace=agones-system agones agones/agones if the namespace was deleted and a smoke test to confirm everything is working.

Based on my understanding, it would be suitable to update this step manually. We need to ensure that the namespace is created or reinstalled if it was deleted, along with conducting a smoke test. However, I am open to any suggestions...

[markmandel]

Yeah, I agree. In which case, I'm making the decision to close this issue as complete!