[kots]: allow deployment whilst installer job running

[mrsimonemms]

Description

In this PR, install means "install or upgrade" unless otherwise stated

Allow deployment whilst an existing Installer job running. This should allow for recovery in the following scenarios:

1. Deployment failed/stopped mid-install and user wants to redeploy same version
2. Deployment failed/stopped mid-install and user wants to deploy a different version
3. User decides to trigger a new deployment whilst an existing deployment is in progress

This appends the cursor to the job name. This is necessary to allow us to change the image on the container, effectively replacing the existing job. The first part of this script then removes any already running jobs/pods, thus clearing up the state.

It then rolls back (or uninstalls if it's the first deployment) before starting the next helm upgrade. For a user, this should then allow them to unstick their issues.

Setting of the ttlSecondsAfterFinished to 0 means that the "redeploy" button works as soon as the job as finished

How to test

Get your Helm deployment into either a pending-install or pending-upgrade state. The script does this anyway for an upgrade, but to do this manually, suggest starting an install job then running:

kubectl delete jobs.batch -n gitpod -l component=gitpod-installer --force --grace-period 0
kubectl delete pod -n gitpod -l component=gitpod-installer --force --grace-period 0

The reason it doesn't work for a redeployment is because Kubernetes sees the request to deploy a job called installer-1234 and, as that's already deployed, it thinks "oooh, nothing to do". There seems to be no way of generating a random value at deploy-time with KOTS - have asked as a question in our channel

Ensure by running helm ls -aA.

Now, run the following - all of these should result in a successful deployment, and will need you to get it to enter the same "pending" state as above:

1. Redeploy the existing deployment.
2. Publish a new version to KOTS and deploy that

For regression, we should also test:

1. A fresh installation (run helm un -n gitpod gitpod)
2. An upgrade that's not in a "pending" state
3. A redeployment that's not in a "pending" state

Release Notes

[kots]: allow deployment whilst installer job running

Documentation

[mrsimonemms]

/werft run no-preview publish-to-kots with-clean-deployment

👍 started the job as gitpod-build-sje-kots-install-cancel.2

[mrsimonemms]

/werft run no-preview publish-to-kots

👍 started the job as gitpod-build-sje-kots-install-cancel.6

[mrsimonemms]

/werft run no-preview publish-to-kots with-clean-deployment

👍 started the job as gitpod-build-sje-kots-install-cancel.7

[mrsimonemms]

/werft run no-preview publish-to-kots

👍 started the job as gitpod-build-sje-kots-install-cancel.9

[mrsimonemms]

/werft run no-preview publish-to-kots

👍 started the job as gitpod-build-sje-kots-install-cancel.13
(with .werft/ from main)

[nandajavarma]

Maybe a --grace-period=0 could be added too, just to be sure?

[mrsimonemms]

Good call

[nandajavarma]

Tested and works like a charm! LGTM! Merging on Friday early evening is bad luck. Will approve on Monday 🤓

[mrsimonemms]

Merging on Friday early evening is bad luck

You are very wise

[corneliusludmann]

Since @nandajavarma is OOO today, I'm happy to add my approval. I am very curious to see how well this works in practice.