waypoint server upgrade on kubernetes does not upgrade if latest tag image is cached.

[izaaklauer]

Describe the bug
If you don't explicitly set a k8s-server-image, running waypoint server upgrade will default to using hashicorp/waypoint:latest. If that's already cached on the node waypoint is running on, the new deployment will use the cached image instead of re-pulling to get the actual latest version.

Steps to Reproduce

• Install waypoint on a single-node kubernetes cluster, without specifying a k8s-server-image
• Wait for the waypoint team to publish a new version
• Run waypoint server upgrade -auto-approve, then check the version of the waypoint server. It will be the previous version.

Expected behavior
Running waypoint server upgrade when a new version is available should always result in an upgrade to the latest version.

Options to fix this:

• Waypoint should detect that a newer version is available, and use that instead of the latest tag.
• We could default to an image-pull-policy of always

Waypoint Platform Versions
Additional version and platform information to help triage the issue if
applicable:

• Waypoint CLI Version: 0.4.1
• Waypoint Server Platform and Version: 0.4.0/0.4.1
• Waypoint Plugin: kubernetes, but this may also be a problem for docker/ecs/nomad.

[briancain]

I think we should enhance the server upgrade command to grab the pull policy for the server pod. If it's not latest and the pull policy isn't always, print a warning that the upgrade won't happen immediately until the pods image is pulled. Then the warning should mention to re-run the command and specify the pull policy or if there's a one-off force latest pull we could include that as a flag and mention to use that in the warning message.

[krantzinator]

Since the default for imagePullPolicy is unset, it falls back to kubernetes defaults. In this case, a policy of Always since the image installed has a tag of latest.

$ waypoint install -platform=kubernetes --accept-tos
...
Waypoint server successfully installed and configured!
...

$ kubectl get pod/waypoint-server-0 -o yaml | grep imagePullPolicy
    imagePullPolicy: Always

Rather than automate a restart of the pod, since that won't be a required action in all upgrade paths, I'm going to add a warning message to the terminal output to ensure the user knows a restart may be required.

[briancain]

Rather than automate a restart of the pod, since that won't be a required action in all upgrade paths, I'm going to add a warning message to the terminal output to ensure the user knows a restart may be required.

👍🏻 , one thing I forgot about when I wrote my comment earlier in this issue is our upgrader takes into account which upgrade strategy is configured for the statefulset that waypoint-server is deployed to. So we shouldn't auto-delete the pod if it's RollingUpgrade. And we already delete the pod if the strategy is OnDelete. I think enhancing our warning messages to make this clearer based on how K8s behaves makes sense to me!