K8s v1.22.0, VM image not available when using CAPZ

[Jont828]

/kind bug

What steps did you take and what happened:

The documentation in the CAPZ quickstart refers to K8s version 1.22.0 in the following command.

clusterctl generate cluster capi-quickstart \
  --kubernetes-version v1.22.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

However, when the cluster is built using the quick start yaml file, it produces the following error in the logs

$ kc logs capz-controller-manager-fb448b845-v6kgp -n capz-system

E0816 20:53:28.718969       1 controller.go:302] controller-runtime/manager/controller/azuremachine "msg"="Reconciler error" "error"="failed to reconcile AzureMachine: failed to create virtual machine: failed to create VM capi-quickstart-control-plane-snb5n in resource group capi-quickstart: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PlatformImageNotFound\" Message=\"The platform image 'cncf-upstream:capi:k8s-1dot22dot0-ubuntu-1804:latest' is not available. Verify that all fields in the storage profile are correct. For more details about storage profile information, please refer to https://aka.ms/storageprofile\" Target=\"imageReference\"" "name"="capi-quickstart-control-plane-snb5n" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AzureMachine"

What did you expect to happen:

Expected the tutorial commands build the VM.

Environment:

• cluster-api-provider-azure version: v0.5.1
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release): Arch Linux

[CecileRobertMichon]

@mboersma can you please take a look? It's trying to use the 18.04 image for 1.22, it should be using 20.04 (we only published 20.04 starting at 1.22).

[CecileRobertMichon]

We have a unit test for this too so not sure what's going on:

https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/master/azure/defaults_test.go#L221

[CecileRobertMichon]

I was able to repro, it's unclear why we're not running into this in #1588

[mboersma]

/assign

[mboersma]

This seems to work fine following the CAPZ make tilt-up workflow.

I wonder if there's something different when clusterctl passes through the --kubernetes-version flag, as opposed to setting the KUBERNETES_VERSION environment variable. I'll try that next.

[mboersma]

I think the problem is that the change to prefer Ubuntu 20.04 for newer Kubernetes versions merged into master but isn't part of a CAPZ release yet. See #1508.

So @Jont828 we sent you on an impossible mission here--apologies! Two ways we can fix this:

• Do a v0.5.2 CAPZ release that will include the correct logic
• Backfill the reference images for Ubuntu 18.04 in the Azure Marketplace (we switched to 20.04-only for the last two patch releases, and for 1.22.0)

Or maybe both? Let me know what you think.

[CecileRobertMichon]

@mboersma we should do both. Someone could still use 1.22 with 0.5.1 and get this issue.

[mboersma]

we should do both.

I backfilled the 18.04 images:

% az vm image list --publisher cncf-upstream --offer capi --all -o table --query "[?version=='2021.08.17']"
Publisher      Offer    Sku                          Version     Urn
-------------  -------  ---------------------------  ----------  ---------------------------------------------------------
cncf-upstream  capi     k8s-1dot19dot14-ubuntu-1804  2021.08.17  cncf-upstream:capi:k8s-1dot19dot14-ubuntu-1804:2021.08.17
cncf-upstream  capi     k8s-1dot20dot10-ubuntu-1804  2021.08.17  cncf-upstream:capi:k8s-1dot20dot10-ubuntu-1804:2021.08.17
cncf-upstream  capi     k8s-1dot21dot4-ubuntu-1804   2021.08.17  cncf-upstream:capi:k8s-1dot21dot4-ubuntu-1804:2021.08.17
cncf-upstream  capi     k8s-1dot22dot0-ubuntu-1804   2021.08.17  cncf-upstream:capi:k8s-1dot22dot0-ubuntu-1804:2021.08.17

I think we should continue to publish 18.04 as well as 20.04 versions for a few more months.

I think once we've done a CAPZ v0.5.2 release which includes #1508, we can close this.

[CecileRobertMichon]

Waiting for #1624 to merge so I can use it to release v0.5.2

[mboersma]

I'm closing this, since CAPZ v0.5.2 has been released with the code changes to prefer Ubuntu 20.04, and the 18.04 images have been backfilled.