CRD fails to apply on upgrade from legacy v1beta1 created versions - preserveUnknownFields must be false in order to use defaults

[plnordquist]

Expected Behavior

No validation error when upgrading from to v3.23.1 from v3.22.1 when my CRDs were created from v1beta1 crds.

Current Behavior

The CustomResourceDefinition "felixconfigurations.crd.projectcalico.org" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
The CustomResourceDefinition "ipamblocks.crd.projectcalico.org" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

Possible Solution

I ended up digging into this issue and I think I found the root cause. The spec.preserveUnknownFields value is set to true if the CRDs are initialized from v1beta1 CRD definitions. The last version I found that has this CRD definition is Calico v3.14.2. When I attempt to upgrade to any version of Calico that has defaults in the OpenAPI spec in the CRD like v3.22.2 or v3.23.1, the api server refuses to upgrade the crd with the errors above. If I upgrade to v3.21.5 I don't see this error. This could be solved again like #4237 by removing the defaults from the CRDs. One proper solution could be to tweak this flag manually to false since as per the deprecation guide at https://kubernetes.io/docs/reference/using-api/deprecation-guide/, this flag is not allowed to be true when creating a v1 CRD and must be specified in the schema at appropriate places where there could be unknown fields.

Steps to Reproduce (for bugs)

1. Setup minikube with Kubernetes v1.21.13
2. Deploy Calico v3.14.2
3. Attempt to upgrade to v3.23.1

Context

Upgrading calico from v3.22.1 to v3.23.1, this would likely happen in v3.22.2 as well due to the defaulted floatingIP option in the Felix Config CRD.

Your Environment

• Calico version v3.22.1 to v3.23.1
• Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes v1.21.13 (bootstrapped with kubeadm)
• Operating System and version: CentOS 7.9

[plnordquist]

I think there's another alternative to use kubectl replace on the crds instead of kubectl apply based on a similar cert-manager issue cert-manager/cert-manager#4197. I don't know how safe it would be to run kubectl replace on the whole calico manifest so I guess I'd need to extract the CRDs and probably backup all of the Calico resources.

[caseydavenport]

I ended up digging into this issue and I think I found the root cause. The spec.preserveUnknownFields value is set to true if the CRDs are initialized from v1beta1 CRD definitions

Yep, definitely correct. @freecaykes is actually working on this patch at the moment to specifically address this by explicitly setting the flag to "false": #6242

In the meantime, it should be 100% safe to use kubectl replace on the CRDs. Using kubectl replace on the entire manifest is a little trickier, depending on if you have made manual modifications to the other resources installed via that manifest that you don't want to lose, but I will say that kubectl replace -f calico.yaml is a regular part of my workflow without issue.