spec.preserveUnknownFields missing while upgrading seldon-core-operator to 1.9.1 from 1.2.2 using helm chart

[TaylorHere]

Describe the bug

upgrade seldon-core-operator to 1.9.1 from 1.2.2 using helm chart

To reproduce

Error: UPGRADE FAILED: cannot patch "seldondeployments.machinelearning.seldon.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

Expected behaviour

CRD missing spec.preserveUnknownFields, append it, everything works

[ukclivecox]

Which version of Kubernetes are running on?

[TaylorHere]

sorry for missing this information, we are running version 1.18.8

[axsaucedo]

The upgrade from pre-Kubernetes 1.18 is inherently fiddly due to the more strict validation that this version introduced - similarly the CRD for post-kubernetes 1.18 is actually differnt - you can actually see that in version 1.3.0 we introduced new CRD changes in the helm chart via an IF statement to install the CRD depending on the k8s version. Due to this, the path to upgrade from pre-18 is similar to what you have done, namely:

1. Start with kubernetes cluster pre 1.18 with seldon core pre-1.3.0
2. Upgrade Kubernetes cluster to post 1.18 (seldon core CRD is now "invalid" but still installed as still in etcd)
3. Manually add spec.preserveUnknownFields, to helm chart and install CRD (so it ignores invalid fields of now invalid CRD)
4. Remove the spec.preserveUnknownFields, from helm chart manually again, and re-install now the current CRD

This is something that we should probably add to the UPGRADING.md page, I'll update it there. Let us know if there's still any questions, otherwise I can close the issue.

[axsaucedo]

Ok added #3444