How to handle breaking changes in apiVersion?

[pmokeev]

Type of question

• How to implement a specific feature
• General operator-related help

Question

Let's imagine, that I have v1alpha1 version of my CRD and it works fine. After some time of developing, I have to migrate to another struct of CRD and v1beta1 version, which has some new required fields, which are not present in the previous spec of apiVersion. So now, I have to support both versions of my CRD, until all users manually migrate to a new one. So, how to handle this case with two working v1alpha1 and v1beta1 versions properly?
What do I have now:

• I've read several related issues: 1, 2, 3, 4. And they generally describe ways of how to softly migrate from one version to another using ConversionWebhook and //+kubebuilder:storageversion annotation.
• I tried to implement ConvertTo(dstRaw conversion.Hub) function, which raises error in case if automatic migration to a new apiVersion is not possible, and got an error conversion webhook for ... connect: connection refused while trying to kubectl get
• I tried to implement two controllers for two apiVersion's, but eventually came to an error CRD for smth has no storage version, which tells me that there cannot be two controllers for different versions of a resource at the same time.

So, in short, my question is: How to handle case, when it's impossible to migrate to a new apiVersion due to breaking changes in spec?

Or am I doing something wrong and ideologically this shouldn't happen?

What did you do?

Migrating to a new apiVersion of CRD.

What did you expect to see?

Support both apiVersion's of CRD.

What did you see instead? Under which circumstances?

For now, I have no idea of how to add to a spec of a new apiVersion CRD new required fields.

Environment

Operator type:
/language go

Kubernetes cluster type:

"vanilla"

$ operator-sdk version
operator-sdk version: "v1.27.0", commit: "5cbdad9209332043b7c730856b6302edc8996faf", kubernetes version: "v1.25.0", go version: "go1.19.5", GOOS: "darwin", GOARCH: "arm64"

$ go version (if language is Go)
go version go1.21.6 darwin/arm64

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:38Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.10", GitCommit:"0fa26aea1d5c21516b0d96fea95a77d8d429912e", GitTreeState:"clean", BuildDate:"2024-01-17T13:38:41Z", GoVersion:"go1.20.13", Compiler:"gc", Platform:"linux/amd64"}

[pmokeev]

@camilamacedo86 @acornett21 @jberkhahn any thoughts?

[pmokeev]

@camilamacedo86 @acornett21 @jberkhahn ping again :)

[camilamacedo86]

Hi @pmokeev

If you need to address changes in the spec of your CRD which will result in breaking changes then, ideally you should create a new version of your API and covert one version to another.

See the docs in kubebuilder, it provides an tutorial: https://master.book.kubebuilder.io/multiversion-tutorial/tutorial

[pmokeev]

@camilamacedo86 thanks for reply!

That's true, but how can I handle these changes? After my experiments, I came to the conclusion that it's impossible to have two controllers for different apiVersion's at the same time. Also (for example) it is impossible to convert one version to another (as I described above). What should I do in this situation?

[acornett21]

@pmokeev Having two controllers doesn't make sense, just for API differences. It seems that your project setup is incorrect. If you look at the link @camilamacedo86 posted, it has all the code on how to to code everything, and more importantly where to code everything. IE what needs to be in the hooks code and what needs to be in the api code and what needs to be in the setupwithmanager code.

[camilamacedo86]

Hi @pmokeev

You can have a separate controller for each version, but you'll need to rename them accordingly and ensure both are initialized in the manager. However, unless there are significant changes that require encapsulating the logic in separate controllers to facilitate the removal of the old controller and CRD version in a future release, I agree with @acornett21—it likely doesn’t make sense to do this just for API differences. It may introduce more overhead than benefit, as you could encapsulate the logic in the same controller and later remove the outdated parts in a future release when they are no longer needed.

[pmokeev]

Thanks for reply!

I guess, my final question is how can encapsulate it?

At one point in time, ectd can store only one ApiVersion of CRD, right? It can be done by providing //+kubebuilder:storageversion annotation.

As I said above, for some reason, I cannot use ConversionHook due to breaking changes in CRD. Also, I cannot use multiple ApiVersion's in For statement inside SetupWithManager function.

So, how can I do this?

you could encapsulate the logic in the same controller and later remove the outdated parts in a future release when they are no longer needed.

Maybe I'm missing something in the docs...

[camilamacedo86]

Hi @pmokeev,

You're correct that etcd will only store one version of the CRD, which is determined by the //+kubebuilder:storageversion annotation. Even though etcd stores only one version, the Kubernetes API server can serve different API versions by converting between them using a hub-and-spoke model. You can learn more about this in the Kubebuilder multi-version tutorial:

• Hub and Spoke Conversion Overview
• Hub and Spoke Concepts

Once you've set up conversion using the hub-and-spoke model, the Kubernetes API server will automatically ensure that clients receive the correct version they requested. When the API server needs to serve a request for a resource in a different version (a "spoke"), it converts from the hub version to the requested version. Therefore, your controller will manage the hub version, and the API server handles the conversion between the hub and spoke versions.

Handling multiple versions within the same controller is only useful if you're not using conversion webhooks. In that case, you could check the version inside the controller and perform specific actions based on the version. Alternatively, you could also create a separate controller for each version if needed. When using conversion webhooks, there's generally no need to handle multiple versions in the controller directly since the conversion is managed automatically by Kubernetes. if you're using the hub-and-spoke model, your controller will typically manage the new version of the CRD. This simplifies the logic within your controller, as you don’t need to handle multiple versions directly.

All the relevant information is described in the tutorials linked above. If you encounter any edge cases or have any specific needs that require handling things differently, feel free to let us know, and we can try to help you work through the details.

[camilamacedo86]

HI @pmokeev and @acornett21

I think we could close this issue , right?
Has anything else that we can do here to help out?

[pmokeev]

Yes, I guess so