Making client-gen work with layout V4

[shaneutt]

What do you want to happen?

There are many projects out there which use kubebuilder to manage their controllers and APIs while also using controller-gen's client-gen command to generate a Golang client for those APIs.

Historically this has "just worked" but with layout v4 things have been breaking:

kubernetes/code-generator#167

The purpose of this request is to get this back into a working state and make support for this even more official so that we can coordinate and avoid breakages in the future.

/cc @camilamacedo86 who requested I put this issue in

Extra Labels

No response

[camilamacedo86]

Hi @shaneutt,

Since you have been using the code-generator with Kubebuilder could you please help us by providing the following information:

1 - What changes were required for you to integrate the Kubebuilder project with the code generator? If you update the makefile target, could you please provide a snippet of the code?
2 - Is it possible to use code-generator and controller tools at the same time? If so, why / when that would be useful?
3- If it is not possible to use both together, could you please let us know when it would be useful to use code-generator instead of controller-gen? What are the advantages?

[shaneutt]

What changes were required for you to integrate the Kubebuilder project with the code generator? If you update the makefile target, could you please provide a snippet of the code?

I've done this for several project, and yes it's generally just adding a Makefile directive and chaining that in with your other generate directives. e.g.make generate.clientsets:

        $(CLIENT_GEN) \
                --go-header-file ./hack/generators/boilerplate.go.txt \
                --clientset-name clientset \
                --input-base '' \
                --input $(REPO)/$(APIS_DIR)/v1alpha1 \
                --input $(REPO)/$(APIS_DIR)/v1beta1 \
                --input $(REPO)/$(APIS_DIR)/v1 \
                --output-base pkg/ \
                --output-package $(REPO)/pkg/ \
                --trim-path-prefix pkg/$(REPO)/

And then you get a generated, updated clientset for all your kubebuilder managed APIs.

Is it possible to use code-generator and controller tools at the same time? If so, why / when that would be useful?

It has historically not been an issue to use them in tandem. I think it's important to note that I'm only talking about the use of client-gen specifically. It's useful because it automatically provides a typed client that can be used for end-users to integrate with your operators, with extremely low amounts of code and maintenance.

[camilamacedo86]

Hi @shaneutt

Thank you that is very interesting.

Regards:

It's useful because it automatically provides a typed client that can be used for end-users to integrate with your operators, with extremely low amounts of code and maintenance.

Can you please let us know more about it?
What is the use case scenario?
Can you please provide this info like: When a user wants to do X OR in the case where required N with client-gen we do Y otherwise, we or user would be required to do Z

[shaneutt]

Can you please provide this info like: When a user wants to do X OR in the case where required N with client-gen we do Y otherwise, we or user would be required to do Z

As a user of the ACME Operator, I want my Golang-based controllers to be able to read and write to resources of API types provided by the ACME Operator with a typed client.

---

For some additional context, we have upstream Kubernetes APIs that do this as well. In Gateway API we provide such a client for our CRDs:

https://github.com/kubernetes-sigs/gateway-api/tree/main/pkg/client/clientset/versioned

[camilamacedo86]

Hi @shaneutt

Thank you for provide the info.

However, sorry I never used the project https://github.com/kubernetes/code-generator

So, could you please help understand better:

As a user of the ACME Operator, I want my Golang-based controllers to be able to read and write to resources of API types provided by the ACME Operator with a typed client.

• When we create a controller and the APIs, we can watch and update the resources managed or observed by the controller. So, could you please explain what the requirement and use case scenario that motivates you to use https://github.com/kubernetes/code-generator? What is done by it that is not achievable with the default setup with controller-gen? OR is it more a matter of preference choice?
• What does it means a typed client? When you use it? What is its purpose?
• What did you try to do in the impossible ACME Operator and motivate you to use the https://github.com/kubernetes/code-generator?

[shaneutt]

* When we create a controller and the APIs, we can watch and update the resources managed or observed by the controller. So, could you please explain what the requirement and use case scenario that motivates you to use https://github.com/kubernetes/code-generator? What is done by it that is not achievable with the default setup with controller-gen? OR is it more a matter of preference choice?

We don't use this in our operators. Our users use these to integrate with our operators.

* What does it means a typed client? When you use it? What is its purpose?

Same as above, it's for end-users or cases where you are building something that integrates with our APIs.

* What did you try to do in the impossible ACME Operator and motivate you to use the https://github.com/kubernetes/code-generator?

The motivation was common practice. Again, this is also something that we provide in upstream Gateway API.

[camilamacedo86]

Hi @shaneutt

We might could add an e2e test to ensure that changes does not break with the integration and make a documentation explaining when to use, why to use, how to use. (ps: it still not clear 100% for me sorry)

[camilamacedo86]

It is no longer blocked.
Help wanted !!!

[camilamacedo86]

What we need to do here:

The goal is to create a sample project using code-generator and a reference document that will explain how, why, and when to use it with Kubebuilder.

Note: The source code in the documentation ideally comes from real samples. For example, the code used in the Cronjob Tutorial is sourced from the cronjob-sample: https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/cronjob-tutorial/testdata/project.

Following is a comprehensive detailed explanation of what needs to be done here and the suggestions on how to do it:

1) Understand the Project Client-Gen and when/why/how to use it with kubebuilder:

• Project Overview: Investigate how https://github.com/kubernetes/code-generator works.
• Usage Scenarios: Identify when and why Kubebuilder users would want to use this project and in what scenarios it can help them.
• Problem Analysis: Determine the specific problems that code-generator solves which are not addressed by controller-gen (used in https://github.com/kubernetes-sigs/controller-tools for manifest generation by default in kubebuilder).

2) Create a new documentation under the Reference section:

See: https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/reference

• The documentation should contain the above information following the kubebuilder/kubernetes docs standards
• Furthermore, should show how to integrate kubebuilder within and an example of use case scenario (ideally most common usage approach)

3) Ensure that example, sample are generated automatically and that the integration with client-gen is either called always on it so that we can validate that properly works:

• Sample Generation: Ensure that the samples in the docs are generated automatically. Refer to the getting started sample as a good example.
• Code Location: The relevant code for automatic documentation generation is located at https://github.com/kubernetes-sigs/kubebuilder/tree/master/hack/docs.
• Makefile Integration: When running make generate (which calls make docs-generate as defined in the Makefile), ensure that the docs are automatically updated with the latest code changes. So, we will need to add in the sample a new Makefile target to integrate the project with code-generator, similar to the approach described in this comment. That should result in us calling always this target to update/generate the sample again which will ensure that Kubebuilder work always with this project and will either address the purpose to validate the integration within.

[thesuperzapper]

As was discussed in #1152, I think the recommended way for users to consume CRDs in Go apps is to use the client.Client from controller-runtime (even outside of Kubebuilder apps).

This then lets you simply import the generated CRD interface types from the Kubebuilder app, and you also get the benefit of being able to use cache.Cache if you want to avoid making requests to the Kubernetes API, at the expense of caching them in memory using a ListWatch.

That is, there is not really a need for explicitly generated client sets anymore.