Cannot deploy workload CRD using Resource Interpreter

[chengleqi]

Please provide an in-depth description of the question you have:
I followed the steps in the examples/customresourceinterpreter/README.md document to deploy the webhook, CRD, and webhook-configuration. After that, I tried to deploy workload.yaml, but received an error:

root@karmada-local:~/karmada/examples/customresourceinterpreter# kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver apply -f workload.yaml
Error from server (BadRequest): error when creating "workload.yaml": Workload in version "v1alpha1" cannot be handled as a Workload: strict decoding error: unknown field "spec.template.metadata.labels"

Upon inspection, I confirmed that my CRD has been correctly deployed to the karmada-apiserver, and the webhook has obtained an EXTERNAL-IP through metallb:

root@karmada-local:~/karmada/examples/customresourceinterpreter# kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver get crds
NAME                                                         CREATED AT
clusteroverridepolicies.policy.karmada.io                    2024-01-03T05:19:15Z
clusterpropagationpolicies.policy.karmada.io                 2024-01-03T05:19:15Z
clusterresourcebindings.work.karmada.io                      2024-01-03T05:19:15Z
cronfederatedhpas.autoscaling.karmada.io                     2024-01-03T05:19:15Z
federatedhpas.autoscaling.karmada.io                         2024-01-03T05:19:15Z
federatedresourcequotas.policy.karmada.io                    2024-01-03T05:19:15Z
multiclusteringresses.networking.karmada.io                  2024-01-03T05:19:15Z
multiclusterservices.networking.karmada.io                   2024-01-03T05:19:15Z
overridepolicies.policy.karmada.io                           2024-01-03T05:19:15Z
propagationpolicies.policy.karmada.io                        2024-01-03T05:19:15Z
resourcebindings.work.karmada.io                             2024-01-03T05:19:15Z
resourceinterpretercustomizations.config.karmada.io          2024-01-03T05:19:15Z
resourceinterpreterwebhookconfigurations.config.karmada.io   2024-01-03T05:19:15Z
serviceexports.multicluster.x-k8s.io                         2024-01-03T05:19:15Z
serviceimports.multicluster.x-k8s.io                         2024-01-03T05:19:15Z
workloads.workload.example.io                                2024-01-03T07:01:25Z
works.work.karmada.io                                        2024-01-03T05:19:15Z

root@karmada-local:~/karmada/examples/customresourceinterpreter# kubectl get svc karmada-interpreter-webhook-example -n karmada-system -o wide
NAME                                  TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE   SELECTOR
karmada-interpreter-webhook-example   LoadBalancer   10.96.171.19   172.18.0.8    443:32166/TCP   63m   app=karmada-interpreter-webhook-example

I used controller-gen to regenerate a CRD based on workload_types.go, and I found that the workload CRD provided by the Karmada is also correct.

What do you think about this question?:

Environment:

• Karmada version: hack/local-up-karmada.sh master branch
• Kubernetes version:
• Others:

[zhzhuang-zju]

Hi~ From your description, the error is reported because field spec.template.metadata.labels does not exist. Refer to #4128 (comment) and #4315 (review), the root cause is controller-gen skips embedded metada when generating CRDs. The modification method is: execute the command controller-gen crd:generateEmbeddedObjectMeta=true paths=./examples/customresourceinterpreter/apis/... output:crd:dir=./examples/customresourceinterpreter/apis/ to regenerate the workload's crd.

[zhzhuang-zju]

Are you interested in filing a PR to revise this issue? If you like, you can implement it by referring to #4315

[chengleqi]

Are you interested in filing a PR to revise this issue? If you like, you can implement it by referring to #4315

Thank you for your solution. I added the crd:generateEmbeddedObjectMeta=true parameter and used controller-gen to regenerate the Workload CRD, then applied it to the karmada-apiserver. After that, I was able to successfully complete the Resource Interpreter example.

And I am happy to submit a PR to fix this issue.

[RainbowMango]

Thanks @chengleqi , just go for it.
/assign @chengleqi

[zhzhuang-zju]

/remove-kind question
/kind bug

[zhzhuang-zju]

There are two points that puzzle me about this question:

• Why was this problem not discovered during the early development of the requirements?
• The e2e test actually covers this scenario, why is no error reported?

Through investigation, I answered the above two questions, as follows:
Refer to https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/2885-server-side-unknown-field-validation,

kubernetes added server-side schema validation, enabling the API server to validate that objects received in the request body in POST, PUT, and PATCH requests contain no unknown or duplicate fields. A new query parameter FieldValidation was introduced to indicate the level(strict/warn/ignore) of field validation the server should perform.

Refer to kubernetes/enhancements#2885, this feature is enabled in version 1.25, and the version of the component karmada apiserver was upgraded to 1.25 in PR #2539, which is why no problem was found during the initial development.

The value and meaning of query parameter FieldValidation is:

• Strict: Strict field validation, errors on validation failure
• Warn: Field validation is performed, but errors are exposed as warnings rather than failing the request
• Ignore: No server side field validation is performed

Refer to https://github.com/kubernetes/apiserver/blob/547675e99460e49cd95ca3253f0740fee2d2853d/pkg/endpoints/handlers/rest.go#L405-L415, server side field validation is in Warn mode by default.
Refer to https://github.com/kubernetes/kubectl/blob/b54e299c96090bb8a461e459251c49cfaf6fa843/pkg/cmd/util/helpers.go#L650-L670, kubectl will automatically use server side field validation in Strict mode.
This is why e2e(server side) does not report an error but kubectl reports an error. If kubectl apply also wants not to report an error, just set flag --validate='warn'.

[zhzhuang-zju]

@RainbowMango Since this problem is related to the version of karmada apiserver, does the PR #4315 which fixed similar issues in operator installation scenario require cherry pick to the previous release version?