Fix missing resource version when updating the scale subresource of custom resource

[knight42]

What type of PR is this?
/kind bug

What this PR does / why we need it:
Not to clear the resource version of custom resource when saving it to etcd.

Which issue(s) this PR fixes:
Fixes #80515

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Scale custom resource unconditionally if resourceVersion is not provided

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Hi @knight42. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[knight42]

/assign @apelisse

[apelisse]

I don't think that we should be doing a GET before the request. Scale is an imperative command and unless you specify the resource-version, it should just be executed as specified.

This comment from #80515 in particular lets me think that there is something wrong:

On that note, I also noticed that if you want to update a CR by crafting the request manually and using curl, you also need to specify the resource version or you will get the same error. I'm not sure if this is expected and related, but I'm writing it down as it might be related. As far as I know, this is not required for some native resources at all, but I'm not sure is this the case for CRDs.

I'd like to understand why that is the case, I suspect it may be a bug/behavior we don't understand in the CRD handler.

[knight42]

@apelisse It makes sense, I' ll try to deep dive into it.

[knight42]

@apelisse I thought I have found out the root cause.

When updating an object, apiserver will check if the object could be updated unconditionally:

https://github.com/kubernetes/apiserver/blob/781c3cd1b3dc5b6f79c68ab0d16fe544600421ef/pkg/registry/generic/registry/store.go#L481-L489

If not, the validation will fail:

https://github.com/kubernetes/apiserver/blob/781c3cd1b3dc5b6f79c68ab0d16fe544600421ef/pkg/registry/generic/registry/store.go#L531-L537

And custom resource could not be updated unconditionally now:

https://github.com/kubernetes/apiextensions-apiserver/blob/102230e288fd77afcf1a6e7258eac008a891d885/pkg/registry/customresource/strategy.go#L158-L161

@apelisse Do you have a clue why custom resource could not do an unconditional update?

[apelisse]

Isn't this object supposed to handle scale resources for CRs: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresource/etcd.go#L202 ?

[knight42]

@apelisse Yeah it is, but the problem is still related to https://github.com/kubernetes/apiserver/blob/781c3cd1b3dc5b6f79c68ab0d16fe544600421ef/pkg/registry/generic/registry/store.go#L531-L537.

Here is how it goes:
When the handler is handling the update request of scale subresource, the scale variable defined here is actually the request body sent from kubectl, which contains no resource version. Later the handler executes cr.SetResourceVersion(scale.ResourceVersion), the resource version of CR is still not set. As a result, r.store.Update, which invokes https://github.com/kubernetes/apiserver/blob/781c3cd1b3dc5b6f79c68ab0d16fe544600421ef/pkg/registry/generic/registry/store.go#L453, returns an error because of missing resource version

So I guess the fix may be simply removing this line:

kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresource/etcd.go

Line 276 in c1d2ac4

cr.SetResourceVersion(scale.ResourceVersion)

since the CR got in the handler contains resource version. What do you think?

[knight42]

@apelisse PTAL

[knight42]

/assign @lavalamp

[liggitt]

one comment update, and one additional test, then this LGTM

[knight42]

@liggitt I found that the "retry on conflicts" mechanism may not be optimal. As shown in https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/80572/pull-kubernetes-bazel-test/1193803978668773382, in extreme cases, such as frequent concurrent patches, we may keep retrying until timeout then return the last error. So I decided to switch to patching the replicas filed on the server side, how is that sound to you?

[liggitt]

two comments on the test, then lgtm

[liggitt]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: knight42, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiextensions-apiserver/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

/hold cancel

[aermakov-zalando]

Any chance for this to be backported to 1.15/1.16? @liggitt

[liggitt]

This is a more invasive change than is typically backported. Note that #81342 made it into 1.16 and modifies kubectl to use patch when scaling, which works with custom resources in 1.15/1.16 servers.

[aermakov-zalando]

@liggitt I see, thanks!