feat: Drop unnecessary listing for the sake of watch reinitialization #616
Conversation
tosi3k
force-pushed
the
clustercache-list-watch
branch
from
0c6045d
to
4029ec6
Compare
| @@ -666,8 +656,10 @@ func (c *clusterCache) watchEvents(ctx context.Context, api kube.APIResourceInfo | |||
|
|
|||
| obj, ok := event.Object.(*unstructured.Unstructured) | |||
| if !ok { | |||
| resourceVersion = "" | |||
There was a problem hiding this comment.
I'd like to see if we can get away without resetting this and using whatever resourceVersion we already had. If we find we get into some nasty loop, we can always re-add it.
There was a problem hiding this comment.
We can't drop resetting the locally held RV here as well - put appropriate rationale in a comment in the code there as well.
| // re-synchronize API state and restart watch if retry watcher failed to continue watching using provided resource version | ||
| case <-w.Done(): | ||
| resourceVersion = "" |
There was a problem hiding this comment.
Maybe see if we can get away without this? If we need it, we'll probably want to add a comment exactly why we needed it.
There was a problem hiding this comment.
We can't drop it - I put appropriate comment in the code to explain the rationale behind resetting resourceVersion.
This is the place where the retry watcher exits early and we are forced to relist in order to get a fresher RV from the server: https://github.com/kubernetes/kubernetes/blob/0fc167103189a4d462c3cc7a17b360c3d998f4bf/staging/src/k8s.io/client-go/tools/watch/retrywatcher.go#L208-L211.
There was a problem hiding this comment.
+1 - this has to remain here.
This should generally be rare, so relist from etcd is fine in those situations.
tosi3k
force-pushed
the
clustercache-list-watch
branch
from
4029ec6
to
b4fa2a7
Compare
tosi3k
changed the title
|
@crenshaw-dev thanks for the review - I'll respond to the comments here later on. FWIW as agreed offline during yesterday's sync, I split the fix into two PRs - this one would just drop unnecessary listing after watch expiry and #617 would make the list API calls target the watch cache instead of etcd. I guess we can proceed with the latter one. |
tosi3k
force-pushed
the
clustercache-list-watch
branch
from
b4fa2a7
to
409990b
Compare
Signed-off-by: Antoni Zawodny <zawodny@google.com>
tosi3k
force-pushed
the
clustercache-list-watch
branch
from
409990b
to
f7c5e6e
Compare
|
|
This LGTM from k8s perspective. |
This change addresses the performance issue existing in the cluster cache described in argoproj/argo-cd#18838.
kube-apiserverlogs for the Pods resource (supposed super-low latency logged for the WATCH requests is due to a bug in Kubernetes: kubernetes/kubernetes#125614):