reduce traffic by caching Node objects

[pohly]

There's an old TODO in the code about this:

sig-storage-lib-external-provisioner/controller/controller.go

Line 1331 in d3bf59b

selectedNode, err = ctrl.client.CoreV1().Nodes().Get(nodeName, metav1.GetOptions{}) // TODO (verult) cache Nodes

Scale testing shows that this is causing significant traffic when provisioning many volumes with selected node: 80kB/s for GET of Nodes, followed by 1.7kB/s for GET of CRDs.

Link to results (likely to go stale):
http://35.243.169.140:3000/d/A354LoAGz/pohlyat-storage-tests?orgId=1&from=1606743419743&to=1606743617019&var-source=adhoc-1333398611282628608-central&var-etcd_type=All&var-etcd_operation=All&var-verb=All&var-resource=All

/cc @verult @jsafrane

[pohly]

Is an informer the right solution? It's less precise than doing a GET (local cache might be stale), but I suppose any caching solution has that problem.

We also need to be aware that the provisioner then gets informed about all nodes, even when it only ever needs a handful of nodes in a larger cluster.

[wongma7]

external-provisioner controller already has an informer, maybe we should deprecate SelectedNode *v1.Node and pass SelectedNode *string to Provision instead, leaving it up to the Provisioner how to retrieve a Node object (and its topology)? https://github.com/kubernetes-csi/external-provisioner/blob/89079cfeb84cbbfcda976504d46e6b2b8352adb7/cmd/csi-provisioner/csi-provisioner.go#L216

I am not too worried about stale Node cache either since I don't think node topology is something that should change often if at all...

[wongma7]

Or, to follow the existing pattern https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/blob/master/controller/controller.go#L473, accept a NodeInformer as an argument so if external-provisioner controller already has one it can pass it in, if not, this controller will create one.

[pohly]

So let's use an informer. I would leave the current behavior unchanged (i.e. use GET as fallback) and let the caller decide whether it wants to use an informer.