Set reasonable QPS/Burst for resources RESTMapper client

[absoludity]

Description of the change

Sets reasonable QPS and Burst for the client config used only in the discovery of the REST API (by the RESTMapper).

Before this change, registering the resources plugin took around 8 seconds due to client throttling:

I0302 00:40:47.539843       1 plugins.go:137] Successfully registered plugin "/plugins/fluxv2-packages/fluxv2-packages-v1alpha1-plugin.so"
I0302 00:40:48.318594       1 request.go:597] Waited for 109.310691ms due to client-side throttling,
 not priority and fairness, request: GET:https://10.96.0.1:443/apis/networking.k8s.io/v1beta1
...
I0302 00:40:56.919228       1 request.go:665] Waited for 8.611529015s due to client-side throttling,
 not priority and fairness, request: GET:https://10.96.0.1:443/apis/kustomize.toolkit.fluxcd.io/v1beta1

I0302 00:40:56.923634       1 plugins.go:137] Successfully registered plugin "/plugins/resources/resources-v1alpha1-plugin.so"
...
I0302 00:40:56.924955       1 server.go:158] Starting server on :50051

With this change, it's under 500ms (with no warnings of throttling):

I0302 05:39:33.515837       1 plugins.go:137] Successfully registered plugin "/plugins/fluxv2-packages/fluxv2-packages-v1alpha1-plugin.so"
I0302 05:39:34.010957       1 plugins.go:137] Successfully registered plugin "/plugins/resources/resources-v1alpha1-plugin.so"

Benefits

Faster startup time for the kubeapps-apis server (see #4329).

Possible drawbacks

Initial hammer of the k8s API server when the rest mapper initialises, but don't see this as an issue.

Applicable issues

• ref kubeapps-apis CrashLoopBackoff #4329

Additional information

I did investigate whether we could use the client-getter to create a client using the chart configured defaults, but the client-getter is very careful to restrict itself to user requests, enforcing a user token, ensuring any service account token from the service is not used, etc., which I do not think we should change. Given that this client is used only for generating the rest mapper, I ended up going for simple constants.

[castelblanque]

Good catch! Would it make sense to have them as environment variables, defaulted to those values if not present? Don't remember if we are using env variables for this kind of setup. I was thinking in the cases of environments where K8s resources are scarce (e.g. Docker desktop extension or small clusters).

[antgamdia]

+1 to provide these vars externally. I mean, we're already passing QPS and burst in the chart, so it is just a matter of using it, no?

[absoludity]

Not quite. I did look at it, but it'll require changing the arguments of the grpc registration function. I (personally) don't see any need to be able to modify them, since unlike the user client, they're only used on startup when initializing the RESTMapper.

But I'm not against passing them through... it's just a matter of updating all the plugins to match. Happy to do so if you both think it's better.

[absoludity]

It might even be a chance to replace the (growing) args to the grpc registration function with a struct so that we don't need to update the function signature each time.

[castelblanque]

It might even be a chance to replace the (growing) args to the grpc registration function with a struct so that we don't need to update the function signature each time.

+1, that would be nice and give flexibility to the registration process.

[absoludity]

OK, I've updated this PR, rebasing on #4372 so it's clear as an example how we can update the args to the plugin registration without having to modify other plugins necessarily. PTAL :)

[antgamdia]

Great, thank you! I do like this approach rather than hardcoding the defaults as constants. Especially, when it comes to rate limitations. This way we ensure Kubeapps can be adjusted for fitting each cluster's performance needs.

[antgamdia]

Ah, ok, so these values are already been set by the kube-api-qps and kube-api-burst CLI flags we added in the past, aren't they?

[absoludity]

Yes, if we're passing them in, I didn't see a reason to use different flags for this client than those already passed in for the user client, so populate the values from the existing flags.

[absoludity]

Landed at #4386