Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Option to disable kubernetes client side rate limiting #334

Closed
hexalellogram opened this issue Aug 13, 2024 · 2 comments · Fixed by #336 or #338
Closed

[Feature Request]: Option to disable kubernetes client side rate limiting #334

hexalellogram opened this issue Aug 13, 2024 · 2 comments · Fixed by #336 or #338
Labels
enhancement New feature or request

Comments

@hexalellogram
Copy link

Describe why this change is needed

Hello, we're trying to use Istio + istio-csr + cert-manager + aws-privateca-issuer in one of our production clusters, where as part of a new deployment hundreds of pods will spin up and request certificates at a time. We suspect that Kubernetes client-side rate limiting is throttling the number of pods that get a new cert at a time, causing our deployments to take much longer and time out.

We've already ruled out similar bottlenecks in istio-csr (cert-manager/istio-csr#352) and cert-manager itself in our configuration. However, we have not seen improvements in the rate of pods that get new certs at a time, leading us to believe that there is still a bottleneck somewhere - we suspect it is aws-privateca-issuer. Looking through this repository I cannot find anywhere where the QPS and burst rate settings are being overridden, so I believe that the defaults for the client-go/rest package are being used.

We're using Kubernetes 1.29 which includes API Priority and Fairness in GA, which eliminates the need for client-side rate limiting.

Describe solutions and alternatives considered (optional)

A way to disable client-side rate limiting entirely (similar to the above linked istio-csr patch), perhaps through a Helm chart option, would be very helpful.

Is there anything else you would like to add?

This is blocking us from being able to implement mTLS in one of our critical production clusters.

@hexalellogram hexalellogram added the enhancement New feature or request label Aug 13, 2024
@ARichman555
Copy link
Contributor

Thank you for raising this issue with the AWS Private CA Issuer plugin. We will review your submission and respond back to you here as soon as possible.

@ARichman555
Copy link
Contributor

Hi, we believe the issue is not the Kubernetes client side rate limiting but rather our Reconcile loop is taking too long. Currently we're issuing the certs synchronously, so each Reconcile loop will wait for the cert to be fully issued on the first run rather than checking later. We will look into adjusting this to issue certs asynchronously.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
2 participants