Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rate limiting of v1beta1.external.metrics.k8s.io #1503

Closed
bpinske opened this issue Jan 12, 2021 · 5 comments · Fixed by #1512
Closed

Rate limiting of v1beta1.external.metrics.k8s.io #1503

bpinske opened this issue Jan 12, 2021 · 5 comments · Fixed by #1512

Comments

@bpinske
Copy link
Contributor

bpinske commented Jan 12, 2021

I have keda deployed in all of my environments and I notice that I am getting spammed in all of them with rate limiting notices. Including environments which are not actually using Keda at all yet. Is there anything that I should be doing to address this, or is this a known issue of some sort? I don't see any documentation around this situation. I haven't seen any ill effect on keda's performance where I am actually using it.

kube-apiserver-ip-10-28-11-137.us-west-1.compute.internal kube-apiserver E0112 17:07:58.690813       1 controller.go:114] loading OpenAPI spec for "v1beta1.external.metrics.k8s.io" failed with: OpenAPI spec does not exist
kube-apiserver-ip-10-28-11-137.us-west-1.compute.internal kube-apiserver I0112 17:07:58.690830       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.external.metrics.k8s.io: Rate Limited Requeue.

Keda version: 2.0.0
Kubernetes Version: 1.18.6

@surki
Copy link
Contributor

surki commented Jan 13, 2021

Thanks @bpinske . I am seeing those errors as well (in AWS EKS 1.18)

@surki
Copy link
Contributor

surki commented Jan 13, 2021

Looks like we should do something like this?: kubernetes-sigs/prometheus-adapter#335

@zroubalik
Copy link
Member

Yeah, update of the https://github.com/kubernetes-sigs/custom-metrics-apiserver deps and related code should help. Though it may not be that easy step, as there might be dependency conflicts between mainly (k8s) deps used by KEDA and the updated version of custom-metrics-adapter :(

@surki
Copy link
Contributor

surki commented Jan 18, 2021

@zroubalik I will make those changes and send a PR "soon" (have done similar changes for other customer metrics servers, so familiar with the details)

@zroubalik
Copy link
Member

@surki that's great, thanks for doing that!

@surki surki mentioned this issue Jan 19, 2021
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants