Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Surface Kubernetes client rate limiter metrics #6007

Merged
merged 1 commit into from
Sep 9, 2022

Conversation

barkbay
Copy link
Contributor

@barkbay barkbay commented Sep 8, 2022

This PR helps to surface the latency intentionally added by the Go client in order to throttle the requests ran against the K8S control plane.

The PR basically adds a new Prometheus histogram named k8s_client_rate_limiter_duration_seconds to collect the observed latency, here is an example of the output:

# HELP k8s_client_rate_limiter_duration_seconds Kubernetes client rate limiter latency in seconds. Broken down by verb, and host.
# TYPE k8s_client_rate_limiter_duration_seconds histogram
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="0.005"} 86
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="0.025"} 86
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="0.1"} 87
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="0.25"} 90
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="0.5"} 96
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="1"} 109
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="2"} 114
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="4"} 122
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="8"} 126
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="15"} 127
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="30"} 127
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="60"} 127
k8s_client_rate_limiter_duration_seconds_bucket{host="k8s_host",verb="GET",le="+Inf"} 127
k8s_client_rate_limiter_duration_seconds_sum{host="k8s_host",verb="GET"} 76.791233874
k8s_client_rate_limiter_duration_seconds_count{host="k8s_host",verb="GET"} 127

@barkbay barkbay added >enhancement Enhancement of existing functionality v2.5.0 labels Sep 8, 2022
Copy link
Contributor

@thbkrkr thbkrkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thbkrkr thbkrkr self-assigned this Sep 9, 2022
@barkbay barkbay merged commit d5c35c6 into elastic:main Sep 9, 2022
@barkbay barkbay deleted the client-rate-limiter-metrics branch September 9, 2022 12:18
fantapsody pushed a commit to fantapsody/cloud-on-k8s that referenced this pull request Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement Enhancement of existing functionality v2.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants