Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metrics: Add a gauge for the number of active blocking queries #6846

Closed
banks opened this issue Nov 28, 2019 · 1 comment · Fixed by #7224
Closed

metrics: Add a gauge for the number of active blocking queries #6846

banks opened this issue Nov 28, 2019 · 1 comment · Fixed by #7224
Assignees
Labels
theme/operator-usability Replaces UX. Anything related to making things easier for the practitioner theme/telemetry Anything related to telemetry or observability type/enhancement Proposed improvement or new feature

Comments

@banks
Copy link
Member

banks commented Nov 28, 2019

We currently have a metric for blocking queries but it's almost useless since it's a counter that increments whenever a blocking query begins:

metrics.IncrCounter([]string{"rpc", "query"}, 1)

What operators (and consul developer 😉 ) really want to know about is how many blocking queries are actually open/being handled right now.

This should be easy:

  • add an uint64 to the server that is accessed atomically (watch struct alignment - probably needs to be at the top of the struct)
  • atomically increment it right on the line linked above when a blocking query starts
  • defer an atomic decrement for when the query returns
  • after each operation emit a gauge metric with the current value

Note that we shouldn't replace or rename the current metric since that is a breaking change for users monitoring stacks, this should be a new gauge, I suggest we call it consul.rpc.queries_blocking or something (that bikeshed needs painting).

@banks banks added type/enhancement Proposed improvement or new feature theme/operator-usability Replaces UX. Anything related to making things easier for the practitioner theme/telemetry Anything related to telemetry or observability labels Nov 28, 2019
@banks banks added this to the 1.7.0 milestone Nov 28, 2019
@mkcp
Copy link
Contributor

mkcp commented Jan 27, 2020

Looking to pick this up later this week

@mkcp mkcp self-assigned this Jan 31, 2020
@hanshasselberg hanshasselberg removed this from the 1.7.0 milestone Feb 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
theme/operator-usability Replaces UX. Anything related to making things easier for the practitioner theme/telemetry Anything related to telemetry or observability type/enhancement Proposed improvement or new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants