Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix double dots .. in metric keys #8416

Closed
mkcp opened this issue Jul 23, 2020 · 3 comments
Closed

Fix double dots .. in metric keys #8416

mkcp opened this issue Jul 23, 2020 · 3 comments
Labels
theme/telemetry Anything related to telemetry or observability
Milestone

Comments

@mkcp
Copy link
Contributor

mkcp commented Jul 23, 2020

In the process of building up the dot-delimited metric keys for http paths, consul.http.GET.v1.token we sometimes emit keys with two contiguous dots, e.g. consul.http.GET.v1.token..count. We suspect that these keys are being emitted as consul.http.GET.v1.token._.count with the empty string being replaced with an underscore, but that underscore gets trimmed by dogstatsd.

We can narrow this down by running a local receiver for the dogstatsd protocol and inspecting the wire traffic to see if it's a problem within Consul, or a quirk of Grafana and/or the dogstatsd protocol.

Once we localize the issue, we probably want to figure out a way to fix it and deprecate the old double-dot metrics.

Screen Shot 2020-07-20 at 1 02 45 PM
Screen Shot 2020-07-20 at 11 35 41 AM

Slack thread identifying the issue https://hashicorp.slack.com/archives/C045NJD17/p1595263451020200

@mkcp mkcp transferred this issue from hashicorp/consul-foundations-metrics-1.9 Jul 30, 2020
@mkcp mkcp added the theme/telemetry Anything related to telemetry or observability label Jul 30, 2020
@mkcp
Copy link
Contributor Author

mkcp commented Jul 30, 2020

A bit of googling confirmed that - and _ are trimmed by the dogstatd. I haven't isolated exactly where this occurs, but that's enough information that we can try to build a workaround.

@mkcp mkcp added this to the 1.9 milestone Aug 18, 2020
@mkcp
Copy link
Contributor Author

mkcp commented Sep 29, 2020

Resolved by #8271

@mkcp
Copy link
Contributor Author

mkcp commented Oct 8, 2020

Resolved in 1.9

@mkcp mkcp closed this as completed Oct 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
theme/telemetry Anything related to telemetry or observability
Projects
None yet
Development

No branches or pull requests

1 participant