-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ref(statsd): Add metric name as a tag for Sentry errors #1797
Conversation
Adding the name as tag will not help us to know, which metrics are being logged most. You simply know the one that's next in line. Compared with the increased complexity, I'm not sure this change is worth it. |
No, but metrics that are logged more often are more likely to be in tags, so I believe it can give us an approximation. Building something to efficiently (we don't want to lose even more metrics, especially these) record the metrics we log and analyzing results is too much effort, vs going through the metrics in the codebase and directly deleting some. In the end, this is about the best effort and finding a temporary solution. If this, still, doesn't provide enough value, we can call it off. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sure there's ways to reduce code duplication here, but since these macro definitions are rarely touched, I think we can go ahead with the change as-is.
0cb4232
to
c6e1d9f
Compare
Co-authored-by: Oleksandr <1931331+olksdr@users.noreply.github.com>
Co-authored-by: Oleksandr <1931331+olksdr@users.noreply.github.com>
* master: ref(statsd): Add metric name as a tag for Sentry errors (#1797)
This reverts commit d873ce2.
From time to time, we still get errors from relay dropping metrics due to the channel being full, see #1680. This PR attempts to add a tag to that Sentry issue, to identify more easily what metrics may be worsening the bottleneck. Next steps TBD, based on conclusions from the errors. This code is likely to be temporary and be removed after we figure out what and how to deal with the errors.
The approach
The original idea was to extract the error and directly include it in the error here:
relay/relay-statsd/src/lib.rs
Lines 115 to 116 in 2ec2a5e
There are two problems with that approach. Firstly, the error types come from statsd's source code and it's hard to consider every case. Secondly, the code is too generic and closed that it's hard to inject something nicely without a decent amount of workarounds. Lastly, I also tried replicating the issue locally generating a lot of metrics (lots of threads, queue size to 0 and 1, sample rate 1.0), but I have not been successful.
In the end, the simplest solution was to add the name as a tag to the scope where there's access to it, wrapping the call where the actual error is logged.