Proposing a way to customize aggregations and percentils for histograms #3238
+141
−63
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Currently, it's only possible to do this at a system-wide level, using
the following agent configuration options:
This patch makes this behaviour customizable on a metric basis, based
on a list of regexes and their corresponding configs; e.g. the following
agent configuration:
would run only the
max
aggregation but report both the 90th and 95thpercentiles on any prefix starting with
my_prefix
; but any other metricwould have the same aggregations as above (ie
max, median, avg, count
),and report only the 95th percentile.
The first matching regex in the configuration will be applied. If none
match, defaults to the config with no regex, or else to the hard-coded
agent default.
In particular, that makes this patch entirely backward compatible.
Motivation
This would come in handy to be able to limit aggregations for some
metrics but not all of them on any given box; in particular to avoid
pushing too many custom metrics with respect to plan limits.
For example, we use Datadog to do some high-level profiling in some
of our codebases; we'd be perfectly happy having only the
median
aggregation for these metrics, which would allow us to lower our
number of custom metrics; but we do not want to stop pushing
other aggregations for other histogram metrics on the same boxes.
Tests
Updated exisiting tests, and added a minimal test on the new feature, but
will add more if/when the Datadog dev team validates the approach. In particular,
tests should be added at least for:
Additional Notes
Related to customer ticket 82067.