feat(span_concentrator): replace std map with hashbrown for better perf #1234
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rformances # Motivations Creating and checking for the existence of the aggregation key is the most expensive operation performed by data-pipeline if there is a high drop rate. Currently this operation is not optimal as equality has to go through virtual dispatch because of limitations on the std hashmap. Changing to hashbrown gives us: * A more efficient hashing function * We can implement the Equivalent trait instead of the borrow trait to compare owned and borrowed key, which performs better * We can drop the Cow and just use either String or &str * We can use the entry API on the hashmap so we don't have to potentially access the hashmap twice These changes bring a 25% perf improvement on the span concentrators benchmarks # Changes * Use hashbrown::HashMap instead of std::HashMap in the stats collector * Replace AggregationKey<'static> with OwnedAggregationKey * implement hashbrown::Equivalent instead of Borrow
BenchmarksComparisonBenchmark execution time: 2025-09-25 16:00:21 Comparing candidate commit dc2c080 in PR branch Found 1 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 2 unstable metrics. scenario:concentrator/add_spans_to_concentrator
CandidateCandidate benchmark detailsGroup 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Group 9
Group 10
Group 11
Group 12
Group 13
Group 14
Group 15
BaselineOmitted due to size. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1234 +/- ##
==========================================
- Coverage 71.65% 71.60% -0.06%
==========================================
Files 355 355
Lines 56317 56309 -8
==========================================
- Hits 40354 40320 -34
- Misses 15963 15989 +26
🚀 New features to boost your workflow:
|
VianneyRuhlmann
approved these changes
Sep 23, 2025
| .into_iter() | ||
| .map(|(key, value)| (Cow::from(key.into_owned()), Cow::from(value.into_owned()))) | ||
| .collect(), | ||
| peer_tags, |
Contributor
There was a problem hiding this comment.
very nit: this should be:
peer_tags,
is_trace_root: span.parent_id == 0,
to match field order in the struct
paullegranddc
force-pushed
the
paullgdc/data-pipeline/stats_collector_hmap
branch
from
f96a538 to
dc2c080
Compare
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
The expected merge time in
|
dd-mergequeue
bot
deleted the
paullgdc/data-pipeline/stats_collector_hmap
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivations
Creating and checking for the existence of the aggregation key is the most expensive operation performed by data-pipeline if there is a high drop rate.
Currently this operation is not optimal as equality has to go through virtual dispatch because of limitations on the std hashmap.
Changing to hashbrown gives us:
These changes bring a 25% perf improvement on the span concentrators benchmarks locally (only 10% in CI??)
Changes