-
Notifications
You must be signed in to change notification settings - Fork 812
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(sdk-metrics): prevent per-reader storages from keeping unreported accumulations in memory #4163
Merged
pichlermarc
merged 13 commits into
open-telemetry:main
from
dynatrace-oss-contrib:fix/4115
Oct 10, 2023
Merged
fix(sdk-metrics): prevent per-reader storages from keeping unreported accumulations in memory #4163
pichlermarc
merged 13 commits into
open-telemetry:main
from
dynatrace-oss-contrib:fix/4115
Oct 10, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… accumulations in memory
Codecov Report
@@ Coverage Diff @@
## main #4163 +/- ##
==========================================
+ Coverage 92.26% 92.72% +0.46%
==========================================
Files 331 268 -63
Lines 9473 7907 -1566
Branches 1999 1682 -317
==========================================
- Hits 8740 7332 -1408
+ Misses 733 575 -158
|
pichlermarc
added
bug
Something isn't working
priority:p1
Bugs which cause problems in end-user applications such as crashes, data inconsistencies, etc
labels
Oct 5, 2023
legendecas
reviewed
Oct 8, 2023
legendecas
approved these changes
Oct 8, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
bug
Something isn't working
priority:p1
Bugs which cause problems in end-user applications such as crashes, data inconsistencies, etc
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which problem is this PR solving?
As described in #4115 (reproducer https://github.com/pichlermarc/repro-4115), when two or more
MetricReader
instances are registered to aMeterProvider
, it causes memory usage to increase until the app runs out of heap space.This happens as
TemporalMetricProcessor
adds entries to a map_unreportedAccumulations
for eachMetricCollectorHandle
for which there is one for eachMetricReader
. When oneMetricReader
collects, the unreported accumulation is first added to all the otherMetricCollectorHandle
entries. When the associatedMetricCollectorHandle
is never used withTemporalMetricsProcesssor#buildMetrics
, its entry in_unreportedAccumulations
is never reset - which causes the array for aMetricCollectorHandle
to grow on each collection cycle until the app runs out of memory. This can happen when aSyncMetricStorage
orAsyncMetricStorage
is created and registered for a singleMetricReader
. In this case,MetricStorageRegistry#getMetricStorage
will be called with aMetricCollectorHandle
, only returning the storages with which theMetricReader
is associated.This never happened with a single
MetricReader
as every TemporalMetricsProcesssor ever registered will be collected in each cycle.This PR fixes the issue by:
MetricCollectorHandle
s on creating theTemporalMetricsProcessor
.MetricCollectorHandles
s forTemporalMetricsProcessor
s that are shared across allMetricReaders
MetricCollectorHandle
forTemporalMetricsProcessor
s that are intended for a singleMetricReader
.collectors
argument fromTemporalMetricsProcesssor#buildMetrics
, but using the initially registeredMetricCollectorHandle
instead to ensure that only pre-registeredMetricReader
s that are expected to callTemporalMetricsProcesssor#buildMetrics
eventually are kept in memory.Fixes #4115
Type of change
How Has This Been Tested?