Fix memory leak by caused by dynamicconfig value logging #6169
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
taylanisikdemir
requested review from
Shaddoll,
neil-xie,
davidporter-id-au,
Groxx,
shijiesheng,
agautam478,
jakobht,
3vilhamster,
sankari165,
dkrotx and
demirkayaender
as code owners
Shaddoll
approved these changes
Jul 16, 2024
Groxx
approved these changes
Jul 16, 2024
davidporter-id-au
approved these changes
Jul 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
Dynamic config values are kept in a map to be compared against previous values for logging purposes. This log was not used in practice at all. It requires huge amounts of memory to keep the last snapshot of all evaluated dynamic config values. The underlying map has no retention so keys like "{configname}{domain}{workflowid}" increase cardinality and stay there forever.
This issue was identified when pprof of a high memory usage history service instance was inspected. The call stack initiated by replication task processor ends up calling
GetBoolPropertyFilteredbyDomainIDAndWorkflowIDwhich populates the underlying logValues map per workflow keys that never get deleted.Why?
Avoid wasting memory and prevent leaks for unused logs.