Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High high memory usage traced to ScopedLogger #2234

Closed
solomon-okolie opened this issue Dec 7, 2023 · 6 comments
Closed

High high memory usage traced to ScopedLogger #2234

solomon-okolie opened this issue Dec 7, 2023 · 6 comments
Labels

Comments

@solomon-okolie
Copy link

solomon-okolie commented Dec 7, 2023

APM Agent version

1.25.1

Environment

Operating system and version: WindowsServer, Windows10

.NET Framework/Core name and version : .NET 6

Application Target Framework(s): .NET 6

Describe the bug

We have console a app that processes messages from a queue and we capture transactions manually like below
Agent.Tracer.CaptureTransaction("TestProcessor", "MQP", () => { // code here }); and it's eating quite a lot of memory after a couple of hours. This looks very similar to #2145. but this time happening in ScopedLogger.
image

We did a little test based on the commit fix for #2145..

Before change
WithConcurrentDictionary
Heap snapshot after 11,000 transactions, and a forced a GC

After change
WithConditionalWeakTable
Heap snapshot after 11,000 transactions, and a forced a GC

@solomon-okolie solomon-okolie added the bug Something isn't working label Dec 7, 2023
@solomon-okolie solomon-okolie changed the title Memory Leak in ScopedLogger High high memory usage traced to ScopedLogger Dec 7, 2023
Mpdreamz added a commit that referenced this issue Dec 13, 2023
This also ensure we no longer use captures state in lambda's by moving the logging
callback to static to disable capturing state.
@Mpdreamz
Copy link
Member

Thanks for raising this, I added an initial fix for this here:

#2242

Including a few other logging related fixes.

I'll do a pass over our logging to find any high cardinality suspects too.

@solomon-okolie
Copy link
Author

Any idea when a new nuget version will be available?

@AlfonsRoerdink
Copy link

I will be great if a new hotfix release can be made because my pods are crashing after a few days because out-of-memory.
Is there an estimation when the new nuget version will be available?

@Mpdreamz
Copy link
Member

Mpdreamz commented Jan 8, 2024

This is now released under 1.25.3.

So sorry and apologies you got bit too by this @AlfonsRoerdink let us know if the update did not resolve your issues.

@AlfonsRoerdink
Copy link

I installed the new version and the application is already running 16 hours without a remarkable memory leakage,
so it seems to be fixed! Thank you very much!

@Mpdreamz
Copy link
Member

Thanks a ton for looping back with us!

I will go into the weekend much more relaxed now :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants