Asp.Net Core API controller memory leak repopend #45098

[PhilOkeeffe]

We recently upgraded from .Net 5 - .Net 7 and straight away we could see the memory going up on all our servers and never releasing. Exactley the same as posted originally by @oferze #45098
I dont know why this post is closed it's not fixed in a version of VS as suggested (17.4.1.) I have 17.5.5. It's not related to visual studio it's .Net 7 for sure as we never had this problem on .net 5

I agree with @oferze https://github.com/oferze that this is a regression with .NET 7 since .NET 5 & 6 somehow does it better.

Our code hits the controller every 5 seconds but regardless of what our code does or how big the objects are that may or may not end up on the large heap or gen 1 - 2 the memory does not release on the worker process for w3wp.exe for a simple end point.

@oferze demonstrates this well with the code below but you will see the same even if you return an empty string and hit this endpoint continously.

Simply create an endpoint and constantly refresh (f5) once published or in debug mode and see the memory grow and not get released.

    [HttpGet("[action]")]
    public IActionResult GetMem(int testNumber)
    {
        var mem = Process.GetCurrentProcess().PagedMemorySize64;
        return Ok($"Mem: {mem:N0}");
    }

[davidfowl]

@PhilOkeeffe There's no proof of a leak. Maybe you're saying there's more memory usage, but there's no leak (or at least no proof of a leak based in provided information).

Our code hits the controller every 5 seconds but regardless of what our code does or how big the objects are that may or may not end up on the large heap or gen 1 - 2 the memory does not release on the worker process for w3wp.exe for a simple end point.

Did you read through the issue?

cc @mangod9 @Maoni0 for .NET 6 vs 7 (using regions) might be the difference here but I can't tell.

[PhilOkeeffe]

Yes I did read it! Did you read it? If you hit a simple endpoint that does next to nothing and the memory grows but doesn't release this is a regression \ leak with how memory is handled in .NET 7 since .NET 5

[davidfowl]

@PhilOkeeffe if this is to be a productive investigation, then we'd need more information. Hitting an endpoint with the above code does not cause a memory leak.

[mangod9]

@PhilOkeeffe, can you please clarify whether the memory continues to grow endlessly, or it stabilizes after hitting a steady state (but is just higher than 5)?

[PhilOkeeffe]

@mangod9 Ok so we have observed the memory continually grow up to 12gb in one example and doesn't release unless the App pool recycle period in minutes is hit or iis is restarted.
Below is the memory consumption of the iis worker process for one of our instances.

@davidfowl Ok I will hold fire on code becuase our code worked on .Net 5 and putting our code to one side I would expect IIS to not use lots of memory for something so simple as hello world style endpoint. Try have a simple endpoint hit every 5 seconds for a period 24 hours and see for yourself!!

[davidfowl]

@PhilOkeeffe I don't see the problem when I try to reproduce it, the GC does its job and collects the memory. Maybe there are more details missing, like machine specs? What sort of machine is IIS running on? How many CPU cors and how much memory?

[PhilOkeeffe]

Hi @davidfowl please find below the Server Spec, but also the CSProj settings of the project attatched and also the Spec of my local machine which also shows memory going up but not releasing.

csprofile.txt

Server Spec
Windows Server 2016 Virtual Machine
CPU 4 virtual processors 2.99 GHZ
16gb RAM
IIS version 10.0.14393.0

App pool
.NET CLR version : No Managed Code
Managed pipeline mode : Classic

C:\Windows\system32>dotnet --list-runtimes
Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Local Dev PC
Windows 10 2.8Ghz 4 Cores 16gb Ram
IIS version 10.0.19041.1
App pool
.NET CLR version : No Managed Code
Managed pipeline mode : Classic

Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[davidfowl]

Does the repro when running on both your dev PC and on the server? Are you running on IIS or IIS Express (inside of visual studio) when your reproduce this problem? Can you reproduce this problem with a new application that does nothing but send requests to that endpoint you showed?

[PhilOkeeffe]

@davidfowl Yes can be reproduced on both dev locally and server. Using IIS express but same for all flavours But most importantly we can even reproduce the behaviour using the oob template for Asp.netcore with react.js hitting the weatherforecast end point 200 times in a minute.

2 Minute Sample (100 hits per min to weatherforecast endpoint.. Start Memory 62mb goes up on every request getting to 99mb but never released irrespective of GC)

1Minute sample

See here hitting up the Weatherforecast endpoint roughly 200 times in a minute you can see a couple of random GC happen but they dont make a dent in the memory being used.

[davidfowl]

Can you do the same with a non-react template? Can you reproduce the issue with a standalone asp.net core web api application? (trying to narrow down the variables)

[PhilOkeeffe]

@davidfowl Yes very same just hitting the home page with a constant refresh . Did you manage to reproduce this on the asp.net core with react template hitting the weatherforecast endpoint? Also please can you tell me what the resolution to this #38722 as it's the exact same bug (worker process (w3wp.exe) memory consumption growing quick and not releasing) ?

[PhilOkeeffe]

@davidfowl Please can you tell me what the resolution to this #38722 as it's the exact same bug (worker process (w3wp.exe) memory consumption growing quick and not releasing) ? Did you manage to reproduce this on the asp.net core with react template hitting the weatherforecast endpoint?

[PhilOkeeffe]

@davidfowl Please can you tell me what the resolution to this #38722 as it's the exact same bug (worker process (w3wp.exe) memory consumption growing quick and not releasing) ? Did you manage to reproduce this on the asp.net core with react template hitting the weatherforecast endpoint?

[davidfowl]

I don't see constant memory growth when I run a load test against an ASP.NET Core API:

[davidfowl]

That was .NET 8. This is what I see with .NET 7:

I'm using bombardier to generate load. Here's what I did:

Put the app in release mode, opened the memory profiler in visual studio:

Then ran a load test for 1 minute and observed the graph:

bombardier-windows-amd64.exe https://localhost:7021/ -d 1m

It spikes > 200MB then flatlines around ~120MB.

Then I collected counters from the process:

See https://learn.microsoft.com/en-us/visualstudio/profiling/dotnet-counters-tool?view=vs-2022
or https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters

Here are the results of the same load test:

and the final results:

You can try using those tools on your setup and see if you can replicate the results.

[PhilOkeeffe]

@davidfowl These are there results for our app running on a dev machine calling our endpoint with non production data...
I have also set the GC mode to workstation via the runtimeconfig.json
{
"runtimeOptions": {
"configProperties": {
"System.GC.Server": false
}
}
}
bombardier-windows-amd64.exe http://localhost:47513/api/measurements -d 1m

[davidfowl]

Doesn't look like a leak to me. Run that test for longer, say 5 minutes and see what the stats look like. I also want to confirm that the default template does not leak.

[PhilOkeeffe]

@davidfowl Ok I will get back to you!

[PhilOkeeffe]

Hello @davidfowl here is the results from a 5 Minute load test on my local machine for this endpoint http://localhost:47513/api/measurements

When i started the load test iis express was using 113 mb after 1 minute it was on 3175mb after 4 minutes it was using 6670mb and the Heap size increases?. I have a runtimeconfig.json file if you have any suggestions for preventing iis from taking all the memory. My understanding is that by default iis will take all 90% of memory unless to specifiy "System.GC.HeapHardLimitPercent"?

[davidfowl]

I have also set the GC mode to workstation via the runtimeconfig.json

I missed this. Did you do this to reduce member usage? Is the above trace also workstation GC? I'm not sure you're running workstation GC though, you can confirm at runtime by exposing an endpoint that shows if that GC mode is on using the GCSettings.IsServerGC API.

My understanding is that by default iis will take all 90% of memory unless to specifiy "System.GC.HeapHardLimitPercent"?

No, IIS is no different from the application running outside of IIS with respect to the GC settings. By default ASP.NET Core will use server GC (make sure you read this section).

The quote to remember:

What survives usually determines how much work GC needs to do; what doesn't survive usually determines how often a GC is triggered.

Hopefully that section of the document makes it a little clearer as to when GC's happen. Setting the "System.GC.HeapHardLimitPercent" is a way to limit the visible GC memory, but it might be worth understanding more about the GC behavior with another type of trace (https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#how-to-start-a-memory-perf-analysis)

PS: I'm not sure what's going on with your Gen2 Size...

[PhilOkeeffe]

@davidfowl Getting back to your last comments Yes I wasn't using workstation GC in the end it was being ignored so I moved the setting into the csproj file where it was working but sadly not making any impact!
The only thing that has made a difference was compacting the LOH at the begining of the endpoint call...
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce; GC.Collect()
The compacting really halted iis memory growth but unfortuently this is not really as option as our endpoint is triggered every 5 seconds and I could see this was causing a performance hit! Just so you know our endpoint has a size of 16,600 bytes in size so will always going on the LOH but this was not a problem on .Net 5?

[davidfowl]

What is this endpoint doing?

[mangod9]

Just so you know our endpoint has a size of 16,600 bytes in size so will always going on the LOH but this was not a problem on .Net 5?

Did you mean 16mb here or 16kb? Adding @cshung since we have been investigating a similar issue where allocations just over 16mb were being made per request which would cause increased working set but not a leak you are observing. We are working on a fix for that particular issue.

to check whether its a similar one, is it possible to test the scenario with DOTNET_GCName=clrgc.dll env. var? This moves the GC implementation back to .NET 6.

[PhilOkeeffe]

@davidfowl @mangod9 @cshung Yes I have tested .Net6 now and it does appear to be a similair issue. When I move the implementation back to .Net 6 it uses much less memory the working set (iis mem) is half the size v's .Net 7!
Do you have a ETA for the .net 7 fix?

I set the dotnet 6 dll via the runtimeconfig
"System.GC.Name": "clrgc.dll"

Dot Net 6 v's 7

Dot net 6

[cshung]

@PhilOkeeffe, can you please confirm if the allocation 16k or 16M? The 16M case is well understood and the fix is currently work in progress here dotnet/runtime#87715. On the other hand, if you could reproduce a working set issue with 16k, then that's something we don't know and we will need to investigate.

[PhilOkeeffe]

Hi @cshung the issue is reproduceable with 16k calls not 16Mb.

[davidfowl]

My take is that your app will not perform well when you build systems that’s allocate 50MB strings per request. I think it’s very likely that your application has a problem that is causing memory growth. There may be regressions with the GC keeping more memory around but if you’re seeing the same issue with older GC then it tells me the application’s allocation patterns are suboptimal.

The reason these issues typically go this way is because it takes a long time to narrow a repro with data that the team can use to investigate. It’s never as simple as it initially seems.

[oferze]

Our case (the other ticket) is a valid use case of returning a big Json to the user as string. Suggesting that the app might have other problems because a 50MB response string "doesn't seem right", is not very productive, because each app's business case is different. We can discuss whether the app's design is good or not (it is) but that is missing the point: the GC works better in .NET 5/6 than in 7. In .NET 7, we proved that allocations stay around for longer than they should, in a minimal API app, uninfluenced by any other factor.

By all means, I'm well aware that investigation takes time, but please don't close a ticket or dismiss it as "your app will not perform well when you build systems that’s allocate 50MB strings per request."

The problem is certainly there.

[mangod9]

@PhilOkeeffe no there isnt a way to move just the GC back to 5.0.

[cshung]

@oferze, the appropriate forum to report .NET Core GC-related issues is the dotnet/runtime repo. There we also have area labels that will allow us to focus on only GC issues. If you could get your issue opened there and have the appropriate labels set, that will certainly get our attention sooner.

In the case of repeated allocation of large objects, we have recently found and fixed a memory utilization issue related to them. Can you try dotnet/runtime#87715 and see if that improves the memory utilization of your application?

Note that the fix is still in-flight - while the underlying issue is well understood, we might have to alter the design of the fix on the PR to be more adaptive.

The fix has nothing to do with @PhilOkeeffe's case though, in his scenario the number of large objects is insignificant. Experiments also indicate that the fix doesn't help with the memory utilization in his app at all. We are still trying to see what has gone wrong there.

[PhilOkeeffe]

Hello @cshung @mangod9 @davidfowl Just keeping in touch has there been any progress on this?
Cheers
Phil

[cshung]

We are in the progress of investigating the issue. Here are some preliminary findings.

tl;dr - we have found something, a general improvement to improve heap balancing somehow tripped a perf heuristic to keep the heap size small.

On my machine, I am able to observe the following:

.NET 7: Private memory after all requests finish, around 800M
.NET 8: Private memory after all requests finish, around 2.2G [Clearly a regression in memory usage]

.NET 7: 1636.18 requests per second
.NET 8: 1632.05 requests per second [Throughput is the same, well within the error bars]

Looking deeper into the reasons why .NET 8 uses more memory, we have:
.NET 7: Performed 9 GCs (usually), including a Gen 2 background GC
.NET 8: Performed 2 GCs (usually), one gen 0 and one gen 1

Looking further into why .NET 8 performs fewer GC, we have:
.NET 7: Gen 0 budget started off with 30 MB and stay more or less the same throughout the 5 minutes of bombarding
.NET 8: Gen 0 budget started off with the same 30 MB, but grow to about 120 - 150MB

Keep looking at why the Gen0 budget for .NET 8 grows faster:
.NET 7: Survived about 1.2 MB during the first GC and stay there more or less the same.
.NET 8: Survived about 2.1 MB during the first GC and keep growing to about 4.5 MB

The 2.1 MB survival caused us to trip a heuristic here. The variable desired_per_heap is the allocation budget.

                        // if min GC size larger than true on die cache, then don't bother
                        // limiting the desired size
                        if ((min_gc_size <= GCToOSInterface::GetCacheSizePerLogicalCpu(TRUE)) &&
                            desired_per_heap <= 2*min_gc_size)
                        {
                            desired_per_heap = min_gc_size;

That's why .NET 7 is able to keep the desired_per_heap low by entering the branch and .NET 8 is not. The extra survived bytes must be the reason.

Keep looking at why .NET 8 survived more:

Since both .NET 7 and .NET 8 run on the same app and have the same initial allocation budget, it is weird to see .NET 8 has more survived bytes. I captured a dump right before the first Gen0 plan phase begin. Now I understand that on .NET 8, the allocations are spread more evenly across all heaps, so in total we allocated more in .NET 8 before we .NET 7 trip the allocation budget line. The survival rate is the same, but since .NET 8 allocated more bytes, the total survived bytes is higher.

Having the allocations spread more evenly across the heaps is generally a good thing because we can balance the work better across the threads when GC happens. It is unfortunate in this case that it tripped the heuristic. I can't really blame the balancing, that should be a good thing, we need to figure out a solution the memory increase though.

[davidfowl]

@cshung did you compare it with .NET 6?

[cshung]

did you compare it with .NET 6?

Nope, I didn't. By default, we will work on the latest version first, which we are most familiar with and most likely to be able to make a fix on.

[davidfowl]

Right, but the issue is comparing .NET 6 and .NET 7, not 8 and 7.

[cshung]

Right, but the issue is comparing .NET 6 and .NET 7, not 8 and 7.

Ooops, missed that. Let's chat offline.

[PhilOkeeffe]

Hello ,
Is there any updates on a fix.? @cshung @mangod9 @davidfowl Can you raise the priority \ get more people looking at this?
Please bear in mind .net 5 is the benchmark as our production app was ported from .Net 5 to 7.
Just to recap the regression appears to get worse as you go up major the versions.
.Net 6 memory usage also continues to grow with .Net 7 the same but much faster growth and from your comments above the regression is a lot worse in in the jump from .net 7 .net 8 so this fix needs to go back to the memory usage profile of .Net 5.

Thanks
Phil

[cshung]

While we don't have a fix yet, I can share some findings on testing around .NET 6. The test is based on running the .NET 6 GC over the .NET 7 runtime. The point of the experiment is to isolate so that we know if there is anything that goes wrong in the GC.

The .NET 6 experiment result is almost identical to .NET 7.

.NET 6: Private memory after all requests finish, around 800M
.NET 7: Private memory after all requests finish, around 800M

.NET 6: 1561.64 requests per second
.NET 7: 1636.18 requests per second

.NET 6: Performed 9 GCs (usually), including a Gen 2 background GC
.NET 7: Performed 9 GCs (usually), including a Gen 2 background GC

.NET 6: Gen 0 budget started off with 30 MB and stay more or less the same throughout the 5 minutes of bombarding
.NET 7: Gen 0 budget started off with 30 MB and stay more or less the same throughout the 5 minutes of bombarding

It is obvious that .NET 8 regressed upon .NET 7, but there is no evidence that there is a regression between .NET 6 and .NET 7 in the GC yet.

In order to move forward, it would be nice if we can run your repro app under the .NET 6 runtime. @PhilOkeeffe, can you help us to get your repro app to run on top of it?

[PhilOkeeffe]

@cshung @davidfowl @mangod9 create a runtimeconfig.template.json file in the root and build run it then it uses .net 6 gc.

put this inside
{
"configProperties": {
"System.GC.Name": "clrgc.dll"
}
}

You will see it's using .net 6gc from the url http://localhost:47513/api/gcmode at the bottom of the page

[cshung]

This is exactly the .NET 6 experiment I did (I used environment variables instead of runtime config but it should be the same) and observed no difference on my end.

[PhilOkeeffe]

@cshung @davidfowl @mangod9 Just to be clear I observed the difference from .net 6 and 7 gc in production (not the sample app) over serveral hours demonstrated from 3 weeks ago. I wasn't activley looking for this comparison, it's just sometihng that stood out as we switched from 7 to 6. I wouldn't get to caught up in the difference between 6&7 as It's difficult to spot a pattern over a small period of time the main issue for us is they both gobble up memory quickly and don't release unlike 5. If you can get 6 & 7 or just 7 to release memory like .net 5 (pre upgrade to 7) then we are in a good place.

3 weeks ago image

2 weeks ago image

[cshung]

We are in the progress of investigating the issue. Here are some preliminary findings.

With respect to the locally-reproducible working set regression I found between 7 and 8, I have found out the reason why heap balancing works better on .NET 8. This is due to the change here.

In particular, we changed:

gc_heap::min_gen0_balance_delta = (dd_min_size (gen0_dd) >> 3);

to

gc_heap::min_gen0_balance_delta = (dd_min_size (gen0_dd) >> 6);

As the min_gen0_balance_delta decreases, the allocator try harder to balance objects across heaps, that's why

The heaps are fuller by the time we perform the first GC (first alloc budget is the same, but old impl trip it faster because old heaps are less balanced so biggest heap reaches budget faster)
=> larger survived bytes during first GC (survival rate is the same, but fuller heap means more survived bytes)
=> larger budget during the first GC (next alloc budget is a function of survived bytes)
=> failed to stay within 2 * cache size
=> failed to cap the budget within a single cache size
=> decreased GC frequency
=> increased memory usage.

At this point, the root cause is known.

[davidfowl]

@cshung Again, the above is about .NET 5 - 7, not .NET 8. I'm glad that issue is root caused but there's another one that wasn't addressed it seems.

[cshung]

@davidfowl, indeed, I am aware of that. With what is given here - the net5-net7 regression is neither reproducible locally nor do we have any meaningful data to work with. There is not much I can do here. That's why I focused on the 7 - 8 regression, at least it is actionable.

[PhilOkeeffe]

@cshung @davidfowl @mangod9 If you need to move the sample application provided onto .Net 5 you could just set the Target framework! Are you close to resolving on .Net7 ?

[langtu]

I had the same problem when strees test WeatherForecastController in default template (return 1000 items for quick mem growing). Managed memory looks good, but unmanaged memory keep increasing until the machine unresponsive.

I run the test in PopOS latest (based on Ubuntu 22.04 with dotnet-sdk-7.0 from Microsoft repo), within Rider 2023.2 and full memory profiler enabled.

In my production app, the behaviour is the same. After a lots of request to web api, total memory usage increase a lots and never reduce, until server crash.

In the graph, the memory stop growing only when the app not receiver any request, and keep growing when new requests coming.

[ghost]

Thank you for contacting us. Due to a lack of activity on this discussion issue we're closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn't been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!

[yarong-lifemap]

Please do not close this issue.

[davidfowl]

Such shade @oferze, no need for snark, the issue can be re-opened.

[oferze]

From past experience, closed tickets were not re-opened. Appreciate your cooperation. Deleted my comment.

[davidfowl]

That happens for multiple reasons, I promise you we don't sit around and close issues for fun (even though we have a lot of them).

I think @cshung made some progress on .NET 7 but it's still unclear if this is unexpected behavior or not in Server GC mode.

[ghost]

Thank you for contacting us. Due to a lack of activity on this discussion issue we're closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn't been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!