.NET 6 Memory Leak/Issue

[azampagl]

Describe the bug

We're noticing a memory issue running a typical ASP.NET Core 6 website behind IIS on a Windows Server. After a few days the worker process (w3wp.exe) memory consumption grows from 2 gigs up to 25 gigs. Performing an IIS Stop / Start "fixes" the issue in the intermediate. I believe the issue is related to a FileWatcher that is not freeing up memory. We've had the issue since .NET 5 and noticed that someone else had a very similar issue (but not necessarily the same) found here #31125 and here #31219. It was supposedly fixed in .NET 6 which is why we upgraded, but we are still having the issue.

Here are some screenshots of dotMemory on the memory data dump when the production server got to 26gigs.

Top level snapshot.

High level inspection page.

Drill down to the Byte[] array section (Similar Retention Section).

Drill down to the Byte[].

Drill down to the OverlappedData section (Instances).

Drill down to an individual OverlappedData.

To Reproduce

I have yet to been able to reproduce this on any of our team's development boxes. This only occurs on production which is making it hard to pinpoint.

Exceptions (if any)

None

Further technical details

• ASP.NET Core version: 6.0.0
• The IDE (VS / VS Code/ VS4Mac) you're running on, and its version: VS2022
• Windows Server 2019 (latest updates)
• SQL Server 2019 (Web Edition)
• IIS 10.0
• ASP.NET Core is running behind IIS in "InProcess"
• Include the output of dotnet --info:

dotnet --info Output

.NET SDK (reflecting any global.json):
 Version:   6.0.100
 Commit:    9e8b04bbff

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19042
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.100\

Host (useful for support):
  Version: 6.0.0
  Commit:  4822e3c3aa

.NET SDKs installed:
  3.1.120 [C:\Program Files\dotnet\sdk]
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.300 [C:\Program Files\dotnet\sdk]
  3.1.301 [C:\Program Files\dotnet\sdk]
  3.1.403 [C:\Program Files\dotnet\sdk]
  3.1.415 [C:\Program Files\dotnet\sdk]
  5.0.104 [C:\Program Files\dotnet\sdk]
  5.0.209 [C:\Program Files\dotnet\sdk]
  5.0.303 [C:\Program Files\dotnet\sdk]
  5.0.400 [C:\Program Files\dotnet\sdk]
  5.0.401 [C:\Program Files\dotnet\sdk]
  5.0.403 [C:\Program Files\dotnet\sdk]
  6.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Including screenshots of the IIS config so we are on the same page / just in case there is something misconfigured that we are unaware of.

Discussion / Side Notes

• I have yet to been able to reproduce this on any developer machine.
• On the first dotMemory screenshot, notice the discrepancy between ".NET total" vs ".NET used". Looks like .NET is allocating more than it needs and then not releasing? On .NET 5 there was significantly more ".NET used", so at least there seems to be some improvement. Regardless, I don't think the app should be using 5gigs, let alone 25 gigs.
• The FileWatcher consumptoin is what is concerning me. We don't use it anywhere in our code. And looking at our libraries, they don't appear to have any type of FileWatcher stuff either. My only though is that it is related to reloadOnChange during the CreateDefaultBuilder (https://github.com/dotnet/aspnetcore/blob/main/src/DefaultBuilder/src/WebHost.cs). Similar to issues Memory leak when restarting host on aspnet core 5 #31125 and IHost Memory leak with CreateDefaultBuilder #31219 where they cleared the sources and everything was fine.
• We don't do anything "crazy" in the sense of processing - it's a standard ASP.NET website with a SQL database using EF Core. We avoid most of the memory leak pitfalls (e.g. no event subscribing, etc). About the only non-trivial thing we do is run some processes in the background using a hosted service after a request is made. I have included the basic code below in case something pops out that is a big no-no. We more or less copy and pasted the code snippets from https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services
• Stackoverflow post that had a similar issue and resolved it by clearing his config.Sources in ConfigureAppConfiguration. https://stackoverflow.com/questions/66791228/net-5-ihost-filesystemwatcher-memory-leak
• Tagging @davidfowl as he was the one who looked like he worked on the previous bug fix / similar issue. Was the fix for the previous issue (Memory leak when restarting host on aspnet core 5 #31125) added to .NET6?

Code Snippets

Web.config

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <system.webServer xdt:Transform="Replace">
        <modules runAllManagedModulesForAllRequests="false">
            <remove name="WebDAVModule" />
        </modules>
        <handlers>
            <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
        </handlers>
        <aspNetCore processPath="dotnet" arguments=".\App.dll" stdoutLogEnabled="true" stdoutLogFile="..\logs\aspnetcore\stdout" requestTimeout="00:05:00" hostingModel="InProcess">
            <environmentVariables>
                <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Production" />
                <environmentVariable name="ASPNETCORE_SUPPRESSSTATUSMESSAGES" value="true" />
            </environmentVariables>
        </aspNetCore>
    </system.webServer>
</configuration>

Program.cs

public class Program
    {
        /// <summary>
        ///
        /// </summary>
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args)
        {
            return Host.CreateDefaultBuilder(args).ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.ConfigureLogging((hostingContext, logging) =>
                {
                    logging.ClearProviders();
                    logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
                    logging.AddEventLog(new EventLogSettings
                    {
                        LogName = "Application",
                        SourceName = hostingContext.Configuration["Name"]
                    });

                    switch (Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"))
                    {
                        case "Production":
                            // Do nothing
                            break;
                        case "Backup":
                            logging.AddDebug();
                            break;
                        case "Staging":
                            logging.AddDebug();
                            break;
                        case "Development":
                            logging.AddDebug();
                            break;
                        default:
                            throw new Exception("Unsupported environment.");
                    }
                });

                webBuilder.UseStartup<Startup>();
            });
        }
    }

Startup.cs (only including this so you can see how to utilize a background hosted service)

...
services.AddDbContext<ApplicationDbContext>(options =>
                options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));

...

services.AddScoped<IScopedWorker, ScopedWorker>();
services.AddSingleton<MySingletonThatAddsToBackground>();
// Follow the host service guide from microsoft.
services.AddHostedService<QueuedHostedService>();
services.AddSingleton<IBackgroundTaskQueue, BackgroundTaskQueue>();

MySingletonThatAddsToBackground.cs

public class MySingletonThatAddsToBackground
{
    private readonly IBackgroundTaskQueue _taskQueue;
    private readonly ILogger _logger;
    private readonly CancellationToken _cancellationToken;
    public IServiceProvider _services { get; }

    public MySingletonThatAddsToBackground(IServiceProvider services, IBackgroundTaskQueue taskQueue,
        ILogger<MonitorLoop> logger,
        IHostApplicationLifetime applicationLifetime)
    {
        _services = services;
        _taskQueue = taskQueue;
        _logger = logger;
        _cancellationToken = applicationLifetime.ApplicationStopping;
    }

    public void DoWorkBackground()
    {
        // Enqueue a background work item
        _taskQueue.QueueBackgroundWorkItem(async token =>
        {
            try
            {
                using (var scope = _services.CreateScope())
                {
                    var scopedWorker = scope.ServiceProvider.GetRequiredService<IScopedWorker>();

                    await scopedWorker.DoWork();
                }
            }
            catch (OperationCanceledException)
            {
                // Prevent throwing if the Delay is cancelled
            }
        });
    }
}

ScopedWorker.cs

public class ScopedWorker : IScopedWorker
{
    private readonly ApplicationDbContext _db;

    public ScopedWorker(ApplicationDbContext db)
    {
        _db = db;
    }

    public void DoWork()
    {
        var customers = _db.MyCustomers.ToListAsync();
        
        // Do stuff to customers.

        await _db.SaveChangesAsync();
    }
}

Questions

• Should we not be running it in InProcess mode to get better dump data?
• Should we utilize Workstation GC vs the default Server GC?
• Should we be setting the private memory section on the IIS app pool? All this really does is recycle the app pool which doesn't really solve the issue. To my knowledge, there is no way to restrict an IIS process memory consumption.
• I'm currently using dotMemory on a memory dump from the production box. Is there something else I should be doing to get better insight?

[pranavkm]

Out of curiosity, is the memory released if you force a GC? I believe dotMemory has an option to do that.

[azampagl]

@pranavkm The app is running in release/production mode and dotMemory is not available on the server. Although it is not advised, would running System.GC.Collect() (https://docs.microsoft.com/en-us/dotnet/api/system.gc.collect?view=net-6.0#System_GC_Collect) have the same effect you want?

[davidfowl]

Is the leak coming from those byte[] pinned by the FileSystemWatcher?

[azampagl]

@davidfowl I believe so... The main byte[] that is taking up 5gigs is pinned via "OverlappedData" from FileSystemWatcher. See screenshot. I'm kind of new to dotMemory, so if tell me how to navigate to what you're looking for I can do so (I can also zoom/team/share screen - whatever is easiest for your team).

[davidfowl]

That memory is pinned so it's not gonna go away with a GC. We need to figure out who is creating those. The best way to go about this would be to capture an allocation trace in production. It'll show you the stack that is causing these allocations.

https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#how-to-collect-top-level-gc-metrics

https://docs.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak (though this is just the dump).

Where is this file system watcher rooted?

[azampagl]

@davidfowl I read your first link. I've put PerfView on the machine. If I'm understanding this correctly I should be able to just run

PerfView.exe /nogui /accepteula /KernelEvents=Process+Thread+ImageLoad /ClrEvents:GC+Stack /BufferSize:3000 /CircularMB:3000 collect

On the production server to capture the allocation trace? (I ran a quick sample and it generates a PerfViewGCCollectOnly.etl.zip file.)

I don't need to target the process ID (w3wp.exe) or anything?

In regards to the dump, is the dotnet-dump preferred over the normal windows memory dump?

[azampagl]

Details for the byte[] array set (some are duplicates from above)

Interesting that it's showing most of it is in Gen2, where 42mb is used/pinned, 5gb used/unpinned, 12.5G free but not released...

[davidfowl]

I don't need to target the process ID (w3wp.exe) or anything?

No it collects the entire machine. If you use dotnet trace it'll require the pid

On the production server to capture the allocation trace? (I ran a quick sample and it generates a PerfViewGCCollectOnly.etl.zip file.)

Yes, you're trying to figure out where the allocations are coming from.

In regards to the dump, is the dotnet-dump preferred over the normal windows memory dump?

Either is fine.

[azampagl]

Unfortunately, the 12 hour PerfView collection I had didn't go well. When I opened it up in PerfView, it was missing the "“GC Heap Alloc Ignore Free (Coarse Sampling)” section. I don't know if it was because of the length of recording or file size. When I sample for say 30 minutes, it's available.

I did notice that after 7 hours, the size stayed at "6,300MB" exactly which was strange, but I let it keep running.

The process grew from 2gigs to about 11.5gigs over that 12 hours span during the night (theoretically our lower traffic time).

I've started a trace again this time using PerfView64 just in case and the /FocusProcess= argument when collecting to focus on the process. I'll report back in a few hours. I didn't want to do an app pool reset yet, so I've started the recording while the current process is sitting at 12gigs

[davidfowl]

@azampagl Do you have the trace still?

cc @Maoni0 @brianrob

[azampagl]

@davidfowl

Started the perfview monitoring at ~9:30AM EST when the process w3wp.exe was already at 12G (it had grown from 2G to 12G overnight). I recorded for 2.5 hours with the following command.

.\PerfView64 /nogui /accepteula /KernelEvents=Process+Thread+ImageLoad /ClrEvents:GC+Stack /BufferSize:3000 /CircularMB:3000 /FocusProcess=69368 collect

Breakdown
All final recordings stopped at ~11:59AM EST. The w3wp.exe process at the time was using 16.5G (so gained 4.5G in about 2.5 hours)

dotnet-dump (high level, I have more screenshots if you need)

PerView

Thoughts

• Why is it not releasing the "free" memory from Gen2 (~7.67gigs)?
• Strange that dotMemory is referencing the bytes with relation to the FileWatcher, but I see minimal mention of the FIleWatcher in the PerfView results.

[azampagl]

@davidfowl @Maoni0 @brianrob

I decided to experiment on our staging website various endpoints to see if I could simulate the leak in a more controlled environment. I made a basic controller with various actions to test various resources. I used Locust (https://locust.io/) with 3 workers to simulate 200 users making requests simultaneously. The staging application is identical to the production one on the same server / same IIS settings (just a different domain). I first tested on my local dev machine, and then the staging website (after each action I would reset the app pool just in case).

The controller for the requests was the following. There was no special action filters, pre-controller processing, etc.

    public class Test : Core
    {
        /// <summary>
        /// 
        /// </summary>
        public Test(IServiceProvider serviceProvider) : base(serviceProvider)
        {
        }

        /// <summary>
        /// Test a normal response.
        /// </summary>
        [HttpGet("test/normalresponse")]
        public async Task<IActionResult> TestNormalResponse()
        {
            Random rnd = new Random();
            return Ok(rnd.Next(1, 10000));
        }

        /// <summary>
        /// Test a process that runs in the background but immediately returns a response.
        /// </summary>
        [HttpGet("test/bgprocess")]
        public async Task<IActionResult> TestBgProcess([FromServices] Managers.MemTest.IMemTestDebounced memTestManager)
        {
            Random rnd = new Random();
            var flightDepartmentID = 1;
            var flightID = rnd.Next(1, 100);
            await memTestManager.FlightPush(flightDepartmentID, flightID);
            return Ok();
        }

        /// <summary>
        /// Test a request that does a basic db query and returns the id.
        /// </summary>
        [HttpGet("test/dbquery")]
        public async Task<IActionResult> TestDbQuery([FromServices] ApplicationDbContext db)
        {
            Random rnd = new Random();
            var id = rnd.Next(1, 10000);
            // ID is a unique key, indexed
            var result = await db.People.Where(x => x.ID == id).AsNoTracking().FirstOrDefaultAsync();
            return Ok(id);
        }

        /// <summary>
        /// Test a process that does a basic db query and returns a DTO. I'm looking to test the DB query AND serialization to
        /// see if any memory issues arise.
        /// </summary>
        [HttpGet("test/dbqueryanddto")]
        public async Task<IActionResult> TestDbQueryAndDto([FromServices] ApplicationDbContext db)
        {
            Random rnd = new Random();
            var id = rnd.Next(1, 10000);
            // ID is a unique key, indexed
            var result = await db.People.Where(x => x.ID == id).AsNoTracking().FirstOrDefaultAsync();

            if (result == null)
                return NotFound();

            return Ok(new DBQueryDTO(result));
        }

        /// <summary>
        /// A basic DTO that just wraps an DB model.
        /// </summary>
        public class DBQueryDTO
        {
            public int ID { get; set; }
            public string NameFirst { get; set; }
            public string NameLast { get; set; }

            public DBQueryDTO()
            {
            }

            public DBQueryDTO(Models.Passenger model)
            {
                ID = model.ID;
                NameFirst = model.NameFirst;
                NameLast = model.NameLast;
            }
        }
    }

Round 1

Environment:

Windows Pro 10, SQL Server 2019 Web Edition, Intel i7, 32 GB RAM

Action: TestNormalResponse, TestBgProcess, TestDbQuery, TestDbQueryAndDto

Memory usage caps out at around 380 MB on all endpoints being blasted. I attached dotMemory and watch the GC kick in pretty regularly.

Locust note: in debug mode on my local dev machine the app was only handling ~30 request per second / RPS

Environment:

Windows Server 2019, SQL Server 2019 Web Edition, Intel Xeon Gold 6226R x2, 64 GB RAM
<ServerGarbageCollection>true</ServerGarbageCollection>, hostingModel="InProcess"

Action: TestNormalResponse

Memory quickly spiked and then memory usage caps out at around ~1,210 MB and periodically GC would kick in and bring it back down to ~1,175 MB. Ran for about 5 minutes. Stopped requests and I waited about 10 minutes, memory never went back down below 1,170 MB.

Locust note: server was able to handle about ~2300 RPS

Action: TestBgProcess

Memory quickly spiked and then memory usage caps out at around ~1,200 MB and periodically GC would kick in and bring it back down to ~1,180 MB. Ran for about 5 minutes. Stopped requests and I waited about 10 minutes, memory never went back down below 1,1600 MB.

Locust note: server was handling about ~2200 RPS

Action: TestDbQuery (here is where things were interesting)

Memory quickly spiked to 3,100 MB and then continuously went up 70-100 MB per second (!). Once the app got to about 7,500 MB it then continued to grow 5-25 MB per second (random spikes in between) and would not let up. After about 2 minutes it grew to 16,000 MB and I eventually had to stop because I didn't want to overload the server. I waited for 30 minutes hoping the GC would kick in and bring it way back down but never did. I took a memory dump to see what was going on:

I eventually called System.GC.Collect() via code (GC.GetTotalMemory(false) before: 5308489584, GC.GetTotalMemory(true) after: 3764229944). No effect. No movement in memory after 10 mins.

Locust note: server was able to handle about ~1500 RPS, a pretty significant drop after DB / EF core usage was introduced - expected.

Action: TestDbQueryAndDto

Didn't even bother testing as the DB endpoint was enough to cause issues.

Round 2

Wanted to see what would happen if I turned hostingModel="OutOfProcess"

Environment:

Windows Server 2019, SQL Server 2019 Web Edition, Intel Xeon Gold 6226R x2, 64 GB RAM
<ServerGarbageCollection>true</ServerGarbageCollection>, hostingModel="OutOfProcess"

Action: TestDbQuery

Process now showed up as dotnet.exe (expected). Memory quickly spiked to 4,100 MB and then continuously went up 50-100 MB per second. Once the app got to about 6,000 MB it then continued to grow 5-50 MB per second (random spikes in between) and would not let up. It did go up noticably slower than the InProcess verion, but after about 4 minutes it grew to 15,200 MB and I eventually had to stop because I didn't want to overload the server. I waited for 30 minutes hoping the GC would kick in and bring it way back down but never did.

Locust note: server was able to handle about ~1400 RPS (slightly less than InProcess version)

Round 3

Wanted to see what would happen if I turned to false (Workstation GC)

Environment:

Windows Server 2019, SQL Server 2019 Web Edition, Intel Xeon Gold 6226R x2, 64 GB RAM
<ServerGarbageCollection>false</ServerGarbageCollection>, hostingModel="OutOfProcess"

Action: TestDbQuery

Memory went to about 700 MB after 30 seconds (significanntly less). Once the app got to about 1,200 MB it then continued to grow 2-10 MB per second (random spikes in between). I let the test run for a solid 6 minutes at full blast and it "only" got to 5,000 MB. That being said it was still growing (although slow), so had I left it run longer it probably would have continued to go up. After I stopped running the request I let it sit to see if the GC would recover but it never did.

Locust note: server was able to handle about ~470 RPS (significant drop! but to be expected since Server GC allows for higher throughput)

Thoughts:

• For whatever reason, the DB query / EF core is causing massive memory spikes when blasted and then the process never lets the memory go (even though according to dotMemory, the memory is "free").
• Maybe I didn't wait enough time to see if the memory would go down eventually in various scenarios? Waited over 3 hours in one scenario, memory never leveled down.
• Workstation GC vs Server GC definitely had preferred memory consumption, but that is expected. What was surprising was to see even the Workstation GC version continue to climb (although slower).
• I know .NET Core will consume as much memory as it is allowed, but it seems strange that it never releases?
• Maybe I need to re-evaluate my expectation and put hard limits on the GC / Heap? Our website is growing in popularity so we're definitely getting more requests per second on average. Should I consider https://docs.microsoft.com/en-us/dotnet/core/run-time-config/garbage-collector e.g DOTNET_GCHeapHardLimit

[azampagl]

For what it's worth, I did apply both DOTNET_GCHighMemPercent (65%) and DOTNET_GCHeapHardLimitPercent (68%) while performing another blast test. Once the box got to about 60% of physical memory I did notice the server was processing significantly less requests per second (went from 2,500 RPS to about 400), which I could only assume was GC kicking in (good). It did slow the growth but I left it go for an extra 10 minutes and it still went through both of those limits and got up to 75% memory before I stopped.