[ASPNET Core] Memory Usage / GC Thrashing

[ctolkien]

Fairly basic ASPNET Core Application, 1 minute after booting sitting there idling, there is a huge amount of GC thrashing and the memory is nearly peaking at 600mb:

Compared with if AI is not enabled, no GC thrashing, memory usage at 150mb:

Only difference is:
services.AddApplicationInsightsTelemetry();

[ctolkien]

Digging in via perfview:

[ctolkien]

Reproduction is here:
https://github.com/ctolkien/ApplicationExplodes

[davidwengier]

I see the same thing on my machine with that repro.

The GCs are caused by LOH allocations, which in my case come from System.Diagnostics.PerformanceMonitor.PerformanceCounterLib.GetData(). The byte array coming back is huge:

(RegistryKey.cs also initializes a few more arrays that end up on the LOH on its way to getting one that big, which is compounding the problem)

item in the screenshot is "230" which is the Process counter.

[ScottHolden]

Spinning up a new ASP.Net Core WebAPI, and running under Kestral gives a full repo:

If we remove just the "\Process" related counters, we don't see this issue:

[xt0rted]

With the following changes (based on the discussion in https://github.com/microsoft/ApplicationInsights-aspnetcore/issues/912) I'm not seeing this go over 50 MB where as before these changes it was hitting 500 MB.

app.csproj

 <PropertyGroup>
   <TargetFramework>netcoreapp2.2</TargetFramework>
+  <ServerGarbageCollection>false</ServerGarbageCollection>
 </PropertyGroup>

runtimeconfig.template.json

{
  "configProperties": {
    "System.GC.RetainVM": false
  }
}

Changing each of the System.GC.Server and System.GC.RetainVM settings yields varying results, all of which are improvements, but setting both to false gives the best.

[ScottHolden]

I've also noted that it is not one particular counter, but in fact anything to do with \Process():

[TimothyMothra]

@cijothomas can you take a look and comment?

[cijothomas]

Thanks for reporting the issue with repros. Yes this is easy to repro, and reproes in all sdk versions, and in both .net core and .net framework apps. This is not a bug inside the sdk itself, but from the perfcounter library. Trying to find owners to get this tracked/fixed appropriately.

[ctolkien]

Hi @cijothomas - I recognise this isn't an issue in AppInsights directly, but net result is this is slowing our devs down, so we're yanking AI out.

[jeffadavidson]

@cijothomas do you any update on this?

[Dmitry-Matveev]

@cijothomas , assigned to you to check on the status of your conversation with the code owners.

[evilpilaf]

This looks exactly like the issue I'm facing and describe here
#1678 (comment)
Is there any ETA on it?

[cijothomas]

@evilpilaf This issue is arising from PerformanceCounter.
The other issue and the screenshot share appears from DiagnosticSource related. They are separate problems.

[SidShetye]

Seeing the same on .NET Core 3.1 and Microsoft.ApplicationInsights.AspNetCore v2.13.1.

If you don't need the AppInsight collected metrics you can disable that module in your startup.cs's ConfigureServices(IServiceCollection services)

services.AddApplicationInsightsTelemetry(opt => {
	opt.EnablePerformanceCounterCollectionModule = false;
});

Obviously this is a work-around till the real issue is resolved but it works for us since we're gathering similar metrics at our hosting layer.

[cijothomas]

This is fixed #1694 from 2.14.0-beta1 onwards.

[ojpbay]

@cijothomas - can you please confirm the exact version of the framework this fix went into. We have current production apps that suffer from similar issues to those mentioned by @evilpilaf and they are on runtime 3.1.112 (build agents are on SDK v3.1.404).