Application insights increases application memory usage

[alexiordan]

With Application Insights enabled my application uses up to 1.2 GB of RAM. With Application Insights disabled, the application uses around 200 MB. Please see the picture below.

Why?

It wouldn't be a problem, but I am get warnings about the used memory raising above the 80% threshold and that I may encounter errors because of that.
I have also other applications running in the same App Service Plan (S2): 3.5 GB RAM.

The application is a .Net Core Asp.Net AppService using SignalRService. It is configured with
<ServerGarbageCollection>false</ServerGarbageCollection> (see:
dotnet/aspnetcore#1976 (comment) )

The difference between the right and the left part of the red bar in the image are only the following 2 lines of code.

services.AddApplicationInsightsTelemetry();
services.AddApplicationInsightsTelemetryProcessor<NoSignalRRequestFilter>();

where NoSignalRRequestFilter class is as follows:

 public class NoSignalRRequestFilter : ITelemetryProcessor
    {
        private ITelemetryProcessor Next { get; set; }

        public NoSignalRRequestFilter(ITelemetryProcessor next)
        {
            this.Next = next;
        }

        public void Process(ITelemetry item)
        {
            var request = item as RequestTelemetry;

            if (request != null && request.Name != null)
                if (request.Name.Contains("hubName1") 
                    || request.Name.Contains("hubName2")
                    || request.Name.Contains("hubName3"))
                    return;

            // Send everything else:
            this.Next.Process(item);
        }
    }

Would someone, please, help me figure this out?

[lmolkova]

it would really help if you could tell the version and flavor of AppInsights SDK you have (nuget packages names and versions) and how you configured it.

[alexiordan]

• Microsoft.ApplicationInsights.AspNetCore 2.7.1
• Microsoft.ApplicationInsights.Web 2.10.0

[lmolkova]

you should not use both of them, please remove Web SDK.
Here is the instruction for ASP.NET Core apps: https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp-net-core

[alexiordan]

Thank you for you fast response! I will republish and let you know how it worked.

[alexiordan]

Below is a picture of memory use with packages as follows:

• Microsoft.ApplicationInsights.AspNetCore 2.7.1
• Microsoft.ApplicationInsights.Web 2.10.0

I have marked the milestones: the first vertical line is where I disabled the AI and the second is where I published with only the AspNetCore SDK.
You can see the memory is growing again.

UPDATE: Currently is above 550 MB
UPDATE2: Currently is above 700 MB

[Dmitry-Matveev]

@alexiordan , Application Insights SDK creates a lot of managed object per each telemetry item tracked (ConcurrentDictionaries to track properties, URIs and so on..) in addition to the telemetry item itself. Therefore, AI is known to pollute GC heap with unused objects that should be collected in due time.

In combination with Server Garbage Collection it leads to severe memory gain as Server-oriented GC allows memory to grow significantly before it kicks in to keep it at server acceptable level. If memory is a scarse resource on the machine, setting Server GC to false sounds like a good idea.

I'd expect with AI SDK and Server GC false that memory would still grow higher than without AI but should become stable / contained on a certain level due to more frequent GC (a.k.a. Desktop GC).

Another workaround to try is to invoke GC from code periodically, but that's significantly less appealing.

[alexiordan]

Thank you @Dmitry-Matveev , for your reply. Finally there is an explanation for this behavior.

Would you mind take a look at the following 3 questions.

1. So this seems to be an expected behavior. Right ?

As can be seen from the picture below - latest screenshot of app with AI enabled - memory seems to be kept below ~1.2 GB

2. Do you by any chance know if this is an expected limit?

I ask because I want to have AI enabled, but I also run multiple apps on the same App Service Plan (S2): 3.5 GB RAM.

3. If I scale up to next Plan (7GB) should I expect memory to grow even above 1.2 GB, given it has more room?

[cijothomas]

@Dmitry-Matveev can you respond to the questions above?

[Dmitry-Matveev]

So this seems to be an expected behavior. Right ?

Yes. Server GC will allow memory to grow significantly cleaning up rarely. Desktop GC will allow memory to grow to an extent while cleaning it up more often.

Do you by any chance know if this is an expected limit?

Can't really say as it depends on what Server GC "thinks" the right time to kick off is. The picture looks like 1.2GB is some kind of threshold for GC (e.g. 33% of avialable RAM) where it kicks off and bounces away to the state with no stale AI objects.

If I scale up to next Plan (7GB) should I expect memory to grow even above 1.2 GB, given it has more room?

I would expect GC may allow memory to grow further. If the assumption about 33% RAM is correct, I'd say GC will kick in at 2.5 GB in that case.

[alexiordan]

About 2 month ago I scaled-up to P2V2 (7GB of RAM) to see the behavior.
From the graphic below (covering last 30 days) I would say your estimation is correct. However it was a single time the application reached that amount of used memory due to external events (publishing etc.).
So I will still keep it under observation.

Are there documented anywhere those percentages?

[israellot]

@alexiordan
I' ve had success taming this problem by turning the system knob System.GC.RetainVM to false

If you take a memory dump and analyze, you will probably notice that the majority of the committed memory is unused overhead, pages of memory that runtime is holding for future use instead.
This knob forces the runtime to release those back to the OS instead of keeping them around.

https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/clr-configuration-knobs.md

[alexiordan]

Thank you, @israellot !
I'll set it right away !

[Marusyk]

Have the same issue #2140
Does anyone know how to use AI in production without memory leaks?

[github-actions]

This issue is stale because it has been open 300 days with no activity. Remove stale label or comment or this will be closed in 7 days.