Detecting .NET 10 Regression in startup performance affecting AWS Lambda

[normj]

Description

I'm testing .NET 10 vs .NET 8 in AWS Lambda to get ready for adding .NET 10 as a managed runtime in AWS Lambda. I know I should have done the performance testing months ago but always something getting in the way.

Breaking down the AWS Lambda environment into a more basic explanation I'm testing both .NET 8 and .NET 10 as self contained applications with no trimming enabled and R2R enabled. In Lambda you are working with a small slice of CPU power in the compute environment. The Lambda function itself is nothing more then a hello world. The .NET Lambda runtime client is started up and is largely a glorified HttpClient making calls to make calls communicating the status of processing an event. The HTTP calls are local going over IP addresses and not using HTTPS.

I'm seeing .NET 10 having a 5 to 10 percent regression in the first invoke where the .NET runtime has to be started.

In Lambda I can't attach a profiler but I added the following event listener to capture the .NET runtime events.

public class ConsoleListener : EventListener
{
    protected override void OnEventSourceCreated(EventSource eventSource)
    {
        // Attach to interesting sources
        if (eventSource.Name == "Microsoft-Windows-DotNETRuntime")
        {
            EnableEvents(eventSource, EventLevel.Informational,
                (EventKeywords)(-1)); // All keywords
        }
    }

    protected override void OnEventWritten(EventWrittenEventArgs e)
    {
        string value = string.Empty;
        if (e.Payload != null)
        {
            value = string.Join(", ", e.Payload);
        }
        Console.WriteLine($"{e.EventName}: {value}");
    }
}

Below are the links for the output of the cold start for both .NET 8 and 10 containing the .NET events.
.NET 8: https://github.com/normj/DotnetStartupPerformanceTest/blob/main/LambdaCaptures/dotnet8-metrics.txt
.NET 10: https://github.com/normj/DotnetStartupPerformanceTest/blob/main/LambdaCaptures/dotnet10-metrics.txt

I know data for this report is light. I'm working on collecting more information but given how close we are to .NET 10 release I didn't want to wait any longer to start a conversation. I'm kicking myself for not finding time to do this sooner. Maybe somebody on the .NET Team can see something important in the metrics files. From my analysis I see more .NET type being loaded, does this mean .NET 10 just does more at startup then .NET 8.

Regression?

This is a regression in cold start for .NET 8

Local Reproduction

In the README of my sample repository I have reproduction using containers which are built with the Lambda function and Lambda's Emulator. The instructions in the README give details on how to run it. Caveat I built the repo on a Windows Laptop and it is hard coded in the Dockerfile for X64 architecture.
https://github.com/normj/DotnetStartupPerformanceTest?tab=readme-ov-file#simulate-locally-with-lambda-emulator

Data

I have collected the metrics captures in the following repo
https://github.com/normj/DotnetStartupPerformanceTest/tree/main

I also added a console application that I explain in the README that I see some similar behavior when I run inside a container when using the --cpus=0.1 switch to simulate the slice of CPU Lambda would give. Without the switch .NET 10 is faster but with the switch .NET 8 is faster although the difference is not as much as I'm seeing in Lambda. The README in the repo gives the instructions on how I run it.

[EgorBo]

@normj can you test it on .NET 9 as well to narrow it down to one release? Does AWS Lambda use Server GC? if so, can you see if DOTNET_GCDynamicAdaptationMode=0 helps?

[normj]

@EgorBo I'll give .NET 9 a try. As for Server GC because my Lambda is set to low resources it has less then a CPU so Server GC is not enabled. I confirmed by updating my Lambda function calling System.Runtime.GCSettings.IsServerGC which returns `false.

[normj]

@EgorBo Here is the output of my test harness. This is the P90 of a 100 cold start invokes as a self contained Lambda function deployed to the generic Linux Lambda runtime instead of the .NET managed runtime. .NET 9 actually showed improvement over .NET 8. In this run I actually swapped down .NET 10 to preview 1 because I thought maybe I could isolate when the regression happen but it happened even at preview 1. When I was analyzing the .NET metrics differences between .NET 8 and 10 there were more loads from System.Runtime.Intrinsics. Is it possible anything with the preview 1 AVX10.2 support that could trigger this.

Total Duration .NET 8 P90 (ms): 680.3749999999999
Total Duration .NET 9 P90 (ms): 670.995
Total Duration .NET 10 P90 (ms): 730.268

[richlander]

@mangod9

[mangod9]

Thanks for reporting @normj. Does this also repro in a local lambda environment or only in cloud lambda invocations? If we can repro locally would be easier to diagnose.

[normj]

@mangod9 There isn't really a local Lambda environment. In the link repo I described how I tried to simulate the issue running a container with a heavily restricted CPU. I'm not a 100% sure it is an accurate reproduction of the issue in Lambda.

I had a chat with @EgorBo and I'm working on providing the event data with the setting set at verbose as opposed to informational which is what I did so far. I'm also trying to run the comparisons of runtimes on ARM to see if this is an X64 only issue.

[normj]

@EgorBo I added new log captures in the repo with the event listener set to verbose.

https://github.com/normj/DotnetStartupPerformanceTest/tree/main/LambdaCaptures

[JulieLeeMSFT]

Thanks for the logs. We are taking a look.