Excessive CPU and memory usage (70+ GB) when running analyzers

[filipnavara]

Version Used: .NET 9.0.6 / Roslyn 4.14.0-3.25279.5+995f12b60535afe75cafbcefada9161d8dc4f31f

Steps to Reproduce:

1. Open the attached project in Visual Studio, start a build
2. Observe that the VBCSCompiler process starts taking CPU time and eventually eats gigabytes of memory (70+ GB on my machine, then starts swapping and slows to crawl)
3. The compilation never finishes

Notably, disabling analyzers in MailClient.Emoji.csproj with <RunAnalyzers>false</RunAnalyzers> workarounds the issue. We didn't experience the issue with .NET 8 SDK when the projects were targeting net8.0 TFM.

Expected Behavior: The build finishes, doesn't consume all the memory.

Actual Behavior: Build doesn't finish and eats all my memory.

[jjonescz]

Thanks for taking the time to report this issue.

Can you capture a binary log with more details via -bl -p:ReportAnalyzer=true which adds analyzer timing information so we can determine which analyzer is the culprit? You should be able to see the results under $csc node in MSBuild binlog viewer. Ideally compare before (net8) and after (net9) and post the results here. Thanks.

[jjonescz]

... if that's reproducible on command-line builds. Or are you seeing this only in Visual Studio builds?

[filipnavara]

Can you capture a binary log with more details via -bl -p:ReportAnalyzer=true which adds analyzer timing information so we can determine which analyzer is the culprit?

I'll give it a try but since the build never finishes I don't think the (partial) log will contain it.

Or are you seeing this only in Visual Studio builds?

The initial observation was in Visual Studio but I suppose it will be reproducible elsewhere as well. I can use Project System Tools to log builds inside VS but since it hangs I don't think it will provide useful information.

I also managed to capture memory dump where you can see the stack trace... but it's 92 GB so not easy to share. That's why I extracted the repro project instead.

The stack trace itself in the dump was not enough to identify the culprit:

ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>.DeepCopy()
ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>.Union(ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>)
ILLink.Shared.DataFlow.ValueSetLattice<ILLink.Shared.DataFlow.SingleValue>.Meet(ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>)
ILLink.Shared.DataFlow.DictionaryLattice<ILLink.RoslynAnalyzer.DataFlow.LocalKey, ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.Shared.DataFlow.ValueSetLattice<ILLink.Shared.DataFlow.SingleValue>>.Meet(ILLink.Shared.DataFlow.DefaultValueDictionary<ILLink.RoslynAnalyzer.DataFlow.LocalKey, ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>>, ILLink.Shared.DataFlow.DefaultValueDictionary<ILLink.RoslynAnalyzer.DataFlow.LocalKey, ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>>)
ILLink.RoslynAnalyzer.DataFlow.LocalStateLattice<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.Shared.DataFlow.ValueSetLattice<ILLink.Shared.DataFlow.SingleValue>>.Meet(ILLink.RoslynAnalyzer.DataFlow.LocalState<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>>, ILLink.RoslynAnalyzer.DataFlow.LocalState<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>>)
ILLink.RoslynAnalyzer.DataFlow.LocalStateAndContextLattice<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContext, ILLink.Shared.DataFlow.ValueSetLattice<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContextLattice>.Meet(ILLink.RoslynAnalyzer.DataFlow.LocalStateAndContext<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContext>, ILLink.RoslynAnalyzer.DataFlow.LocalStateAndContext<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContext>)
[Native to Managed Transition]
[Managed to Native Transition]
ILLink.Shared.DataFlow.ForwardDataFlowAnalysis<ILLink.RoslynAnalyzer.DataFlow.LocalStateAndContext<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContext>, ILLink.RoslynAnalyzer.DataFlow.LocalDataFlowState<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContext, ILLink.Shared.DataFlow.ValueSetLattice<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContextLattice>, ILLink.RoslynAnalyzer.DataFlow.LocalStateAndContextLattice<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContext, ILLink.Shared.DataFlow.ValueSetLattice<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContextLattice>, ILLink.RoslynAnalyzer.DataFlow.BlockProxy, ILLink.RoslynAnalyzer.DataFlow.RegionProxy, ILLink.RoslynAnalyzer.DataFlow.ControlFlowGraphProxy, ILLink.RoslynAnalyzer.TrimAnalysis.TrimAnalysisVisitor, ILLink.RoslynAnalyzer.DataFlow.FeatureChecksValue>.Fixpoint(ILLink.RoslynAnalyzer.DataFlow.ControlFlowGraphProxy, ILLink.RoslynAnalyzer.TrimAnalysis.TrimAnalysisVisitor)
ILLink.RoslynAnalyzer.DataFlow.LocalDataFlowAnalysis<ILLink.Shared.DataFlow.ValueSet<System.__Canon>, ILLink.RoslynAnalyzer.DataFlow.FeatureContext, ILLink.Shared.DataFlow.ValueSetLattice<System.__Canon>, ILLink.RoslynAnalyzer.DataFlow.FeatureContextLattice, System.__Canon, ILLink.RoslynAnalyzer.DataFlow.FeatureChecksValue>.AnalyzeMethod(ILLink.RoslynAnalyzer.DataFlow.MethodBodyValue, ref ILLink.RoslynAnalyzer.DataFlow.InterproceduralState<ILLink.Shared.DataFlow.ValueSet<System.__Canon>, ILLink.Shared.DataFlow.ValueSetLattice<System.__Canon>>)
ILLink.RoslynAnalyzer.DataFlow.LocalDataFlowAnalysis<ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContext, ILLink.Shared.DataFlow.ValueSetLattice<ILLink.Shared.DataFlow.SingleValue>, ILLink.RoslynAnalyzer.DataFlow.FeatureContextLattice, ILLink.RoslynAnalyzer.TrimAnalysis.TrimAnalysisVisitor, ILLink.RoslynAnalyzer.DataFlow.FeatureChecksValue>.InterproceduralAnalyze()
ILLink.RoslynAnalyzer.DynamicallyAccessedMembersAnalyzer.Initialize.AnonymousMethod__1(Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalysisContext)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteOperationBlockActions.AnonymousMethod__56_4((Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction blockAction, Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalysisContext context))
Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteAndCatchIfThrows_NoLock<(System.__Canon, Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalysisContext)>(Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer, System.Action<(System.__Canon, Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalysisContext)>, (System.__Canon, Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalysisContext), Microsoft.CodeAnalysis.Diagnostics.AnalysisContextInfo?, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteAndCatchIfThrows<(Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction, Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalysisContext)>(Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer, System.Action<(Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction, Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalysisContext)>, (Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction, Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalysisContext), Microsoft.CodeAnalysis.Diagnostics.AnalysisContextInfo?, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteOperationBlockActions.AnonymousMethod__56_2(System.Collections.Generic.HashSet<Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction>, Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.AnalyzerDiagnosticReporter, System.Func<Microsoft.CodeAnalysis.Diagnostic, System.Threading.CancellationToken, bool>, Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecutionData, (Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor this, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction> startActions, Microsoft.CodeAnalysis.SyntaxNode declaredNode, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation> operationBlocks, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation> operations, Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder<Microsoft.CodeAnalysis.Diagnostics.OperationAnalyzerAction> ephemeralActions), System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteBlockActionsCore<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction, Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction, (Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction>, Microsoft.CodeAnalysis.SyntaxNode, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder<Microsoft.CodeAnalysis.Diagnostics.OperationAnalyzerAction>)>(System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction>, Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecutionData, System.Action<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction, System.Collections.Generic.HashSet<Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction>, Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecutionData, (Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction>, Microsoft.CodeAnalysis.SyntaxNode, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder<Microsoft.CodeAnalysis.Diagnostics.OperationAnalyzerAction>), System.Threading.CancellationToken>, System.Action<Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.AnalyzerDiagnosticReporter, System.Func<Microsoft.CodeAnalysis.Diagnostic, System.Threading.CancellationToken, bool>, Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecutionData, (Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction>, Microsoft.CodeAnalysis.SyntaxNode, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder<Microsoft.CodeAnalysis.Diagnostics.OperationAnalyzerAction>), System.Threading.CancellationToken>, System.Action<System.Collections.Generic.HashSet<Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction>, Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.AnalyzerDiagnosticReporter, System.Func<Microsoft.CodeAnalysis.Diagnostic, System.Threading.CancellationToken, bool>, Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecutionData, (Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction>, Microsoft.CodeAnalysis.SyntaxNode, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder<Microsoft.CodeAnalysis.Diagnostics.OperationAnalyzerAction>), System.Threading.CancellationToken>, (Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction>, Microsoft.CodeAnalysis.SyntaxNode, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder<Microsoft.CodeAnalysis.Diagnostics.OperationAnalyzerAction>), System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteOperationBlockActions(System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockStartAnalyzerAction>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.OperationBlockAnalyzerAction>, Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer, Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis.ISymbol, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, Microsoft.CodeAnalysis.SemanticModel, Microsoft.CodeAnalysis.Text.TextSpan?, bool, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver<Microsoft.CodeAnalysis.CSharp.SyntaxKind>.ExecuteDeclaringReferenceActions.__executeOperationsBlockActions|6(System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.IOperation>, Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder<Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver<Microsoft.CodeAnalysis.CSharp.SyntaxKind>.ExecutableCodeBlockAnalyzerActions>)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver<Microsoft.CodeAnalysis.CSharp.SyntaxKind>.ExecuteDeclaringReferenceActions.__executeExecutableCodeActions|2()
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver<Microsoft.CodeAnalysis.CSharp.SyntaxKind>.ExecuteDeclaringReferenceActions(Microsoft.CodeAnalysis.SyntaxReference, Microsoft.CodeAnalysis.Diagnostics.SymbolDeclaredCompilationEvent, Microsoft.CodeAnalysis.Diagnostics.AnalysisScope, Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver<Microsoft.CodeAnalysis.CSharp.SyntaxKind>.GroupedAnalyzerActions, Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver<Microsoft.CodeAnalysis.CSharp.SyntaxKind>.GroupedAnalyzerActions, bool, bool, bool, bool, bool, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver<Microsoft.CodeAnalysis.CSharp.SyntaxKind>.ExecuteDeclaringReferenceActions(Microsoft.CodeAnalysis.Diagnostics.SymbolDeclaredCompilationEvent, Microsoft.CodeAnalysis.Diagnostics.AnalysisScope, bool, Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.IGroupedAnalyzerActions, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.TryProcessSymbolDeclaredAsync(Microsoft.CodeAnalysis.Diagnostics.SymbolDeclaredCompilationEvent, Microsoft.CodeAnalysis.Diagnostics.AnalysisScope, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.TryProcessEventCoreAsync(Microsoft.CodeAnalysis.Diagnostics.CompilationEvent, Microsoft.CodeAnalysis.Diagnostics.AnalysisScope, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.ProcessEventAsync(Microsoft.CodeAnalysis.Diagnostics.CompilationEvent, Microsoft.CodeAnalysis.Diagnostics.AnalysisScope, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.ProcessCompilationEventsCoreAsync(Microsoft.CodeAnalysis.Diagnostics.AnalysisScope, bool, System.Threading.CancellationToken)
Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.ProcessCompilationEventsAsync.AnonymousMethod__2()
System.Threading.Tasks.Task<System.Threading.Tasks.Task<Microsoft.CodeAnalysis.Diagnostics.CompilationCompletedEvent>>.InnerInvoke()
System.Threading.Tasks.Task.Execute()
System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, object, bool)
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, object, bool)
System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task)
System.Threading.Tasks.Task.ExecuteEntry(bool)
System.Threading.ThreadPoolWorkQueue.Dispatch()

[Youssef1313]

That stack trace belongs to https://github.com/dotnet/runtime/blob/main/src/tools/illink/src/ILLink.RoslynAnalyzer/DynamicallyAccessedMembersAnalyzer.cs

[filipnavara]

It does reproduce in command line build but the truncated .binlog doesn't contain the information about the culprit. I let it run for a minute and all the times for generators and analyzers sum up to less than 1 second in total.

[filipnavara]

That stack trace belongs to https://github.com/dotnet/runtime/blob/main/src/tools/illink/src/ILLink.RoslynAnalyzer/DynamicallyAccessedMembersAnalyzer.cs

I didn't try to profile the compiler so I cannot say for certain it's the root cause. However, feel free to move the issue to dotnet/runtime if you think it's a problem in the analyzer. It's somewhat likely that the AOT analyzer would be the cause.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/illink
See info in area-owners.md if you want to be subscribed.

[jjonescz]

It might be possible to disable all the diagnostics belonging to that analyzer or manually remove it from Analyzer item group via some MSBuild code to verify whether it's that analyzer causing the issue.