[AOT publish error] EXEC : error : Arithmetic operation resulted in an overflow.

[glen-nicol]

Description

I created a new AOT minimal API project using the template available in VS22 and then added just a couple routes that reference a subsection of a mature code base. This builds and runs from debugger. When I try to publish using dotnet publish project.csproj -r win-x64 -c Release I get an internal compiler error below. It looks related and very similar to #66787.

This project does use LanguageExt which does use a ton of generics for the functional monads. But I don't think there is a circular generic cycle as mentioned by Jkotas.

          This failure mode typically means that the project libraries contain exponential generic explosion or infinite generic cycle. Having more than 2GB array elements won't solve that problem.

Originally posted by @jkotas in #66787 (comment)

Any tips on tools or automated methods for searching for a cycle? Ideally the error message from compiler would recognize problem and say its not supported if that is truly the case.

Reproduction Steps

Cannot provide example at this moment as my specific case involves proprietary code. But I imagine heavy use of generics from LanguageExt library is related.

Expected behavior

Native assembly is generated from publish step or an error indicating generic cycle was detected if that is the true cause that cannot be supported.

Actual behavior

System.OverflowException: Arithmetic operation resulted in an overflow.
at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowOverflowException() + 0x24
at Internal.TypeSystem.LockFreeReaderHashtable2.Expand(TValue[]) + 0x3b at Internal.TypeSystem.LockFreeReaderHashtable2.AddOrGetExistingInner(TValue, Boolean&) + 0x52
at Internal.TypeSystem.LockFreeReaderHashtable2.CreateValueAndEnsureValueIsInTable(TKey) + 0x26 at Internal.TypeSystem.InstantiatedType.<GetMethods>d__22.MoveNext() + 0x134 at ILCompiler.UsageBasedMetadataManager.GetConditionalDependenciesDueToEETypePresence(List1&, NodeFactory, TypeDesc) + 0x853
at ILCompiler.DependencyAnalysis.EETypeNode.GetConditionalStaticDependencies(NodeFactory) + 0x189c
at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.GetStaticDependenciesImpl(DependencyNodeCore1) + 0xf3
at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.GetStaticDependencies(DependencyNodeCore1) + 0x2b
at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ProcessMarkStack() + 0xb5 at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ComputeMarkedNodes() + 0x50
at ILCompiler.ILScanner.ILCompiler.IILScanner.Scan() + 0x14
at ILCompiler.Program.g__RunScanner|4_0(Program.<>c__DisplayClass4_0&) + 0x191
at ILCompiler.Program.Run() + 0x2444
at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass221_0.<.ctor>b__0(ParseResult result) + 0x315

Regression?

No response

Known Workarounds

No response

Configuration

SDK: 8.0.101 64 bit
Windows 10 22H2 64 bit
TargetFramework: 8.0-windows

Other information

No response

[ghost]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I created a new AOT minimal API project using the template available in VS22 and then added just a couple routes that reference a subsection of a mature code base. This builds and runs from debugger. When I try to publish using dotnet publish project.csproj -r win-x64 -c Release I get an internal compiler error below. It looks related and very similar to #66787.

This project does use LanguageExt which does use a ton of generics for the functional monads. But I don't think there is a circular generic cycle as mentioned by Jkotas.

          This failure mode typically means that the project libraries contain exponential generic explosion or infinite generic cycle. Having more than 2GB array elements won't solve that problem.

Originally posted by @jkotas in #66787 (comment)

Any tips on tools or automated methods for searching for a cycle? Ideally the error message from compiler would recognize problem and say its not supported if that is truly the case.

Reproduction Steps

Cannot provide example at this moment as my specific case involves proprietary code. But I imagine heavy use of generics from LanguageExt library is related.

Expected behavior

Native assembly is generated from publish step or an error indicating generic cycle was detected if that is the true cause that cannot be supported.

Actual behavior

System.OverflowException: Arithmetic operation resulted in an overflow.
at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowOverflowException() + 0x24
at Internal.TypeSystem.LockFreeReaderHashtable2.Expand(TValue[]) + 0x3b at Internal.TypeSystem.LockFreeReaderHashtable2.AddOrGetExistingInner(TValue, Boolean&) + 0x52
at Internal.TypeSystem.LockFreeReaderHashtable2.CreateValueAndEnsureValueIsInTable(TKey) + 0x26 at Internal.TypeSystem.InstantiatedType.<GetMethods>d__22.MoveNext() + 0x134 at ILCompiler.UsageBasedMetadataManager.GetConditionalDependenciesDueToEETypePresence(List1&, NodeFactory, TypeDesc) + 0x853
at ILCompiler.DependencyAnalysis.EETypeNode.GetConditionalStaticDependencies(NodeFactory) + 0x189c
at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.GetStaticDependenciesImpl(DependencyNodeCore1) + 0xf3
at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.GetStaticDependencies(DependencyNodeCore1) + 0x2b
at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ProcessMarkStack() + 0xb5 at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ComputeMarkedNodes() + 0x50
at ILCompiler.ILScanner.ILCompiler.IILScanner.Scan() + 0x14
at ILCompiler.Program.g__RunScanner|4_0(Program.<>c__DisplayClass4_0&) + 0x191
at ILCompiler.Program.Run() + 0x2444
at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass221_0.<.ctor>b__0(ParseResult result) + 0x315

Regression?

No response

Known Workarounds

No response

Configuration

SDK: 8.0.101 64 bit
Windows 10 22H2 64 bit
TargetFramework: 8.0-windows

Other information

No response

Author:	glen-nicol
Assignees:	-
Labels:	area-NativeAOT-coreclr
Milestone:	-

[MichalStrehovsky]

Cannot provide example at this moment as my specific case involves proprietary code.

Would it be possible to at least share the *.ilc.rsp file (you can find it under obj directory after publishing? I want to see if there's some trimming misconfiguration. You can censor out file paths or sensitive assembly names if there are any (replace them with Asm1/Asm2/...).

This project does use LanguageExt which does use a ton of generics for the functional monads. But I don't think there is a circular generic cycle as mentioned by Jkotas.

LanguageExt is known to have problems with AOT in .NET:

• For Mono-based AOT, it just doesn't work: WASM AOT compilation runs indefinitely and consumes all system memory with LanguageExt.Core #69166
• For PublishReadyToRun with CoreCLR, a feature was added to just abort ahead of time compiling and fall back to JIT because it couldn't compile in the past due to generic cycles: Compiling application with LanguageExt as a dependency and ReadyToRun enabled using .NET 6 #66079

Don't know the status with PublishAot (this is the first report), but I'm not optimistic based on the above. Some libraries simply won't work with AOT because they're built around having a JIT and being able to compile unbounded numbers of native method bodies.

Ideally the error message from compiler would recognize problem and say its not supported if that is truly the case.

Usually there are IL3054 warnings for this. Are there none in the output before the crash? Could be that this is so pathologic that we ran out of memory before cutoffs are hit.

[glen-nicol]

Sure thing Michal,

Here is ilc.rsp (.txt added for github). I noticed at the end of it it seems like multiple nosinglewarnassembly options didn't include the full name of the assembly and instead removed the last bit followed by a dot. Most likely assuming it was a file extension? But as you can see from some of the assemblies I didn't redact many of my projects start with Domain. or IO..

I did see warnings about AOT analysis and trimming. But I admit I kinda forgot about them when the error looked like an internal error and didn't indicate immediate relation to the warnings. here is a snippet from the output.

Generating native code
C:\Users\nicolg\.nuget\packages\languageext.core\4.4.3\lib\netstandard2.0\LanguageExt.Core.dll : warning IL3053: Assembly 'LanguageExt.Core' produced AOT analysis warnings. [C:\git\...]
C:\Users\nicolg\.nuget\packages\newtonsoft.json\13.0.1\lib\netstandard2.0\Newtonsoft.Json.dll : warning IL2104: Assembly 'Newtonsoft.Json' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [C:\git\...]
C:\Users\nicolg\.nuget\packages\languageext.core\4.4.3\lib\netstandard2.0\LanguageExt.Core.dll : warning IL2104: Assembly 'LanguageExt.Core' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [C:\git\...]
C:\Users\nicolg\.nuget\packages\system.reactive\5.0.0\lib\net5.0\System.Reactive.dll : warning IL2104: Assembly 'System.Reactive' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [C:\git\...]
EXEC : error : Arithmetic operation resulted in an overflow. [C:\...] ....

I will look at the language ext issues more closely. But this does have me curious, does F# have similar problems with AOT with all its generics?

[glen-nicol]

I also just realized I never reported this part of the error (sorry!) but now seems relavant to ilc.rsp files. This is the very last line after the stack trace.

C:\Users\nicolg\.nuget\packages\microsoft.dotnet.ilcompiler\8.0.1\build\Microsoft.NETCore.Native.targets(305,5): error MSB3073: The command ""C:\Users\nicolg\.nuget\packages\runtime.win-x64.microsoft.dotnet.ilcompiler\8.0.1\tools\\ilc" @"obj\Release\net8.0-windows\win-x64\native\TopLevelAssembly.ilc.rsp"" exited with code 1. [C:\git\repo\subdir\TopLevelAssembly.csproj]

[MichalStrehovsky]

Thanks! It seems like the project is configured correctly (I wanted to double check it's not the same issue as #66787).

It really just results in a extremely big native compilation. If the compiler didn't fail here, it would probably still compile forever because we're taking hundreds of millions of methods and it was not done yet when it hit limits. Typical larger app will have tens of thousands of methods.

[MichalStrehovsky]

I think this is a generic cycle that is similar to the one seen in #99409.

You should be able to at least get more information by adding <ItemGroup><IlcArg Include="--maxgenericcycle:1" /></ItemGroup> to the project file.

I expect you'll see AOT warnings. If you add <PropertyGroup><TrimmerSingleWarn>false</TrimmerSingleWarn></PropertyGroup> you should get more details.

If there is a cycle, there's not much we can do.

[glen-nicol]

Thanks for the tip Michal, unfortunately we have since pivoted and I won't be able to go back to try it out. But we still use C# a lot and I do use LanguageExt plenty so its possible we will hit this again and will need to look that argument to diagnose. Until then I will close this.