Compiling application with LanguageExt as a dependency and ReadyToRun enabled using .NET 6 #66079
Comments
dotnet-issue-labeler
bot
added
the
untriaged
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
|
I ran into the same or at least a similar issue today. After waiting for 3hours for compilation of my (rather simple) functions project to end I was presented with the following error message. Note that while I waited the memory usage crew continuously. Crossgen2 always used at least 1 CPU core to a 100%. I didn't check earlier, but after about 2 hours the disk usage also was consistently at about 60-80% (at that time memory usage was at 22Gib and there basically wasn't any memory left, so I assume disk usage might have been caused by swapping). Also I tried running this via github action on ubuntu-linux, but I stopped it after 1.5 hours.
Note: '....\Microsoft.NET.CrossGen.targets' originally was 'C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.CrossGen.targets' |
|
As recommended by the As a result, the build now seems to hang indefinitely. Crossgen2 is not running, cpu is idle, memory usage remains low. Using Which just gets repeated, for ever. |
|
Trimming I've updated my reproduction case, in summary: <!-- I've added this section to the `AndreSteenveld.CrossgrenLangueageExt6.csproj` file, it was taken from: https://github.com/louthy/language-ext/issues/673#issuecomment-778358899 -->
<Target Name="ConfigureTrimming" BeforeTargets="PrepareForILLink">
<ItemGroup>
<ManagedAssemblyToLink Condition="'%(Filename)' == 'LanguageExt.Core'">
<IsTrimmable>true</IsTrimmable>
<TrimMode>link</TrimMode>
</ManagedAssemblyToLink>
</ItemGroup>
</Target>$ time dotnet publish ./AndreSteenveld.CrossgenLanguageExt6.csproj \
--configuration Release \
--runtime win-x64 \
--self-contained true \
-p:PublishSingleFile=true \
-p:PublishReadyToRun=true \
-p:PublishReadyToRunShowWarnings=true \
-p:PublishTrimmed=true
Microsoft (R) Build Engine version 17.1.0+ae57d105c for .NET
Copyright (C) Microsoft Corporation. All rights reserved.
Determining projects to restore...
Restored C:\Users\asteenveld\source\repos\AndreSteenveld.CrossgenLanguageExt\AndreSteenveld.CrossgenLanguageExt6.csproj (in 15.47 sec).
AndreSteenveld.CrossgenLanguageExt6 -> C:\Users\asteenveld\source\repos\AndreSteenveld.CrossgenLanguageExt\bin\Release\net6.0\win-x64\AndreSteenveld.CrossgenLanguageExt6.dll
C:\Users\asteenveld\.nuget\packages\languageext.core\4.0.3\lib\netstandard2.0\LanguageExt.Core.dll : warning IL2104: Assembly 'LanguageExt.Core' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [C:\Users\asteenveld\source\repos\AndreSteenveld.CrossgenLanguageExt\AndreSteenveld.CrossgenLanguageExt6.csproj]
Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
AndreSteenveld.CrossgenLanguageExt6 -> C:\Users\asteenveld\source\repos\AndreSteenveld.CrossgenLanguageExt\bin\Release\net6.0\win-x64\publish\
real 1m35.580s
user 0m0.000s
sys 0m0.093s |
|
While debugging this issue for somebody trying to run this package with ReadyToRun on AWS I noticed that when you can get the crossgen2 compiler to complete it turns the 11 Meg LanguageExt.Core.dll into a 270+ Meg dll. This library uses a lot of generics and Reflection.Emit over generic types. I wonder if the crossgen2 is getting into some overaggressive mode trying to generate ReadyToRun code for all of the possible types. |
|
I have managed to repro this locally. The problem is indeed caused by generic explosion, the proximate bug the debug compiler build hits is an integer overflow when resizing the generic unification hashtable. As the next experimental step I'm thinking about introducing a heuristic "generic complexity" and using it to trim too complex instantiations; I'm also looking into ways to somehow record the increasing generic complexity as part of the dependency analysis as a tool for understanding what triggers the infinite instantiation loop. |
ghost
added
the
in-pr
|
Moving to 8 due to the risk associated with the fix. |
ghost
removed
the
in-pr
ghost
added
the
in-pr
This change modifies Crossgen2 to use the generic cycle detector originally implemented for NativeAOT to trim infinite generic expansion as observed in the LanguageExt public nuget package and tracked by the issue dotnet#66079 At this point I have only implemented a back-compat option to opt out of the generic cycle detection in case it turns out to cause problems for certain payloads. I haven't added the option to set the cutoff level - in practice, LanguageExt compilation still fails when the cutoff is set to anything higher than 0; moreover for Crossgen2 skipping precompilation of certain methods merely incurs slightly reduced performance, not runtime failures as is the case with NativeAOT. Thanks Tomas Rebase against latest main Temporary instrumentation to track dependency level Improve dependency level instrumentation Instrumentation for dumping dropped cycles Add more places for cycle detection More instrumentation Log generic cycles for each module
So I have a trivial depth cutoff test working; without enabling generic cycle detection Crossgen2 crashes after about an hour on my box with the Arithmetic Overflow as described in the issue dotnet#66079. As next step I'll work on the equivalent breadth test, that is somewhat more tricky. Thanks Tomas
This change modifies Crossgen2 to use the generic cycle detector originally implemented for NativeAOT to trim infinite generic expansion as observed in the LanguageExt public nuget package and tracked by the issue #66079 For now the generic cycle detector is opt-in as it's relatively costly in terms of compiler performance and it seems to be a relatively niche case. We can follow up by optimizing the detector if it turns out to be more prevalent or if we make the call to turn it on by default, I have mentioned several options for that in the PR. Thanks Tomas
ghost
removed
the
in-pr
|
FYI, I have merged in Crossgen2 support for handling this situation cleanly. One caveat is that for now we decided to make the new switch an opt-in because this issue is the first one we discovered hitting this problem and the implementation makes the compilation slower. It should ship as part of .NET 8 Preview 6 and it would be naturally great if @AndreSteenveld and / or some of the other contributors to this thread could verify that it fixes their issues. The relevant option that needs to be put in the project file is as follows: <PublishReadyToRunCrossgen2ExtraArgs>--enable-generic-cycle-detection</PublishReadyToRunCrossgen2ExtraArgs> Thanks in advance! P.S. The change implements two additional switches that control the internal heuristic, |
As part of investigation of the bug dotnet#66079 and implementation of the fix dotnet#71426 I noticed that one aggravating factor is that Crossgen2 starts analyzing the same assembly multiple times on its various parallel threads; this busywork additionally makes the app compete for access to the same metadata, making the initial analysis even slower. In accordance with Michal's suggestion from the PR thread dotnet#71426 I propose to modify the generic cycle detector to run the initial analysis single-threaded if the module in question is expected to be "generics-heavy" using the number of TypeSpec rows in its ECMA metadata as an indicator of module generic complexity. I have written a simple managed app scanning all runtime framework assemblies, ASP.NET assemblies and assemblies used in internal CoreCLR testing. I found out that the largest number of TypeSpec rows is in FSharp.Core (3855), followed by Microsoft.CodeAnalysis (3148). Based on these findings I have set the initial value of the cutoff for single-threaded analysis to 5000. Thanks Tomas
ghost
locked as resolved and limited conversation to collaborators
Description
When compiling an application with .NET 5 and ReadyToRun enabled a binary is produced, trying to do the same with .NET 6 just "hangs". If a publish profile is used and
-p:PublishReadyToRunCrossgen2ExtraArgs='--verbose'is provided it looks like crossgen2 is really busy processing all possible combinations of all possible generic classes. (That is more speculation from my side than anything else though)Reproduction Steps
I've built a small application which can be found here; AndreSteenveld.CrossgenLanguageExt. Summarizing the readme; Running
dotnet publish ./AndreSteenveld.CrossgenLanguageExt5.csproj -p:PublishSingleFile=true -p:PublishReadyToRun=true --configuration Release --runtime win-x64 --self-contained trueproduces a binary as expected. But doing the same for a .NET 6 project doesn't.Expected behavior
I'm not expecting any super fast compilation, but nevertheless I do expect it to finish some time.
Actual behavior
Compile doesn't finish, even after several hours.
Regression?
I've read in the release notes that crossgen2 was released with .NET 6, so I'm assuming there is another (older) version of crossgen for .NET 5. So that would be my first suspect.
Known Workarounds
No response
Configuration
Running
dotnet --infooutputs the following:My compile target is also windows, x86-64. I do want to try to reproduce this on linux if possible, although I'm not expecting that it does compile (within a reasonable timeframe) on linux either.
Other information
As this is an issue which seems related to
LanguageExt.CoreI've also opened a ticket over there. louthy/language-ext#1000 If there is anything else I can do to help please feel free to reach out :)The text was updated successfully, but these errors were encountered: