-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[wasm][aot] Link step failing for Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests
- wasm-opt
getting killed
#51961
Comments
Tagging subscribers to 'arch-wasm': @lewing Issue Details
The binlog should have details of why this failed. The binlog is included in the helix data.
|
This probably happens because the code is just too big. |
The current failure:
|
wasm-opt
fails for System.Text.Json.Tests
wasm-ld
fails for System.Text.Json.Tests
This should be fixed by #53603 and the flto changes |
I think this is still a problem. Hit this in release/6.0-rc2:
CC @layomia |
Also likely causing issue for new logging generator tests. See #59092 (comment) |
That is failing when compiling |
wasm-ld
fails for System.Text.Json.Tests
System.Text.Json.Tests
- wasm-opt
getting killed
Yes to help determine what is causing the failure, there were a few temporary commits where the the new functional code and tests were no longer called, but the CI issue remained. |
It might be the linker running out of memory again? Would disabling optimizations in linker |
This will get fixed by #59397 , same as it did on |
I see this
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-60112-merge-1c763c950edf44f2b0/System.Text.Json.Tests/1/console.0ff5206b.log?sv=2019-07-07&se=2021-10-27T13%3A48%3A50Z&sr=c&sp=rl&sig=Vlj0U9Lm4w1sYqrLDYGVsnvTmUldx6%2B0sr2PWre4tyI%3D this ran on |
In some cases they fail when compiling an assembly's bitcode to wasm I plan to:
|
- `Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests` - `Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests` - `System.Text.Json.SourceGeneration.Roslyn3.11.Unit.Tests` - `System.Text.Json.SourceGeneration.Roslyn4.0.Unit.Tests` - `System.Text.RegularExpressions.Generators.Tests` Some individual ones are disabled due to dotnet#58226, and dotnet#60899 . Fixes dotnet#51961 .
This adds support for setting per-project optimization flags, and sets them for these projects so they don't OOM. Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests System.Text.Json.SourceGeneration.Roslyn3.11.Unit.Tests System.Text.Json.SourceGeneration.Roslyn4.0.Unit.Tests System.Text.RegularExpressions.Generators.Tests Some individual ones are disabled due to #58226, and #60899 . Additionally, if a AOT build fails, then on linux it dumps the last few lines from dmesg, to help identify it was an oom-kill. Fixes #51961 . Co-authored-by: Larry Ewing <lewing@microsoft.com>
reopenning since we've re-disabled tests |
System.Text.Json.Tests
- wasm-opt
getting killedMicrosoft.Extensions.Logging.Generators.Roslyn3.11.Tests
- wasm-opt
getting killed
@ilyas1974 we are running into this over and over, what can we do to get more memory on the VM. |
Struggling to repro this locally. 😰 |
Oh no, I can reproduce it locally, and learned something very cursed about Docker |
Okay. Findings:
|
The problem is AOTing roslyn, a linked Microsoft.CodeAnalysis.CSharp.dll still contains 30k methods, AOT creates 80k methods out of this because roslyn contains a lot of valuetypes causing a lot of generic instances to be added, running the aot compiler on this assembly alone takes 40 seconds, the .bc -> .o step takes 10 minutes (!). |
We might want to add some functionality to the build system to exclude certain assemblies like roslyn from AOTing. |
We do have that, as a hack (https://github.com/dotnet/runtime/blob/main/src/tasks/AotCompilerTask/MonoAOTCompiler.cs#L488-L489). |
So what's the play, try to shoehorn the test as-is onto Helix, or try to wire up the AOT skip? |
I'd suggest skipping AOTing for these assemblies, they take forever to AOT which also impacts CI performance even if it works. |
Failure is at runtime/src/mono/mono/mini/aot-runtime.c Lines 2300 to 2303 in 9c8b804
|
Is there a PR which skips the AOT step for these assemblies ? |
An alternative would be compiling with O1, i.e.:
Running with this reduced the total runtime from 16 minutes to 11 minutes on my machine, it might solve the OOM problem as well. |
@directhex @radical - the latest rolling build (which has #63958) still hit this. Still looks to be an OOM, but I don't know how to get more information about the exact cause.
|
System.Text.Json.TestsMicrosoft.Extensions.Logging.Generators.Roslyn3.11.Tests, log
The binlog should have details of why this failed. The binlog is included in the helix data.
The text was updated successfully, but these errors were encountered: