Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Text.Json.SourceGeneration.Tests are disabled in WASM #59228

Closed
layomia opened this issue Sep 16, 2021 · 10 comments · Fixed by #59396
Closed

System.Text.Json.SourceGeneration.Tests are disabled in WASM #59228

layomia opened this issue Sep 16, 2021 · 10 comments · Fixed by #59396
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono disabled-test The test is disabled in source code against the issue
Milestone
7.0.0

Comments

@layomia
Copy link
Contributor

layomia commented Sep 16, 2021

The System.Text.Json.SourceGeneration.Roslyn.x.tests appear to be timing out during the AOT compilation phase. From PR #59042:

https://helix.dot.net/api/2019-06-17/jobs/770ad140-31bb-4858-9396-d50162076efd/workitems/System.Text.Json.SourceGeneration.Roslyn3.11.Tests/console

  [39/47] xunit.runner.utility.netcoreapp10.dll.bc -> xunit.runner.utility.netcoreapp10.dll.o [took 20.326s]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : Failed to compile /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc -> /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.o [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error :  "/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc -o /tmp/tmpkKeaiw.tmp [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : clang version 13.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project 5852582532b3eb3ea8da51a1e272d8d017bd36c9) [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : Target: wasm32-unknown-emscripten [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : Thread model: posix [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : InstalledDir: /datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/bin [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error :  (in-process) [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error :  "/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/bin/clang++" -cc1 -triple wasm32-unknown-emscripten -emit-obj --mrelax-relocations -disable-free -main-file-name System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc -mrelocation-model static -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -target-cpu generic -fvisibility hidden -debug-info-kind=limited -dwarf-version=4 -debugger-tuning=gdb -v -fcoverage-compilation-dir=/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish -resource-dir /datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/lib/clang/13.0.0 -Oz -Werror=implicit-function-declaration -fdebug-compilation-dir=/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish -ferror-limit 19 -fgnuc-version=4.2.1 -fno-inline-functions -vectorize-slp -iwithsysroot/include/SDL -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/tmpkKeaiw.tmp -x ir /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : clang -cc1 version 13.0.0 based upon LLVM 13.0.0git default target x86_64-unknown-linux-gnu [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : emcc: error: '/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc -o /tmp/tmpkKeaiw.tmp' failed (received SIGKILL (-9)) [took 433.767s] [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
XHarness artifacts: /datadisks/disk1/work/9D6C08B7/w/AA0A0963/uploads/xharness-output

In that same PR I'm disabling the tests to unblock green CI.

cc @eerhardt @lewing @radical @SamMonoRT @vargaz
@layomia layomia added arch-wasm WebAssembly architecture area-System.Text.Json disabled-test The test is disabled in source code against the issue labels Sep 16, 2021
@layomia layomia added this to the 7.0.0 milestone Sep 16, 2021
@ghost
Copy link

ghost commented Sep 16, 2021

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

The System.Text.Json.SourceGeneration.Roslyn.x.tests appear to be timing out during the AOT compilation phase. From PR #59042:

https://helix.dot.net/api/2019-06-17/jobs/770ad140-31bb-4858-9396-d50162076efd/workitems/System.Text.Json.SourceGeneration.Roslyn3.11.Tests/console

  [39/47] xunit.runner.utility.netcoreapp10.dll.bc -> xunit.runner.utility.netcoreapp10.dll.o [took 20.326s]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : Failed to compile /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc -> /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.o [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error :  "/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc -o /tmp/tmpkKeaiw.tmp [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : clang version 13.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project 5852582532b3eb3ea8da51a1e272d8d017bd36c9) [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : Target: wasm32-unknown-emscripten [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : Thread model: posix [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : InstalledDir: /datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/bin [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error :  (in-process) [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error :  "/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/bin/clang++" -cc1 -triple wasm32-unknown-emscripten -emit-obj --mrelax-relocations -disable-free -main-file-name System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc -mrelocation-model static -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -target-cpu generic -fvisibility hidden -debug-info-kind=limited -dwarf-version=4 -debugger-tuning=gdb -v -fcoverage-compilation-dir=/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish -resource-dir /datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/lib/clang/13.0.0 -Oz -Werror=implicit-function-declaration -fdebug-compilation-dir=/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish -ferror-limit 19 -fgnuc-version=4.2.1 -fno-inline-functions -vectorize-slp -iwithsysroot/include/SDL -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/tmpkKeaiw.tmp -x ir /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : clang -cc1 version 13.0.0 based upon LLVM 13.0.0git default target x86_64-unknown-linux-gnu [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/9D6C08B7/p/build/wasm/WasmApp.Native.targets(293,5): error : emcc: error: '/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/9D6C08B7/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/wasm_build/obj/wasm/System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll.bc -o /tmp/tmpkKeaiw.tmp' failed (received SIGKILL (-9)) [took 433.767s] [/datadisks/disk1/work/9D6C08B7/w/AA0A0963/e/publish/ProxyProjectForAOTOnHelix.proj]
XHarness artifacts: /datadisks/disk1/work/9D6C08B7/w/AA0A0963/uploads/xharness-output

In that same PR I'm disabling the tests to unblock green CI.

cc @eerhardt @lewing @radical @SamMonoRT @vargaz

Author: layomia
Assignees: -
Labels:

arch-wasm, area-System.Text.Json, disabled-test
Milestone: 7.0.0

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Sep 16, 2021
@layomia layomia mentioned this issue Sep 16, 2021
Merged
@danmoseley
Copy link
Member

@steveisok is already tracking this one. Curious how it got in actually.

@danmoseley danmoseley modified the milestones: 7.0.0, 6.0.0 Sep 17, 2021
@lewing
Copy link
Member

lewing commented Sep 17, 2021

@danmoseley pretty sure this is the host killing the aot compiler for some reason, probably OOM in this case.

@radical @steveisok we can try turning down the parallelism on the json tests as a first step.

@danmoseley
Copy link
Member

Oops, mixed this up with the unrelated Android issue triggered by the digit in the name Roslyn3.11.Tests

@lewing lewing removed the untriaged New issue has not been triaged by the area owner label Sep 17, 2021
@lewing lewing modified the milestones: 6.0.0, 7.0.0 Sep 17, 2021
@radical
Copy link
Member

radical commented Sep 20, 2021

@radical @steveisok we can try turning down the parallelism on the json tests as a first step.

Running it serially didn't help. It is still getting killed in the same way.

@radical
Copy link
Member

radical commented Sep 20, 2021 •
edited
Loading

OOM:

python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

...
[  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[  24770]  1000 24770    14474     2510   155648        0             0 python3
[  24775]  1000 24775  1906650  1837676 15310848        0             0 clang++

...

out of memory: Killed process 24775 (clang++) total-vm:7626600kB, anon-rss:7350704kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14952kB oom_score_adj:0

https://gist.github.com/radical/3c45276841d3c4e2d2cc91d5069a4d7a

@danmoseley
Copy link
Member

It would be invaluable if there was logging such that we knew what tests were in flight. There is likely one or two tests that are memory heavy and just need disabling for WASM.

I'm guessing there is no test results file already written when it's killed?

Cc @dotnet/area-system-text-json

@radical
Copy link
Member

radical commented Sep 20, 2021

The tests haven't run at this point. We do the AOT part of the build on helix, because it's very slow. And this is getting killed while compiling one of the assemblies (System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll).

@vargaz
Copy link
Contributor

vargaz commented Sep 21, 2021

This might be fixed by:
#59352
which causes input bitcode to be smaller, so emcc runs faster/uses less memory.

@danmoseley
Copy link
Member

@radical ah, I missed that.

radical added a commit to radical/runtime that referenced this issue Sep 21, 2021
@radical
[wasm] Enable System.Text.Json.SourceGeneration.Tests
b4f830b 
Fixed by dotnet#59352 .

Fixes dotnet#59228 .
@radical radical mentioned this issue Sep 21, 2021
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Sep 21, 2021
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Sep 22, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Nov 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono disabled-test The test is disabled in source code against the issue
Projects
None yet
Milestone
7.0.0
Development

Successfully merging a pull request may close this issue.

[wasm] Enable System.Text.Json.SourceGeneration.Tests radical/runtime
5 participants
@radical @lewing @vargaz @danmoseley @layomia