[wasm][aot] RangeError: Maximum call stack size exceeded in System.Text.RegularExpressions.Tests

[radical]

Seen on main rolling build:
Build
Log

[21:39:13] fail: RangeError: Maximum call stack size exceeded
[21:39:13] fail: RangeError: Maximum call stack size exceeded
[21:39:13] fail:     at jit_info_table_index (<anonymous>:wasm-function[121904]:0x18e0147)
[21:39:13] fail:     at jit_info_table_find (<anonymous>:wasm-function[121903]:0x18e0078)
[21:39:13] fail:     at mono_jit_info_table_find_internal (<anonymous>:wasm-function[121902]:0x18dffb9)
[21:39:13] fail:     at mini_jit_info_table_find_ext (<anonymous>:wasm-function[134483]:0x1a4b0f0)
[21:39:13] fail:     at mono_find_jit_info_ext (<anonymous>:wasm-function[134482]:0x1a4af06)
[21:39:13] fail:     at unwinder_unwind_frame (<anonymous>:wasm-function[134500]:0x1a4b969)
[21:39:13] fail:     at mono_walk_stack_full (<anonymous>:wasm-function[134464]:0x1a4a85e)
[21:39:13] fail:     at mono_walk_stack_with_ctx (<anonymous>:wasm-function[134462]:0x1a4a6ec)
[21:39:13] fail:     at mono_runtime_walk_stack_with_ctx (<anonymous>:wasm-function[134448]:0x1a4a462)
[21:39:13] fail:     at mono_stack_walk (<anonymous>:wasm-function[121987]:0x18e30ab)
[21:39:13] fail:     at is_running_protected_wrapper (<anonymous>:wasm-function[123237]:0x1901809)
[21:39:13] fail:     at mono_thread_interruption_checkpoint_request (<anonymous>:wasm-function[123251]:0x1901b4a)
[21:39:13] fail:     at mono_thread_interruption_checkpoint (<anonymous>:wasm-function[123250]:0x1901b1c)
[21:39:13] fail:     at aot_wrapper_icall_mono_thread_interruption_checkpoint (<anonymous>:wasm-function[66920]:0xf05864)
[21:39:13] fail:     at aot_wrapper_corlib_System_dot_Runtime_dot_CompilerServices_System_dot_Runtime_dot_CompilerServices_dot_RuntimeHelpers__SufficientExecutionStack_pinvoke_bool_bool_ (<anonymous>:wasm-function[75022]:0x106220a)
[21:39:13] fail:     at corlib_System_Runtime_CompilerServices_RuntimeHelpers_TryEnsureSufficientExecutionStack (<anonymous>:wasm-function[74477]:0x10517fb)
[21:39:13] fail:     at System_Text_RegularExpressions_System_Threading_StackHelper_TryEnsureSufficientExecutionStack (<anonymous>:wasm-function[96157]:0x138dd1d)
[21:39:13] fail:     at System_Text_RegularExpressions_System_Text_RegularExpressions_RegexPrefixAnalyzer__FindCaseSensitivePrefixg__Process_17_0_System_Text_RegularExpressions_RegexNode_System_Text_ValueStringBuilder_ (<anonymous>:wasm-function[96329]:0x139ba31)
[21:39:13] fail:     at System_Text_RegularExpressions_System_Text_RegularExpressions_RegexPrefixAnalyzer__FindCaseSensitivePrefixg__Process_17_0_System_Text_RegularExpressions_RegexNode_System_Text_ValueStringBuilder_ (<anonymous>:wasm-function[96329]:0x139be7a)
[21:39:13] fail:     at System_Text_RegularExpressions_System_Text_RegularExpressions_RegexPrefixAnalyzer__FindCaseSensitivePrefixg__Process_17_0_System_Text_RegularExpressions_RegexNode_System_Text_ValueStringBuilder_ (<anonymous>:wasm-function[96329]:0x139be7a)

... <frames repeat>

[ghost]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Seen on main rolling build:
Build
Log

[21:39:13] fail: RangeError: Maximum call stack size exceeded
[21:39:13] fail: RangeError: Maximum call stack size exceeded
[21:39:13] fail:     at jit_info_table_index (<anonymous>:wasm-function[121904]:0x18e0147)
[21:39:13] fail:     at jit_info_table_find (<anonymous>:wasm-function[121903]:0x18e0078)
[21:39:13] fail:     at mono_jit_info_table_find_internal (<anonymous>:wasm-function[121902]:0x18dffb9)
[21:39:13] fail:     at mini_jit_info_table_find_ext (<anonymous>:wasm-function[134483]:0x1a4b0f0)
[21:39:13] fail:     at mono_find_jit_info_ext (<anonymous>:wasm-function[134482]:0x1a4af06)
[21:39:13] fail:     at unwinder_unwind_frame (<anonymous>:wasm-function[134500]:0x1a4b969)
[21:39:13] fail:     at mono_walk_stack_full (<anonymous>:wasm-function[134464]:0x1a4a85e)
[21:39:13] fail:     at mono_walk_stack_with_ctx (<anonymous>:wasm-function[134462]:0x1a4a6ec)
[21:39:13] fail:     at mono_runtime_walk_stack_with_ctx (<anonymous>:wasm-function[134448]:0x1a4a462)
[21:39:13] fail:     at mono_stack_walk (<anonymous>:wasm-function[121987]:0x18e30ab)
[21:39:13] fail:     at is_running_protected_wrapper (<anonymous>:wasm-function[123237]:0x1901809)
[21:39:13] fail:     at mono_thread_interruption_checkpoint_request (<anonymous>:wasm-function[123251]:0x1901b4a)
[21:39:13] fail:     at mono_thread_interruption_checkpoint (<anonymous>:wasm-function[123250]:0x1901b1c)
[21:39:13] fail:     at aot_wrapper_icall_mono_thread_interruption_checkpoint (<anonymous>:wasm-function[66920]:0xf05864)
[21:39:13] fail:     at aot_wrapper_corlib_System_dot_Runtime_dot_CompilerServices_System_dot_Runtime_dot_CompilerServices_dot_RuntimeHelpers__SufficientExecutionStack_pinvoke_bool_bool_ (<anonymous>:wasm-function[75022]:0x106220a)
[21:39:13] fail:     at corlib_System_Runtime_CompilerServices_RuntimeHelpers_TryEnsureSufficientExecutionStack (<anonymous>:wasm-function[74477]:0x10517fb)
[21:39:13] fail:     at System_Text_RegularExpressions_System_Threading_StackHelper_TryEnsureSufficientExecutionStack (<anonymous>:wasm-function[96157]:0x138dd1d)
[21:39:13] fail:     at System_Text_RegularExpressions_System_Text_RegularExpressions_RegexPrefixAnalyzer__FindCaseSensitivePrefixg__Process_17_0_System_Text_RegularExpressions_RegexNode_System_Text_ValueStringBuilder_ (<anonymous>:wasm-function[96329]:0x139ba31)
[21:39:13] fail:     at System_Text_RegularExpressions_System_Text_RegularExpressions_RegexPrefixAnalyzer__FindCaseSensitivePrefixg__Process_17_0_System_Text_RegularExpressions_RegexNode_System_Text_ValueStringBuilder_ (<anonymous>:wasm-function[96329]:0x139be7a)
[21:39:13] fail:     at System_Text_RegularExpressions_System_Text_RegularExpressions_RegexPrefixAnalyzer__FindCaseSensitivePrefixg__Process_17_0_System_Text_RegularExpressions_RegexNode_System_Text_ValueStringBuilder_ (<anonymous>:wasm-function[96329]:0x139be7a)

Author:	radical
Assignees:	-
Labels:	arch-wasm, blocking-clean-ci, area-Codegen-AOT-mono
Milestone:	-

[radical]

cc @vargaz

[lewing]

cc @radekdoulik

[lewing]

runtime/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexPrefixAnalyzer.cs

Line 54 in 25237fa

static bool Process(RegexNode node, ref ValueStringBuilder vsb)

looks like the problem. It calls TryEnsureSufficientExecutionStack and the mono icall may need some improvement

[lewing]

cc @lambdageek

[lambdageek]

@lewing @radical Is this test actually expected to recurse deeply? There's not a bug where the test is unintentionally getting into an infinite recursion?

It looks like in this case we overflowed in a method called from the wrapper to do a stack walk. Maybe those are expensive on wasm? In any case, maybe we can double MONO_MIN_EXECUTION_STACK_SIZE on Wasm to 128 as if it was a 64-bit platform:

runtime/src/mono/mono/metadata/icall.c

Lines 118 to 120 in c159108

	#define MONO_MIN_EXECUTION_STACK_SIZE (128 * 1024)
	#else
	#define MONO_MIN_EXECUTION_STACK_SIZE (64 * 1024)

(I checked and it seems like the stack bounds computation will get non-fake values on Emscripten - so doing this stack check should work)

[stephentoub]

Is this test actually expected to recurse deeply?

I can't tell which test in particular this is from the log, but we do have some regex tests meant to stress recursive processing by analyzing really deeply-nested expressions, e.g. "((((((((((((((((((((((((((a)))))))))))))))))))", except much, much more nesting. The code has calls to TryEnsureSufficientExecutionStack on code paths that recur, and this is being triggered from one of those.

[lewing]

@radekdoulik please see if you can get the stack check functioning properly.

[radical]

Hitting this on windows too, on #62352 - which enables AOT tests on windows : Build, and log

[ilonatommy]

The error changed, now it's out of memory on Win and Linux: #71126 (comment). When running with HighAOTResourceRequiringProject Windows is passing, Linux still failing with (Run):

src/mono/wasm/build/WasmApp.Native.targets(353,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Failed to compile /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net7.0/browser-wasm/wasm/for-publish/Microsoft.CodeAnalysis.CSharp.dll.bc -> /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net7.0/browser-wasm/wasm/for-publish/Microsoft.CodeAnalysis.CSharp.dll.o
 "/usr/local/emscripten/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=7 -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/usr/local/emscripten/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -O2 -g -v -c /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net7.0/browser-wasm/wasm/for-publish/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpagVIA7.tmp
clang version 15.0.0 (https://github.com/llvm/llvm-project fbce4a78035c32792b0a13cf1f169048b822c06b)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /usr/local/emscripten/emsdk/upstream/bin
 (in-process)
 "/usr/local/emscripten/emsdk/upstream/bin/clang-15" -cc1 -triple wasm32-unknown-emscripten -emit-obj --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name Microsoft.CodeAnalysis.CSharp.dll.bc -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=4 -debugger-tuning=gdb -v -fcoverage-compilation-dir=/__w/1/s/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests -resource-dir /usr/local/emscripten/emsdk/upstream/lib/clang/15.0.0 -O2 -Werror=implicit-function-declaration -fdebug-compilation-dir=/__w/1/s/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests -ferror-limit 19 -fvisibility default -fgnuc-version=4.2.1 -vectorize-loops -vectorize-slp -iwithsysroot/include/SDL -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/tmpagVIA7.tmp -x ir /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net7.0/browser-wasm/wasm/for-publish/Microsoft.CodeAnalysis.CSharp.dll.bc
clang -cc1 version 15.0.0 based upon LLVM 15.0.0git default target x86_64-unknown-linux-gnu
emcc: error: '/usr/local/emscripten/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=7 -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/usr/local/emscripten/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -O2 -g -v -c /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net7.0/browser-wasm/wasm/for-publish/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpagVIA7.tmp' failed (received SIGKILL (-9)) [took 664.787s]

[radical]

The error changed, now it's out of memory on Win and Linux: #71126 (comment). When running with HighAOTResourceRequiringProject Windows is passing, Linux still failing with (Run):

this is a build error, and the error in the issue description is when the tests are run.