Test failure JIT\\jit64\\opt\\cse\\hugeexpr1\\hugeexpr1.cmd

[v-wenyuxu]

Run: runtime-coreclr outerloop 20220824.6

Failed test:

R2R-CG2 windows x86 Checked no_tiered_compilation @ Windows.10.Amd64.Open

- JIT\\jit64\\opt\\cse\\hugeexpr1\\hugeexpr1.cmd

Error message:

Out of memory.

Return code:      1
Raw output file:      C:\h\w\9DB7088E\w\B91F0A66\uploads\Reports\JIT.jit64\opt\cse\hugeexpr1\hugeexpr1.output.txt
Raw output:
BEGIN EXECUTION
hugeexpr1.dll
1 file(s) copied.
3:34:27.37
Response file: C:\h\w\9DB7088E\w\B91F0A66\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll.rsp
C:\h\w\9DB7088E\w\B91F0A66\e\JIT\jit64\opt\cse\hugeexpr1\IL-CG2\hugeexpr1.dll
-o:C:\h\w\9DB7088E\w\B91F0A66\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll
--targetarch:x86
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\9DB7088E\p\System..dll
-r:C:\h\w\9DB7088E\p\Microsoft..dll
-r:C:\h\w\9DB7088E\p\mscorlib.dll
-r:C:\h\w\9DB7088E\p\netstandard.dll
-O
" "dotnet" "C:\h\w\9DB7088E\p\crossgen2\crossgen2.dll" @"C:\h\w\9DB7088E\w\B91F0A66\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll.rsp"   -r:C:\h\w\9DB7088E\w\B91F0A66\e\JIT\jit64\opt\cse\hugeexpr1\IL-CG2*.dll"
Emitting R2R PE file: C:\h\w\9DB7088E\w\B91F0A66\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll
3:36:47.46
Crossgen2 failed with exitcode - -532462766
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\9DB7088E\p
C:\h\w\9DB7088E\w\B91F0A66\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.cmd
Expected: True
Actual:   False


Stack trace
   at JIT_jit64._opt_cse_hugeexpr1_hugeexpr1_._opt_cse_hugeexpr1_hugeexpr1_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[jakobbotsch]

cc @ivdiazsa, maybe this needs to be redisabled for x86.

[mangod9]

@AntonLapounov another crossgen2 failure but this time on x86

[AntonLapounov]

It has been intermittently failing for a long time: #61825.

[AntonLapounov]

I have looked at the dump; this is an OutOfMemoryException.

0:000> .ecxr; .frame 3; dt __pException
...
03 02f7e72c 725318a2     coreclr!JIT_NewArr1+0x14e1c5 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 2626] 
Local var @ 0x2f7e714 Type Exception*
0x72efd960 
   +0x000 __VFN_table : 0x72e68638 
   =72efc264 g_OOMException   : 0x72efd960 Exception
   +0x004 m_innerException : (null) 

0:000> !clrstack
OS Thread Id: 0xd94 (0)
Child SP       IP Call Site
02F7E6C8 778ff04c [HelperMethodFrame: 02f7e6c8] 
02F7E734 725318a2 System.Array.Resize[[System.Byte, System.Private.CoreLib]](Byte[] ByRef, Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Array.cs @ 59]
02F7E74C 2feca1f3 ILCompiler.DependencyAnalysis.ReadyToRun.CopiedMetadataBlobNode.GetData(ILCompiler.DependencyAnalysis.NodeFactory, Boolean)
02F7E7EC 3fe46dd0 ILCompiler.DependencyAnalysis.ReadyToRunObjectWriter.EmitPortableExecutable()
02F7E8F4 2fec0ca3 ILCompiler.ReadyToRunCodegenCompilation.Compile(System.String)
02F7E968 26b29b41 ILCompiler.Program.RunSingleCompilation(System.Collections.Generic.Dictionary`2, ILCompiler.InstructionSetSupport, System.String, System.Collections.Generic.Dictionary`2, System.Collections.Generic.HashSet`1, ILCompiler.CompilerTypeSystemContext)
02F7EBC0 04dc1979 ILCompiler.Program.Run(System.String[])
02F7EDA0 04dc0ae6 ILCompiler.Program.Main(System.String[])

[mangod9]

@trylek do you know if this might be related to running multiple tests in parallel on helix machines? how would we harden this scenario ?

[trylek]

For the non-merged tests (still a majority) parallelization is controlled by the xunit console but I have no idea whether we have any "official" knobs for controlling that, some time ago we discussed this with Andrew in context of GC tests that are sometimes also sensitive to concurrent execution. I'll take a closer look and I'll try to figure out how to introduce the support for tests that need exclusive access to the running machine.

[mangod9]

this is marked as blocking outerloop. Moved it to 8 for now to determine if we can reduce parallelization for certain tests.

[v-wenyuxu]

Failed again in: runtime-coreclr outerloop 20220918.1

Failed test:

R2R-CG2 windows x86 Checked no_tiered_compilation @ Windows.10.Amd64.Open

- JIT\\jit64\\opt\\cse\\hugeexpr1\\hugeexpr1.cmd

Error message:

Out of memory.

Return code:      1
Raw output file:      C:\h\w\ABD809AD\w\B8EC0A3D\uploads\Reports\JIT.jit64\opt\cse\hugeexpr1\hugeexpr1.output.txt
Raw output:
BEGIN EXECUTION
hugeexpr1.dll
1 file(s) copied.
12:03:45.16
Response file: C:\h\w\ABD809AD\w\B8EC0A3D\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll.rsp
C:\h\w\ABD809AD\w\B8EC0A3D\e\JIT\jit64\opt\cse\hugeexpr1\IL-CG2\hugeexpr1.dll
-o:C:\h\w\ABD809AD\w\B8EC0A3D\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll
--targetarch:x86
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\ABD809AD\p\System..dll
-r:C:\h\w\ABD809AD\p\Microsoft..dll
-r:C:\h\w\ABD809AD\p\mscorlib.dll
-r:C:\h\w\ABD809AD\p\netstandard.dll
-O
" "dotnet" "C:\h\w\ABD809AD\p\crossgen2\crossgen2.dll" @"C:\h\w\ABD809AD\w\B8EC0A3D\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll.rsp"   -r:C:\h\w\ABD809AD\w\B8EC0A3D\e\JIT\jit64\opt\cse\hugeexpr1\IL-CG2*.dll"
Emitting R2R PE file: C:\h\w\ABD809AD\w\B8EC0A3D\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll
12:06:20.20
Crossgen2 failed with exitcode - -532462766
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\ABD809AD\p
C:\h\w\ABD809AD\w\B8EC0A3D\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.cmd
Expected: True
Actual:   False


Stack trace
   at JIT_jit64._opt_cse_hugeexpr1_hugeexpr1_._opt_cse_hugeexpr1_hugeexpr1_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[v-wenyuxu]

Failed again in: runtime-coreclr outerloop 20220927.3

Failed test:

R2R-CG2 windows x86 Checked no_tiered_compilation @ Windows.10.Amd64.Open

- JIT\\jit64\\opt\\cse\\hugeexpr1\\hugeexpr1.cmd

Error message:

Out of memory.

Return code:      1
Raw output file:      C:\h\w\ACEC09FF\w\B42109D2\uploads\Reports\JIT.jit64\opt\cse\hugeexpr1\hugeexpr1.output.txt
Raw output:
BEGIN EXECUTION
hugeexpr1.dll
1 file(s) copied.
20:49:04.64
Response file: C:\h\w\ACEC09FF\w\B42109D2\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll.rsp
C:\h\w\ACEC09FF\w\B42109D2\e\JIT\jit64\opt\cse\hugeexpr1\IL-CG2\hugeexpr1.dll
-o:C:\h\w\ACEC09FF\w\B42109D2\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll
--targetarch:x86
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\ACEC09FF\p\System..dll
-r:C:\h\w\ACEC09FF\p\Microsoft..dll
-r:C:\h\w\ACEC09FF\p\mscorlib.dll
-r:C:\h\w\ACEC09FF\p\netstandard.dll
-O
" "dotnet" "C:\h\w\ACEC09FF\p\crossgen2\crossgen2.dll" @"C:\h\w\ACEC09FF\w\B42109D2\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll.rsp"   -r:C:\h\w\ACEC09FF\w\B42109D2\e\JIT\jit64\opt\cse\hugeexpr1\IL-CG2*.dll"
Emitting R2R PE file: C:\h\w\ACEC09FF\w\B42109D2\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll
20:51:20.36
Crossgen2 failed with exitcode - -532462766
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\ACEC09FF\p
C:\h\w\ACEC09FF\w\B42109D2\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.cmd
Expected: True
Actual:   False


Stack trace
   at JIT_jit64._opt_cse_hugeexpr1_hugeexpr1_._opt_cse_hugeexpr1_hugeexpr1_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[v-wenyuxu]

Failed again in: runtime-coreclr r2r 20221009.1

Failed test:

R2R-CG2 windows x86 Checked no_tiered_compilation @ Windows.10.Amd64.Open

- JIT\\jit64\\opt\\cse\\hugeexpr1\\hugeexpr1.cmd

Error message:

Out of memory.

Return code:      1
Raw output file:      C:\h\w\A81C0921\w\B109097F\uploads\Reports\JIT.jit64\opt\cse\hugeexpr1\hugeexpr1.output.txt
Raw output:
BEGIN EXECUTION
hugeexpr1.dll
1 file(s) copied.
7:30:40.12
Response file: C:\h\w\A81C0921\w\B109097F\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll.rsp
C:\h\w\A81C0921\w\B109097F\e\JIT\jit64\opt\cse\hugeexpr1\IL-CG2\hugeexpr1.dll
-o:C:\h\w\A81C0921\w\B109097F\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll
--targetarch:x86
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\A81C0921\p\System..dll
-r:C:\h\w\A81C0921\p\Microsoft..dll
-r:C:\h\w\A81C0921\p\mscorlib.dll
-r:C:\h\w\A81C0921\p\netstandard.dll
-O
" "dotnet" "C:\h\w\A81C0921\p\crossgen2\crossgen2.dll" @"C:\h\w\A81C0921\w\B109097F\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll.rsp"   -r:C:\h\w\A81C0921\w\B109097F\e\JIT\jit64\opt\cse\hugeexpr1\IL-CG2*.dll"
Emitting R2R PE file: C:\h\w\A81C0921\w\B109097F\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.dll
7:33:07.78
Crossgen2 failed with exitcode - -532462766
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\A81C0921\p
C:\h\w\A81C0921\w\B109097F\e\JIT\jit64\opt\cse\hugeexpr1\hugeexpr1.cmd
Expected: True
Actual:   False


Stack trace
   at JIT_jit64._opt_cse_hugeexpr1_hugeexpr1_._opt_cse_hugeexpr1_hugeexpr1_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[ivdiazsa]

I think we should disable this test for Windows x86 for the time being, and keep this issue open until we investigate how we can fix it. Thoughts @mangod9 @AntonLapounov @trylek?

[mangod9]

should be fine if this is x86 only.

[trylek]

I can send out a PR for adding the issues.targets entry. For the above discussion regarding test parallelization on x86, I no longer think it is related. The machines are physically x64 and even if the generated XUnit wrapper is run as an x86 app (I'm not sure about that), it's a very short app that basically only runs separate processes for the individual test cases. These processes run in separate address spaces that shouldn't be constrained by 32-bit considerations. I'm guessing it's just that Crossgen2 gets too close to the 2 GB limit and there's possibly a bit of non-determinism due to internal parallelization that probably affects the exact amount of GC memory and causes the test to pass or fail with a certain probability. We can take a closer look what exact data is causing the problem (CopiedMetadataBlobNode as suggested by the call stack shared by Anton can hardly be causing this by itself as it's basically the MSIL metadata within the managed test binary, I don't think that can be that big).

[BruceForstall]

This failed in the last outerloop run:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=49391&view=ms.vss-test-web.build-test-results-tab&runId=998924&resultId=103575&paneView=debug

[trylek]

As the test is disabled in issues.targets, we no longer need the "blocking outerloop" label.

[trylek]

This is basically a 32-bit compilation limit, I don't think we can do anything reasonable about it in .NET 8 timeframe, moving to future as 64-bit platforms are sufficiently widespread so that this is basically a corner case that shouldn't affect most customers.