Test failure: Loader/classloader/InlineArray/InlineArrayInvalid/InlineArrayInvalid.cmd

[v-wenyuxu]

Failed in: runtime-coreclr r2r-extra 20250208.1

Failed tests:

R2R-CG2 windows x64 Checked gcstress0xc @ Windows.10.Amd64.Open
    - Loader/classloader/InlineArray/InlineArrayInvalid/InlineArrayInvalid.cmd

Error message:

 
Assert failure(PID 5344 [0x000014e0], Thread: 4580 [0x11e4]): !CREATE_CHECK_STRING(pMT && pMT->Validate())

CORECLR! Object::ValidateInner + 0x11A (0x00007ff8`664f7a3a)
CORECLR! Object::Validate + 0xA1 (0x00007ff8`664f78e1)
CORECLR! GcInfoDecoder::ReportStackSlotToGC + 0x139 (0x00007ff8`66944d19)
CORECLR! GcInfoDecoder::ReportUntrackedSlots + 0x14D (0x00007ff8`6694503d)
CORECLR! GcInfoDecoder::EnumerateLiveSlots + 0x169C (0x00007ff8`6694320c)
CORECLR! EECodeManager::EnumGcRefs + 0x2A1 (0x00007ff8`663c4071)
CORECLR! GcStackCrawlCallBack + 0x52A (0x00007ff8`666914ba)
CORECLR! Thread::StackWalkFramesEx + 0x268 (0x00007ff8`6655e818)
CORECLR! Thread::StackWalkFrames + 0x16D (0x00007ff8`6655e50d)
CORECLR! ScanStackRoots + 0x24D (0x00007ff8`6668e04d)
    File: D:�_work1ssrccoreclr�mobject.cpp:560
    Image: C:hwAEAF0937pcorerun.exe


Return code:      1
Raw output file:      C:hwAEAF0937wAA9609B8uploadsclassloaderInlineArrayInlineArrayInvalidoutput.txt
Raw output:
BEGIN EXECUTION
InlineArrayInvalid.dll
InvalidCSharpInlineArray.dll
TestLibrary.dll
        3 file(s) copied.
 3:49:16.58
Response file: C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInlineArrayInvalid.dll.rsp
C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidIL-CG2InlineArrayInvalid.dll
-o:C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInlineArrayInvalid.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwAEAF0937pSystem.*.dll
-r:C:hwAEAF0937pMicrosoft.*.dll
-r:C:hwAEAF0937p\xunit.*.dll
-r:C:hwAEAF0937pmscorlib.dll
-r:C:hwAEAF0937p
etstandard.dll
"Running CrossGen2:  "C:hwAEAF0937pcrossgen2crossgen2.exe" @"C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInlineArrayInvalid.dll.rsp"   -r:C:hwAEAF0937wAA9609B8eLoaderLoaderIL-CG2*.dll  -r:C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidIL-CG2*.dll"
Emitting R2R PE file: C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInlineArrayInvalid.dll
"Running R2RDump:  "dotnet" "C:hwAEAF0937p
2rdump
2rdump.dll" --header --sc --in C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInlineArrayInvalid.dll --out C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInlineArrayInvalid.dll.r2rdump --val"
 3:49:17.08
 3:49:17.08
Response file: C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInvalidCSharpInlineArray.dll.rsp
C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidIL-CG2InvalidCSharpInlineArray.dll
-o:C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInvalidCSharpInlineArray.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwAEAF0937pSystem.*.dll
-r:C:hwAEAF0937pMicrosoft.*.dll
-r:C:hwAEAF0937p\xunit.*.dll
-r:C:hwAEAF0937pmscorlib.dll
-r:C:hwAEAF0937p
etstandard.dll
"Running CrossGen2:  "C:hwAEAF0937pcrossgen2crossgen2.exe" @"C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInvalidCSharpInlineArray.dll.rsp"   -r:C:hwAEAF0937wAA9609B8eLoaderLoaderIL-CG2*.dll  -r:C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidIL-CG2*.dll"
Emitting R2R PE file: C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInvalidCSharpInlineArray.dll
"Running R2RDump:  "dotnet" "C:hwAEAF0937p
2rdump
2rdump.dll" --header --sc --in C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInvalidCSharpInlineArray.dll --out C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidInvalidCSharpInlineArray.dll.r2rdump --val"
 3:49:17.56
 3:49:17.56
Response file: C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidTestLibrary.dll.rsp
C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidIL-CG2TestLibrary.dll
-o:C:hwAEAF0937wAA9609B8eLoaderclassloaderInlineArrayInlineArrayInvalidTestLibrary.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--me

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor156|0_157(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[amanasifkhalid]

Looks like a GC hole exposed by GCStress:

set DOTNET_TieredCompilation=0
set DOTNET_GCStress=0xC

cc @dotnet/jit-contrib

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[v-wenyuxu]

Failed in: runtime-coreclr gcstress-extra 20250209.1

Failed tests:

coreclr windows x64 Checked gcstress0xc_jitminopts_heapverify1 @ Windows.10.Amd64.Open
    - Loader/classloader/explicitlayout/NestedStructs/case05/case05.cmd

Error message:

 
Assert failure(PID 2844 [0x00000b1c], Thread: 3800 [0x0ed8]): !CREATE_CHECK_STRING(pMT && pMT->Validate())

CORECLR! Object::ValidateInner + 0x11A (0x00007ffb`4a387a3a)
CORECLR! Object::Validate + 0xA1 (0x00007ffb`4a3878e1)
CORECLR! GcInfoDecoder::ReportStackSlotToGC + 0x139 (0x00007ffb`4a7d4d19)
CORECLR! GcInfoDecoder::ReportUntrackedSlots + 0x14D (0x00007ffb`4a7d503d)
CORECLR! GcInfoDecoder::EnumerateLiveSlots + 0x169C (0x00007ffb`4a7d320c)
CORECLR! EECodeManager::EnumGcRefs + 0x2A1 (0x00007ffb`4a254071)
CORECLR! GcStackCrawlCallBack + 0x52A (0x00007ffb`4a5214ba)
CORECLR! Thread::StackWalkFramesEx + 0x268 (0x00007ffb`4a3ee818)
CORECLR! Thread::StackWalkFrames + 0x16D (0x00007ffb`4a3ee50d)
CORECLR! ScanStackRoots + 0x24D (0x00007ffb`4a51e04d)
    File: D:�_work1ssrccoreclr�mobject.cpp:560
    Image: C:hwAE160997pcorerun.exe


Return code:      1
Raw output file:      C:hwAE160997wA71308BCuploadsclassloaderexplicitlayoutNestedStructscase05output.txt
Raw output:
BEGIN EXECUTION
 "C:hwAE160997pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  case05.dll 
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 2/9/2025 10:56:35 PM
Processing C:corescorerun.exe.2844.dmp
Invoking: C:Program Files (x86)Windows Kits10Debuggersdcdb.exe -c "$<C:hwAE160997		mpw2oqmy.tmp" -z "C:corescorerun.exe.2844.dmp"
stdout: 
Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:corescorerun.exe.2844.dmp]
User Mini Dump File with Full Memory: Only application data is available


************* Path validation summary **************
Response                         Time (ms)     Location
OK                                             C:hwAE160997pPDB
Symbol search path is: C:hwAE160997pPDB
Executable search path is: 
Windows 10 Version 14393 MP (4 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
10.0.14393.6343 (rs1_release.230913-1727)
Machine Name:
Debug session time: Sun Feb  9 22:56:39.000 2025 (UTC + 0:00)
System Uptime: 0 days 0:16:30.459
Process Uptime: 0 days 0:00:03.000
...............................

************* Symbol Loading Error Summary **************
Module name            Error
ntdll                  The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(b1c.ed8): Unknown exception - code c0000602 (first/second chance not available)
For analysis of this file, run !analyze -v
*** WARNING: Unable to verify checksum for coreclr.dll
KERNELBASE!RaiseFailFastException+0xae:
00007ffb`758b973e 803ddd77120000  cmp     byte ptr [KERNELBASE!local_unwind+0x75f2 (00007ffb`759e0f22)],0 ds:00007ffb`759e0f22=00
0:000> cdb: Reading initial command '$<C:hwAE160997		mpw2oqmy.tmp'
0:000> .load C:Users
unner.dotnetsossos.dll
0:000> ~*k

.  0  Id: b1c.ed8 Suspend: 0 Teb: 00000069`c9076000 Unfrozen
Child-SP          RetAddr           Call Site
00000069`c936c510 00007ffb`4a8fbc0b KERNELBASE!RaiseFailFastException+0xae
00000069`c936cae0 00007ffb`4a8fb4d5 coreclr!_DbgBreakCheck+0x2eb
00000069`c936dc30 00007ffb`4a8fb6cd coreclr!_DbgBreakCheckNoThrow+0x55
00000069`c936dcd0 00007ffb`4a387a3a coreclr!DbgAssertDialog+0x19d
00000069`c936ddc0 00007ffb`4a3878e1 coreclr!Object::ValidateInner+0x11a
00000069`c936dea0 00007ffb`4a7d4d19 coreclr!Object::Validate+0xa1
00000069`c936dee0 00007ffb`4a7d503d coreclr!GcInfoDecoder::ReportStackSlotToGC+0x139
(Inline Function) --------`-------- coreclr!GcInfoDecoder::ReportSlotToGC+0x10d
00000069`

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 123
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor125|0_126(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[AndyAyersMS]

This second batch of failures includes quite a few under minopts which makes it less likely to be a jit issue. For instance

https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-fd5c7483c98a476194/Regression_1.3.4/1/console.ea602b66.log?helixlogtype=result

(I could not repro this locally in a handful of attempts)

@mangod9 @jkotas anything recently changed in the runtime that might be worth considering?

[mangod9]

If this started recently and failures occur during exception handling could be related to #111937. But will need to look closer.

[jkotas]

I have investigated https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-fd5c7483c98a476194/Loader.0.3/1/console.eec09497.log?helixlogtype=result - Test_NestedStructsWithExplicitLayout_Case05.TestEntryPoint() test failure.

I was able to catch the crash with time travel debugging (I can share the trace if you need it). I can confirm that this is caused by #111937. We may want to revert this change since it is causing havoc in the runtime tests reliability.

The problem is that the GC stackwalk gets lost after RhThrowEx frame at the following stacktrace. We do not report GC roots for Test_NestedStructsWithExplicitLayout_Case05.TestEntryPoint frame, and then crash later when the execution unwinds and we start reporting the (now invalid) object references in this frame again.

00 000000e2`8eb720a8 00007ffd`d32dec36     coreclr!WKS::GCHeap::StressHeap [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 50091] 
01 (Inline Function) --------`--------     coreclr!_GCStress::StressGcTriggerPolicy::Trigger+0x69 [D:\a\_work\1\s\src\coreclr\vm\gcstress.h @ 301] 
02 (Inline Function) --------`--------     coreclr!_GCStress::GCSBase<0,_GCStress::EeconfigFastGcSPolicy,_GCStress::CoopGcModePolicy,mpl::null_type>::MaybeTrigger+0xbe [D:\a\_work\1\s\src\coreclr\vm\gcstress.h @ 408] 
03 000000e2`8eb720b0 00007ffd`d32ebe2b     coreclr!MethodDesc::DoPrestub+0x5a6 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 2860] 
04 000000e2`8eb723f0 00007ffd`d37aa685     coreclr!PreStubWorker+0x54b [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 2701] 
05 000000e2`8eb72650 00007ffd`79f9eef6     coreclr!ThePreStub+0x55 [D:\a\_work\1\s\src\coreclr\vm\amd64\ThePreStubAMD64.asm @ 21] 
06 000000e2`8eb72700 00007ffd`79f9e86d     System_Private_CoreLib!System.Runtime.EH.DispatchEx+0x2c6 [_/src/coreclr/nativeaot/Runtime.Base/src/System/Runtime/ExceptionHandling.cs @ 807] 
07 000000e2`8eb72860 00007ffd`d37a9393     System_Private_CoreLib!System.Runtime.EH.RhThrowEx+0x2d [_/src/coreclr/nativeaot/Runtime.Base/src/System/Runtime/ExceptionHandling.cs @ 666] 
08 000000e2`8eb72890 00007ffd`d33aff07     coreclr!CallDescrWorkerInternal+0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm @ 74] 
09 000000e2`8eb728d0 00007ffd`d36ee70e     coreclr!DispatchCallSimple+0x187 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 239] 
0a 000000e2`8eb72a00 00007ffd`d36fecca     coreclr!DispatchManagedException+0x45e [D:\a\_work\1\s\src\coreclr\vm\exceptionhandling.cpp @ 5717] 
0b (Inline Function) --------`--------     coreclr!ProcessCLRExceptionNew+0xc8e [D:\a\_work\1\s\src\coreclr\vm\exceptionhandling.cpp @ 978] 
0c 000000e2`8eb741c0 00007ffe`31e83f8f     coreclr!ProcessCLRException+0xcca [D:\a\_work\1\s\src\coreclr\vm\exceptionhandling.cpp @ 1003] 
0d 000000e2`8eb74470 00007ffe`31d34d22     ntdll!RtlpExecuteHandlerForUnwind+0xf [minkernel\ntos\rtl\amd64\xcptmisc.asm @ 254] 
0e 000000e2`8eb744a0 00007ffe`31d32d5c     ntdll!RtlUnwindEx+0x352 [minkernel\ntos\rtl\amd64\exdsptch.c @ 1538] 
0f 000000e2`8eb74bd0 00007ffd`d36edb2c     ntdll!RtlUnwind+0xfc [minkernel\ntos\rtl\amd64\exdsptch.c @ 1077] 
10 000000e2`8eb75140 00007ffd`d36fec93     coreclr!ClrUnwindEx+0x1c [D:\a\_work\1\s\src\coreclr\vm\exceptionhandling.cpp @ 5759] 
11 (Inline Function) --------`--------     coreclr!ProcessCLRExceptionNew+0xc57 [D:\a\_work\1\s\src\coreclr\vm\exceptionhandling.cpp @ 960] 
12 000000e2`8eb751b0 00007ffe`31e83f0f     coreclr!ProcessCLRException+0xc93 [D:\a\_work\1\s\src\coreclr\vm\exceptionhandling.cpp @ 1003] 
13 000000e2`8eb75460 00007ffe`31d33b78     ntdll!RtlpExecuteHandlerForException+0xf [minkernel\ntos\rtl\amd64\xcptmisc.asm @ 132] 
14 000000e2`8eb75490 00007ffe`31d32b86     ntdll!RtlDispatchException+0x2c8 [minkernel\ntos\rtl\amd64\exdsptch.c @ 680] 
15 000000e2`8eb75be0 00007ffe`2f5bbb0a     ntdll!RtlRaiseException+0x206 [minkernel\ntos\rtl\amd64\raise.c @ 240] 
16 000000e2`8eb76410 00007ffd`d39b5f63     KERNELBASE!RaiseException+0x8a [minkernel\kernelbase\xcpt.c @ 954] 
17 000000e2`8eb76510 00007ffd`d39b4f45     coreclr!__RethrowException+0x33 [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 1380] 
18 000000e2`8eb76540 00007ffe`31e43886     coreclr!__FrameHandler4::CxxCallCatchBlock+0x275 [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 1484] 
19 000000e2`8eb76610 00007ffd`d32ebe2b     ntdll!RcFrameConsolidation+0x6 [minkernel\ntos\rtl\amd64\capture.asm @ 1176] 
1a 000000e2`8eb7dd80 00007ffd`d37aa685     coreclr!PreStubWorker+0x54b [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 2701] 
1b 000000e2`8eb7dfe0 00007ffd`73f751e0     coreclr!ThePreStub+0x55 [D:\a\_work\1\s\src\coreclr\vm\amd64\ThePreStubAMD64.asm @ 21] 
1c 000000e2`8eb7e090 00007ffd`73f7510d     case05!Test_NestedStructsWithExplicitLayout_Case05.TestEntryPoint()+0x40
1d 000000e2`8eb7e0f0 00007ffd`d37a9393     case05!__GeneratedMainWrapper.Main()+0x1d
1e 000000e2`8eb7e140 00007ffd`d33afc4e     coreclr!CallDescrWorkerInternal+0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm @ 74] 
1f 000000e2`8eb7e180 00007ffd`d30b5bba     coreclr!MethodDescCallSite::CallTargetWorker+0x8ce [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 545] 
20 (Inline Function) --------`--------     coreclr!MethodDescCallSite::Call_RetArgSlot+0x87 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.h @ 439] 
21 000000e2`8eb7e630 00007ffd`d30b54aa     coreclr!RunMainInternal+0x35a [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1223] 
22 000000e2`8eb7e850 00007ffd`d30b559e     coreclr!``RunMain'::`30'::__Body::Run'::`5'::__Body::Run+0x7a [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1291] 
23 000000e2`8eb7e8a0 00007ffd`d30b577b     coreclr!`RunMain'::`30'::__Body::Run+0x8e [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1293] 
24 000000e2`8eb7e910 00007ffd`d30afb70     coreclr!RunMain+0x16b [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1293] 
25 000000e2`8eb7e9f0 00007ffd`d315caf5     coreclr!Assembly::ExecuteMainMethod+0x2e0 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1419] 
26 000000e2`8eb7ee80 00007ffd`d397e708     coreclr!CorHost2::ExecuteAssembly+0x525 [D:\a\_work\1\s\src\coreclr\vm\corhost.cpp @ 350] 
27 000000e2`8eb7f0c0 00007ff6`1a8a3662     coreclr!coreclr_execute_assembly+0x1a8 [D:\a\_work\1\s\src\coreclr\dlls\mscoree\exports.cpp @ 494] 
28 000000e2`8eb7f180 00007ff6`1a8a7d7d     corerun+0x13662
29 000000e2`8eb7f750 00007ff6`1a8b2cf9     corerun!GetCurrentClrDetails+0x27d
2a 000000e2`8eb7f890 00007ff6`1a8b2c12     corerun!GetCurrentClrDetails+0xb1f9
2b 000000e2`8eb7f8e0 00007ff6`1a8b2ace     corerun!GetCurrentClrDetails+0xb112
2c 000000e2`8eb7f950 00007ff6`1a8b2d6e     corerun!GetCurrentClrDetails+0xafce
2d 000000e2`8eb7f980 00007ffe`313ce8d7     corerun!GetCurrentClrDetails+0xb26e
2e 000000e2`8eb7f9b0 00007ffe`31ddbf2c     KERNEL32!BaseThreadInitThunk+0x17 [clientcore\base\win32\client\thread.c @ 77] 
2f 000000e2`8eb7f9e0 00000000`00000000     ntdll!RtlUserThreadStart+0x2c [minkernel\ldr\rtlstrt.c @ 1184]