cli xUnit tests fail since https://github.com/dotnet/runtime build 20220616.9

[RussKie]

Description

The update from the runtime fails at the test leg, see https://dev.azure.com/dnceng/public/_build/results?buildId=1833158&view=results.

It looks like the test runner (or the process?) reports a non-zero exit code despite no failed tests, e.g.:

Reproduction Steps

1. git clone https://github.com/dotnet/winforms
2. switch to darc-main-240f38e5-32c6-4ca8-8d1a-35df12541c87 branch
3. build.cmd -test

Expected behavior

All tests pass. The current main is green: https://dev.azure.com/dnceng/public/_build/results?buildId=1830484&view=results

Actual behavior

The test runner reports incorrect value and fails.

Regression?

By the looks of it.

Known Workarounds

No response

Configuration

No response

Other information

No response

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jkotas]

This is crashing during shutdown at this callstack:

(9360.870c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
combase!TLSAddToMap+0x5b:
00007ffe`79eb4393 8b02            mov     eax,dword ptr [rdx] ds:00000000`00000220=????????
0:000> k
Call Site
combase!TLSAddToMap+0x5b [onecore\com\combase\tls\tls.cxx @ 182] 
combase!TLSPreallocateData+0xb4 [onecore\com\combase\tls\tls.cxx @ 378] 
combase!COleTls::TLSAllocData+0x13 [onecore\com\combase\tls\tls.cxx @ 434] 
combase!InternalTlsAllocData+0x13 [onecore\com\combase\tls\tls.cxx @ 1127] 
combase!COleTls::{ctor}+0xc8 [onecore\com\combase\ih\tls.h @ 574] 
combase!COleStaticMutexSem::Request+0xf0 [onecore\com\combase\common\internal\olesem.cxx @ 123] 
combase!CSpyMalloc_Free+0x34 [onecore\com\combase\class\memapi.cxx @ 902] 
clbcatq!SafeFree+0xa [com\complus\src\inc\svcmem.h @ 66] 
clbcatq!operator delete+0xa [com\complus\src\inc\svcmem.h @ 95] 
clbcatq!CComCLBCatalog::`scalar deleting destructor'+0x48
clbcatq!CComCLBCatalog::Release+0x26 [com\complus\src\comcat\catalogqueries\clbcatq\comclbcatalog.cpp @ 348] 
combase!CComCatalog::~CComCatalog+0xa2
ucrtbase!<lambda_f03950bc5685219e0bcd2087efbe011e>::operator()+0xb0 [minkernel\crts\ucrt\src\appcrt\startup\onexit.cpp @ 206] 
ucrtbase!__crt_seh_guarded_call<int>::operator()<<lambda_7777bce6b2f8c936911f934f8298dc43>,<lambda_f03950bc5685219e0bcd2087efbe011e> &,<lambda_3883c3dff614d5e0c5f61bb1ac94921c> >+0x3b [VCCRT\vcruntime\inc\internal_shared.h @ 204] 
ucrtbase!__acrt_lock_and_call+0x1c [minkernel\crts\ucrt\inc\corecrt_internal.h @ 974] 
ucrtbase!_execute_onexit_table+0x3d [minkernel\crts\ucrt\src\appcrt\startup\onexit.cpp @ 160] 
ucrtbase!__crt_state_management::wrapped_invoke<int (__cdecl*)(_onexit_table_t *),_onexit_table_t *,int>+0x2c [minkernel\crts\ucrt\inc\corecrt_internal_state_isolation.h @ 418] 
combase!dllmain_crt_process_detach+0x45 [VCCRT\vcstartup\src\startup\dll_dllmain.cpp @ 182] 
combase!dllmain_dispatch+0xfc [VCCRT\vcstartup\src\startup\dll_dllmain.cpp @ 293] 
ntdll!LdrpCallInitRoutine+0x6b [minkernel\ntdll\ldr.c @ 231] 
ntdll!LdrShutdownProcess+0x170 [minkernel\ntdll\ldrinit.c @ 7245] 
ntdll!RtlExitUserProcess+0xad [minkernel\ntdll\rtlstrt.c @ 1570] 
KERNEL32!ExitProcessImplementation+0xb [clientcore\base\win32\client\process.c @ 2522] 
ucrtbase!exit_or_terminate_process+0x44 [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 143] 
ucrtbase!common_exit+0x79 [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 280] 
dotnet!__scrt_common_main_seh+0x173 [d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 295] 
KERNEL32!BaseThreadInitThunk+0x10 [clientcore\base\win32\client\thread.c @ 75] 
ntdll!RtlUserThreadStart+0x2b [minkernel\ntdll\rtlstrt.c @ 1152] 

[jkotas]

Checked CoreCLR build is failing with assert (it may be a different problem - not sure):

0:033> k
Call Site
KERNELBASE!DebugBreak [minkernel\kernelbase\debug.c @ 143] 
KERNELBASE!wil::details::DebugBreak+0x2 [onecore\internal\sdk\inc\wil\opensource\wil\result_macros.h @ 1829] 
coreclr!DbgAssertDialog+0x1b9 [C:\runtime\src\coreclr\utilcode\debug.cpp @ 599] 
coreclr!CONTRACT_ASSERT+0x320 [C:\runtime\src\coreclr\inc\contract.inl @ 593] 
coreclr!EEContract::DoChecks+0x213 [C:\runtime\src\coreclr\vm\eecontract.cpp @ 121] 
coreclr!HelperMethodFrame::Pop+0xaf [C:\runtime\src\coreclr\vm\frames.cpp @ 1679] 
coreclr!FrameWithCookie<HelperMethodFrame>::Pop+0x5 [C:\runtime\src\coreclr\vm\frames.h @ 3249] 
coreclr!MngdNativeArrayMarshaler::ConvertSpaceToNative+0x2df [C:\runtime\src\coreclr\vm\ilmarshalers.cpp @ 4359] 
xunit_console!ILStubClass.IL_STUB_CLRtoCOM(System.Guid*, System.String[], UInt32, LCID, DispatchID*)+0xb2
System_Windows_Forms_Primitives_Tests!System.Windows.Forms.Primitives.Tests.Interop.Oleaut32.IDispatchTests.IDispatch_GetIDsOfNames_Invoke_Success+0x206
coreclr!CallDescrWorkerInternal+0x83
coreclr!RuntimeMethodHandle::InvokeMethod+0xb59 [C:\runtime\src\coreclr\vm\reflectioninvocation.cpp @ 739] 

[jkotas]

@dotnet/interop-contrib Could you please take a look?

[jkotas]

(I have been running System.Windows.Forms.Primitives.Tests test.)

[RussKie]

Thank you @jkotas for the quick check.

For sanity I decided to rebuild several commits previously built successfully, and now I wonder if this is a CI environmental issue than a .NET runtime issue... It looks like the issue may have started sometime over the weekend, but we didn't change anything in dotnet/winforms codebase...

/cc: @dotnet/dnceng for any ideas.

[premun]

Having a quick look at some of the main builds - this seems to be something inside of the Xunit runner but it's flakey?

For instance this build - 1833634 - only some legs failed.

However, I don't see anything infra related, especially if you can reproduce locally?

[AaronRobinsonMSFT]

I'll take a look.

[AaronRobinsonMSFT]

Running just that test doesn't cause a failure. I am going to assume some of the new ComWrappers or LibraryImport work has a bug in it that missed a GC mode transition - boo.

[AaronRobinsonMSFT]

This is because of the managed IMallocSpy implementation being used. It has helped to identify we were missing a transition to Preemptive mode in some places. There is also a bug in add/ref logic for things I found with a CHK build of coreclr. I will be pushing up a change to dotnet/runtime and to dotnet/winforms today.