baseservices\exceptions\stackoverflow\stackoverflowtester fails under pgo stress on windows arm #81360

AndyAyersMS · 2023-01-30T16:38:04Z

https://dev.azure.com/dnceng-public/public/_build/results?buildId=151537&view=ms.vss-test-web.build-test-results-tab

Fails for full PGO and random GDV, eg

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1

  Starting:    baseservices.exceptions.XUnitWrapper (parallel test collections = on, max threads = 2)
    baseservices\exceptions\stackoverflow\stackoverflowtester\stackoverflowtester.cmd [FAIL]
      
      Return code:      1
      Raw output file:      C:\h\w\A4B60915\w\AFE90948\uploads\Reports\baseservices.exceptions\stackoverflow\stackoverflowtester\stackoverflowtester.output.txt
      Raw output:
      BEGIN EXECUTION
       "C:\h\w\A4B60915\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  stackoverflowtester.dll 
      Running stackoverflow test(smallframe main)
      "Stack overflow."
      "Repeat 32317 times:"
      "--------------------------------"
      "   at TestStackOverflow.Program.InfiniteRecursionB()"
    
      ...

      Running stackoverflow3 test()
      Call number 50 to the Execute method
      Call number 100 to the Execute method
      Call number 150 to the Execute method
      "Stack overflow."
      ""
      "Assert failure(PID 2200 [0x00000898], Thread: 8624 [0x21b0]): Consistency check failed: FAILED: m_crawl.pFrame->IsTransitionToNativeFrame()"
      ""
      "<no module>! <no symbol> + 0x0 (0x00000000)"
      "    File: D:\a\_work\1\s\src\coreclr\vm\stackwalk.cpp Line: 2281"
      "    Image: C:\h\w\A4B60915\p\corerun.exe"
      ""
      ""
      Exit code: 0xC0000602, expected 0xC00000FD or 0x800703E9

The text was updated successfully, but these errors were encountered:

ghost · 2023-01-30T16:38:13Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

https://dev.azure.com/dnceng-public/public/_build/results?buildId=151537&view=ms.vss-test-web.build-test-results-tab

Fails for full PGO and random GDV, eg

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1

  Starting:    baseservices.exceptions.XUnitWrapper (parallel test collections = on, max threads = 2)
    baseservices\exceptions\stackoverflow\stackoverflowtester\stackoverflowtester.cmd [FAIL]
      
      Return code:      1
      Raw output file:      C:\h\w\A4B60915\w\AFE90948\uploads\Reports\baseservices.exceptions\stackoverflow\stackoverflowtester\stackoverflowtester.output.txt
      Raw output:
      BEGIN EXECUTION
       "C:\h\w\A4B60915\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  stackoverflowtester.dll 
      Running stackoverflow test(smallframe main)
      "Stack overflow."
      "Repeat 32317 times:"
      "--------------------------------"
      "   at TestStackOverflow.Program.InfiniteRecursionB()"
    
      ...

      Running stackoverflow3 test()
      Call number 50 to the Execute method
      Call number 100 to the Execute method
      Call number 150 to the Execute method
      "Stack overflow."
      ""
      "Assert failure(PID 2200 [0x00000898], Thread: 8624 [0x21b0]): Consistency check failed: FAILED: m_crawl.pFrame->IsTransitionToNativeFrame()"
      ""
      "<no module>! <no symbol> + 0x0 (0x00000000)"
      "    File: D:\a\_work\1\s\src\coreclr\vm\stackwalk.cpp Line: 2281"
      "    Image: C:\h\w\A4B60915\p\corerun.exe"
      ""
      ""
      Exit code: 0xC0000602, expected 0xC00000FD or 0x800703E9

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`, `untriaged`
Milestone:	-

AndyAyersMS · 2023-01-30T16:39:03Z

arm/windows is no longer supported.

Might want to consider disabling this test

AndyAyersMS · 2023-02-01T19:55:16Z

Can't build locally: #81493.

Will download from CI instead.

AndyAyersMS · 2023-02-01T20:13:54Z

I can repro with downloaded bits; investigating.

AndyAyersMS · 2023-02-01T20:35:33Z

Looks like we catch the overflow and have a "partial context" and then assert after walking one frame:

 	KernelBase.dll!wil::details::DebugBreak(void)	Unknown
 	coreclr.dll!CHECK::Setup(const char * message, const char * condition, const char * file, int line) Line 198	C++
>	coreclr.dll!StackFrameIterator::NextRaw() Line 2281	C++
 	[Inline Frame] coreclr.dll!StackFrameIterator::Next() Line 1571	C++
 	coreclr.dll!Thread::StackWalkFramesEx(REGDISPLAY * pRD, StackWalkAction(*)(CrawlFrame *, void *) pCallback, void * pData, unsigned int flags, Frame * pStartFrame) Line 929	C++
 	coreclr.dll!Thread::StackWalkFrames(StackWalkAction(*)(CrawlFrame *, void *) pCallback, void * pData, unsigned int flags, Frame * pStartFrame) Line 1007	C++
 	coreclr.dll!LogCallstackForLogWorker(Thread * pThread) Line 307	C++
 	coreclr.dll!LogStackOverflowStackTraceThread(void * arg) Line 591	C++
 	[External Code]

@janvorli think this is something you probably should look at?

To repro, on windows arm box, with runfo bits from CI:

set DOTNET_ReadyToRun=0
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1

set CORE_ROOT=...

corerun.exe stackoverflow3.dll

janvorli · 2023-02-03T15:28:45Z

@AndyAyersMS I'll take a look.

AndyAyersMS · 2023-02-14T16:00:00Z

@janvorli should I assign this over to you?

janvorli · 2023-02-14T16:44:10Z

@AndyAyersMS I have just assigned that to myself.

JulieLeeMSFT · 2023-03-06T16:37:47Z

@janvorli, did you have a chance to look into this? Our goal is to resolve CI test failures within a month. Thanks.

janvorli · 2023-03-06T21:27:12Z

@JulieLeeMSFT not yet, but taking a look now.

janvorli · 2023-03-10T17:39:46Z

@JulieLeeMSFT I have found the culprit. The problem happens in case the stack overflow happens in native code and there is an explicit frame between the managed and native code. Some time ago, I have added a fix for a problem when the stack overflow happened in native code, but I haven't found that it actually works only in case there is no explicit frame between the managed code and the failing native code frame. That fix makes the FaultingExceptionFrame that is created for the stack overflow to contain context of the managed code frame, so the stack walker uses that to move to the next frame. But while doing so, it hits the other explicit frame that it doesn't expect there and fires an assert.

I am working on a fix, figuring out the cleanest way to fix it.

JulieLeeMSFT · 2023-03-10T19:52:06Z

@janvorli, great that you root-caused this issue!!

@JulieLeeMSFT I have found the culprit. The problem happens in case the stack overflow happens in native code and there is an explicit frame between the managed and native code. Some time ago, I have added a fix for a problem when the stack overflow happened in native code, but I haven't found that it actually works only in case there is no explicit frame between the managed code and the failing native code frame. That fix makes the FaultingExceptionFrame that is created for the stack overflow to contain context of the managed code frame, so the stack walker uses that to move to the next frame. But while doing so, it hits the other explicit frame that it doesn't expect there and fires an assert.

I am working on a fix, figuring out the cleanest way to fix it.

JulieLeeMSFT · 2023-03-30T22:41:38Z

Failed in runtime-coreclr pgostress test: https://dev.azure.com/dnceng-public/public/_build/results?buildId=219898&view=ms.vss-test-web.build-test-results-tab&runId=4120128&resultId=121377&paneView=debug

Failed on Arm64: coreclr windows arm64 Checked fullpgo_random_gdv_edge @ Windows.11.Arm64.Open

Error message:

Return code:      1
Raw output file:      C:\h\w\ABE2098D\w\9B8E0845\uploads\Reports\baseservices.exceptions\stackoverflow\stackoverflowtester\stackoverflowtester.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\ABE2098D\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  stackoverflowtester.dll
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeat 32326 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeat 8 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"--------------------------------"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(smallframe secondary)
"Stack overflow."
"Repeat 32379 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"
"   at System.Threading.Thread+StartHelper.RunWorker()"
"   at System.Threading.Thread+StartHelper.Run()"
"   at System.Threading.Thread.StartCallback()"
""
Running stackoverflow test(largeframe secondary)
"Stack overflow."
"Repeat 8 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"--------------------------------"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"
"   at System.Threading.Thread+StartHelper.RunWorker()"
"   at System.Threading.Thread+StartHelper.Run()"
"   at System.Threading.Thread.StartCallback()"
""
Running stackoverflow3 test()
Call number 50 to the Execute method
Call number 100 to the Execute method
Call number 150 to the Execute method
"Stack overflow."
""
"Assert failure(PID 8936 [0x000022e8], Thread: 6124 [0x17ec]): Consistency check failed: FAILED: m_crawl.pFrame->IsTransitionToNativeFrame()"
""
"<no module>! <no symbol> + 0x0 (0x00000000)"
"    File: D:\a_work\1\s\src\coreclr\vm\stackwalk.cpp Line: 2284"
"    Image: C:\h\w\ABE2098D\p\corerun.exe"
""
""
Exit code: 0xC0000602, expected 0xC00000FD or 0x800703E9
Expected: 100
Actual: 105
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:\cores since 3/28/2023 4:54:28 PM
Processing C:\cores\corerun.exe.8936.dmp
Unable to find cdb.exe at C:\Program Files (x86)\Windows Kits\10\Debuggers\arm\cdb.exe
Processing C:\cores\corerun.exe.7800.dmp
Unable to find cdb.exe at C:\Program Files (x86)\Windows Kits\10\Debuggers\arm\cdb.exe
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\ABE2098D\p
C:\h\w\ABE2098D\w\9B8E0845\e\baseservices\exceptions\stackoverflow\stackoverflowtester\stackoverflowtester.cmd
Expected: True
Actual:   False

Stack trace

   at baseservices_exceptions._stackoverflow_stackoverflowtester_stackoverflowtester_._stackoverflow_stackoverflowtester_stackoverflowtester_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

JulieLeeMSFT · 2023-04-07T17:18:11Z

Failed again. runtime-coreclr pgostress

baseservices\exceptions\stackoverflow\stackoverflowtester\stackoverflowtester.cmd

Running stackoverflow3 test()
Call number 50 to the Execute method
Call number 100 to the Execute method
Call number 150 to the Execute method
"Stack overflow."
""
"Assert failure(PID 5888 [0x00001700], Thread: 4160 [0x1040]): Consistency check failed: FAILED: m_crawl.pFrame->IsTransitionToNativeFrame()"
""
"<no module>! <no symbol> + 0x0 (0x00000000)"
"    File: D:\a_work\1\s\src\coreclr\vm\stackwalk.cpp Line: 2284"
"    Image: C:\h\w\A51808B0\p\corerun.exe"
""
""
Exit code: 0xC0000602, expected 0xC00000FD or 0x800703E9
Expected: 100
Actual: 105
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:\cores since 4/1/2023 7:54:15 AM
Processing C:\cores\corerun.exe.5888.dmp
Unable to find cdb.exe at C:\Program Files (x86)\Windows Kits\10\Debuggers\arm\cdb.exe
Processing C:\cores\corerun.exe.9204.dmp
Unable to find cdb.exe at C:\Program Files (x86)\Windows Kits\10\Debuggers\arm\cdb.exe
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=C:\h\w\A51808B0\p
> C:\h\w\A51808B0\w\AE9408F0\e\baseservices\exceptions\stackoverflow\stackoverflowtester\stackoverflowtester.cmd
Expected: True
Actual:   False


Stack trace
   at baseservices_exceptions._stackoverflow_stackoverflowtester_stackoverflowtester_._stackoverflow_stackoverflowtester_stackoverflowtester_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

JulieLeeMSFT · 2023-04-07T22:45:23Z

Ping @janvorli. Checking back on the status.

janvorli · 2023-04-12T23:19:51Z

@JulieLeeMSFT thank you for reminding me, this has fallen off my radar due to my vacation and a sick leave after that. I'll get back to it as soon as I can.

ghost added the untriaged New issue has not been triaged by the area owner label Jan 30, 2023

dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jan 30, 2023

AndyAyersMS added arch-arm32 os-windows and removed untriaged New issue has not been triaged by the area owner labels Jan 30, 2023

AndyAyersMS added this to the 8.0.0 milestone Jan 30, 2023

AndyAyersMS added the blocking-clean-ci-optional Blocking optional rolling runs label Jan 30, 2023

runfoapp bot mentioned this issue Jan 30, 2023

Infrastructure - Status/Health #702

Closed

JulieLeeMSFT assigned AndyAyersMS Jan 30, 2023

janvorli assigned janvorli and unassigned AndyAyersMS Feb 14, 2023

AndyAyersMS mentioned this issue Apr 7, 2023

JIT: scalable profile counter mode #84427

Merged

JulieLeeMSFT added the arch-arm64 label Apr 7, 2023

JulieLeeMSFT added the area-VM-coreclr label Apr 17, 2023

JulieLeeMSFT removed the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Apr 17, 2023

EgorBo mentioned this issue Apr 18, 2023

Don't re-instrument methods which don't need that #84772

Closed

janvorli mentioned this issue Apr 24, 2023

Fix edge case in stack overflow handling #85272

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Apr 24, 2023

janvorli closed this as completed in #85272 Apr 25, 2023

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Apr 25, 2023

ghost locked as resolved and limited conversation to collaborators May 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

baseservices\exceptions\stackoverflow\stackoverflowtester fails under pgo stress on windows arm #81360

baseservices\exceptions\stackoverflow\stackoverflowtester fails under pgo stress on windows arm #81360

AndyAyersMS commented Jan 30, 2023

ghost commented Jan 30, 2023

AndyAyersMS commented Jan 30, 2023

AndyAyersMS commented Feb 1, 2023

AndyAyersMS commented Feb 1, 2023

AndyAyersMS commented Feb 1, 2023

janvorli commented Feb 3, 2023

AndyAyersMS commented Feb 14, 2023

janvorli commented Feb 14, 2023

JulieLeeMSFT commented Mar 6, 2023

janvorli commented Mar 6, 2023

janvorli commented Mar 10, 2023

JulieLeeMSFT commented Mar 10, 2023

JulieLeeMSFT commented Mar 30, 2023 •

edited

Loading

JulieLeeMSFT commented Apr 7, 2023

JulieLeeMSFT commented Apr 7, 2023

janvorli commented Apr 12, 2023

baseservices\exceptions\stackoverflow\stackoverflowtester fails under pgo stress on windows arm #81360

baseservices\exceptions\stackoverflow\stackoverflowtester fails under pgo stress on windows arm #81360

Comments

AndyAyersMS commented Jan 30, 2023

ghost commented Jan 30, 2023

AndyAyersMS commented Jan 30, 2023

AndyAyersMS commented Feb 1, 2023

AndyAyersMS commented Feb 1, 2023

AndyAyersMS commented Feb 1, 2023

janvorli commented Feb 3, 2023

AndyAyersMS commented Feb 14, 2023

janvorli commented Feb 14, 2023

JulieLeeMSFT commented Mar 6, 2023

janvorli commented Mar 6, 2023

janvorli commented Mar 10, 2023

JulieLeeMSFT commented Mar 10, 2023

JulieLeeMSFT commented Mar 30, 2023 • edited Loading

JulieLeeMSFT commented Apr 7, 2023

JulieLeeMSFT commented Apr 7, 2023

janvorli commented Apr 12, 2023

JulieLeeMSFT commented Mar 30, 2023 •

edited

Loading