Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failure: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd #85663

Open
BruceForstall opened this issue May 2, 2023 · 4 comments
Labels
area-crossgen2-coreclr disabled-test The test is disabled in source code against the issue
Milestone

Comments

@BruceForstall
Copy link
Member

https://dev.azure.com/dnceng-public/public/_build/results?buildId=258972&view=ms.vss-test-web.build-test-results-tab&runId=5063424&paneView=debug

R2R-CG2 windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open

11:16:57.724 Running test: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Internal.JitInterface.CorInfoImpl.JitCompileMethod(IntPtr ByRef, IntPtr, IntPtr, IntPtr, Internal.JitInterface.CORINFO_METHOD_INFO ByRef, UInt32, IntPtr ByRef, UInt32 ByRef)
   at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(ILCompiler.DependencyAnalysis.IMethodNode, Internal.IL.MethodIL)
   at Internal.JitInterface.CorInfoImpl.CompileMethod(ILCompiler.DependencyAnalysis.ReadyToRun.MethodWithGCInfo, ILCompiler.Logger)
   at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOneMethod|5(ILCompiler.DependencyAnalysisFramework.DependencyNodeCore`1<ILCompiler.DependencyAnalysis.NodeFactory>, Int32)
   at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOnThread|4(Int32)
   at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompilationThread|3(System.Object)
   at System.Threading.Thread.StartCallback()

Return code:      1
Raw output file:      C:\h\w\9A640888\w\B2420953\uploads\JitBlue\DevDiv_255294\DevDiv_255294\output.txt
Raw output:
BEGIN EXECUTION
DevDiv_255294.dll
        1 file(s) copied.
11:16:57.97
Response file: C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\DevDiv_255294.dll.rsp
C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\IL-CG2\DevDiv_255294.dll
-o:C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\DevDiv_255294.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\9A640888\p\System.*.dll
-r:C:\h\w\9A640888\p\Microsoft.*.dll
-r:C:\h\w\9A640888\p\xunit.*.dll
-r:C:\h\w\9A640888\p\mscorlib.dll
-r:C:\h\w\9A640888\p\netstandard.dll
-O
" "dotnet" "C:\h\w\9A640888\p\crossgen2\crossgen2.dll" @"C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\DevDiv_255294.dll.rsp"   -r:C:\h\w\9A640888\w\B2420953\e\JIT\Regression\Regression_3\IL-CG2\*.dll  -r:C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\IL-CG2\*.dll"
11:17:39.72
Crossgen2 failed with exitcode - -1073741819
Test failed. Trying to see if dump file was created in C:\cores since 5/2/2023 11:16:57 AM
Processing C:\cores\dotnet.exe.6080.dmp
Unable to find cdb.exe at C:\Program Files (x86)\Windows Kits\10\Debuggers\arm\cdb.exe
Finished looking for *.dmp. No new files created.
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> C:\h\w\9A640888\w\B2420953\e\JIT\Regression\Regression_3\../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd
11:17:40.332 Failed test: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd

Also failed in the previous build in:

R2R-CG2 linux arm64 Checked no_tiered_compilation @ (Alpine.315.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.15-helix-arm64v8

https://dev.azure.com/dnceng-public/public/_build/results?buildId=258902&view=ms.vss-test-web.build-test-results-tab&runId=5058938&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=116049

06:28:59.323 Running test: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd
/root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh: line 317: -r:/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/*.dll: No such file or directory
/root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh: line 258:  7183 Killed                  $__Command

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/JitBlue/DevDiv_255294/DevDiv_255294/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/DevDiv_255294.dll
06:28:59
Response file: /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.rsp
/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/DevDiv_255294.dll
-o:/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll
-r:/root/helix/work/correlation/System.*.dll
-r:/root/helix/work/correlation/Microsoft.*.dll
-r:/root/helix/work/correlation/xunit.*.dll
-r:/root/helix/work/correlation/mscorlib.dll
--verify-type-and-field-layout
--method-layout:random
--targetarch:arm64
--targetos:linux
-O
Running CrossGen2:  dotnet /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.rsp  
Running R2RDump:  dotnet /root/helix/work/correlation/R2RDump/R2RDump.dll --header --sc --in /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll --out /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.r2rdump --val
Error: System.BadImageFormatException: The file is not a ReadyToRun image
   at ILCompiler.Reflection.ReadyToRun.ReadyToRunReader.Initialize(IAssemblyMetadata metadata) in /_/src/coreclr/tools/aot/ILCompiler.Reflection.ReadyToRun/ReadyToRunReader.cs:line 480
   at R2RDump.Program.Run() in /_/src/coreclr/tools/r2rdump/Program.cs:line 447
06:29:13
Crossgen2 failed with exitcode: 137
in ReleaseLock
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 5/2/2023 6:28:59 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh
06:29:14.668 Failed test: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd

@dotnet/jit-contrib

@BruceForstall BruceForstall added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels May 2, 2023
@BruceForstall BruceForstall added this to the 8.0.0 milestone May 2, 2023
@ghost
Copy link

ghost commented May 2, 2023

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

https://dev.azure.com/dnceng-public/public/_build/results?buildId=258972&view=ms.vss-test-web.build-test-results-tab&runId=5063424&paneView=debug

R2R-CG2 windows arm Checked no_tiered_compilation @ Windows.11.Arm64.Open

11:16:57.724 Running test: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Internal.JitInterface.CorInfoImpl.JitCompileMethod(IntPtr ByRef, IntPtr, IntPtr, IntPtr, Internal.JitInterface.CORINFO_METHOD_INFO ByRef, UInt32, IntPtr ByRef, UInt32 ByRef)
   at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(ILCompiler.DependencyAnalysis.IMethodNode, Internal.IL.MethodIL)
   at Internal.JitInterface.CorInfoImpl.CompileMethod(ILCompiler.DependencyAnalysis.ReadyToRun.MethodWithGCInfo, ILCompiler.Logger)
   at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOneMethod|5(ILCompiler.DependencyAnalysisFramework.DependencyNodeCore`1<ILCompiler.DependencyAnalysis.NodeFactory>, Int32)
   at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOnThread|4(Int32)
   at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompilationThread|3(System.Object)
   at System.Threading.Thread.StartCallback()

Return code:      1
Raw output file:      C:\h\w\9A640888\w\B2420953\uploads\JitBlue\DevDiv_255294\DevDiv_255294\output.txt
Raw output:
BEGIN EXECUTION
DevDiv_255294.dll
        1 file(s) copied.
11:16:57.97
Response file: C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\DevDiv_255294.dll.rsp
C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\IL-CG2\DevDiv_255294.dll
-o:C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\DevDiv_255294.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\9A640888\p\System.*.dll
-r:C:\h\w\9A640888\p\Microsoft.*.dll
-r:C:\h\w\9A640888\p\xunit.*.dll
-r:C:\h\w\9A640888\p\mscorlib.dll
-r:C:\h\w\9A640888\p\netstandard.dll
-O
" "dotnet" "C:\h\w\9A640888\p\crossgen2\crossgen2.dll" @"C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\DevDiv_255294.dll.rsp"   -r:C:\h\w\9A640888\w\B2420953\e\JIT\Regression\Regression_3\IL-CG2\*.dll  -r:C:\h\w\9A640888\w\B2420953\e\JIT\Regression\JitBlue\DevDiv_255294\DevDiv_255294\IL-CG2\*.dll"
11:17:39.72
Crossgen2 failed with exitcode - -1073741819
Test failed. Trying to see if dump file was created in C:\cores since 5/2/2023 11:16:57 AM
Processing C:\cores\dotnet.exe.6080.dmp
Unable to find cdb.exe at C:\Program Files (x86)\Windows Kits\10\Debuggers\arm\cdb.exe
Finished looking for *.dmp. No new files created.
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> C:\h\w\9A640888\w\B2420953\e\JIT\Regression\Regression_3\../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd
11:17:40.332 Failed test: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd

Also failed in the previous build in:

R2R-CG2 linux arm64 Checked no_tiered_compilation @ (Alpine.315.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.15-helix-arm64v8

https://dev.azure.com/dnceng-public/public/_build/results?buildId=258902&view=ms.vss-test-web.build-test-results-tab&runId=5058938&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=116049

06:28:59.323 Running test: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd
/root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh: line 317: -r:/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/*.dll: No such file or directory
/root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh: line 258:  7183 Killed                  $__Command

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/JitBlue/DevDiv_255294/DevDiv_255294/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/DevDiv_255294.dll
06:28:59
Response file: /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.rsp
/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/DevDiv_255294.dll
-o:/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll
-r:/root/helix/work/correlation/System.*.dll
-r:/root/helix/work/correlation/Microsoft.*.dll
-r:/root/helix/work/correlation/xunit.*.dll
-r:/root/helix/work/correlation/mscorlib.dll
--verify-type-and-field-layout
--method-layout:random
--targetarch:arm64
--targetos:linux
-O
Running CrossGen2:  dotnet /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.rsp  
Running R2RDump:  dotnet /root/helix/work/correlation/R2RDump/R2RDump.dll --header --sc --in /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll --out /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.r2rdump --val
Error: System.BadImageFormatException: The file is not a ReadyToRun image
   at ILCompiler.Reflection.ReadyToRun.ReadyToRunReader.Initialize(IAssemblyMetadata metadata) in /_/src/coreclr/tools/aot/ILCompiler.Reflection.ReadyToRun/ReadyToRunReader.cs:line 480
   at R2RDump.Program.Run() in /_/src/coreclr/tools/r2rdump/Program.cs:line 447
06:29:13
Crossgen2 failed with exitcode: 137
in ReleaseLock
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 5/2/2023 6:28:59 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh
06:29:14.668 Failed test: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd

@dotnet/jit-contrib

Author: BruceForstall
Assignees: -
Labels:

area-CodeGen-coreclr, blocking-outerloop

Milestone: 8.0.0

trylek added a commit to trylek/runtime that referenced this issue May 8, 2023
This test failure is known and tracked under

dotnet#85663

I'm adding the issues.targets entry to baseline the failure and
make outerloop green again.

Thanks

Tomas
BruceForstall pushed a commit that referenced this issue May 9, 2023
…85928)

* Exclude the test DevDiv_255294.cmd on Windows arm in issues.targets

This test failure is known and tracked under

#85663

I'm adding the issues.targets entry to baseline the failure and
make outerloop green again.

Thanks

Tomas

* Address Bruce's PR feedback
@BruceForstall
Copy link
Member Author

This looks like the same kind of issue as #85747: a huge (though very simple) method that crashes during crossgen.

@BruceForstall BruceForstall removed their assignment May 10, 2023
@BruceForstall BruceForstall added disabled-test The test is disabled in source code against the issue area-crossgen2-coreclr untriaged New issue has not been triaged by the area owner and removed blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI labels May 10, 2023
@mangod9 mangod9 modified the milestones: 8.0.0, Future Jul 31, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Jul 31, 2023
@v-wenyuxu
Copy link

Failed in: runtime-coreclr outerloop 20240412.1

Failed tests:

R2R-CG2 linux x64 Checked no_tiered_compilation @ (Alpine.317.Amd64.Open)Ubuntu.2204.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-amd64
    - JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd

Error message:

 /root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh: line 327: -r:/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/*.dll: No such file or directory
/root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh: line 266:  7583 Killed                  $__Command

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/JitBlue/DevDiv_255294/DevDiv_255294/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/DevDiv_255294.dll
10:54:43
Response file: /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.rsp
/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/DevDiv_255294.dll
-o:/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll
-r:/root/helix/work/correlation/System.*.dll
-r:/root/helix/work/correlation/Microsoft.*.dll
-r:/root/helix/work/correlation/xunit.*.dll
-r:/root/helix/work/correlation/mscorlib.dll
--verify-type-and-field-layout
--method-layout:random
--targetarch:x64
--targetos:linux
Running CrossGen2:  dotnet /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.rsp  
Running R2RDump:  dotnet /root/helix/work/correlation/R2RDump/R2RDump.dll --header --sc --in /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll --out /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.r2rdump --val
Error: System.BadImageFormatException: The file is not a ReadyToRun image
   at ILCompiler.Reflection.ReadyToRun.ReadyToRunReader.Initialize(IAssemblyMetadata metadata) in /_/src/coreclr/tools/aot/ILCompiler.Reflection.ReadyToRun/ReadyToRunReader.cs:line 467
   at R2RDump.Program.Run() in /_/src/coreclr/tools/r2rdump/Program.cs:line 449
10:54:55
Crossgen2 failed with exitcode: 137
in ReleaseLock
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 4/12/2024 10:54:43 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 128
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor322|0_323(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

@mikelle-rogers
Copy link
Member

The comment above looks like a different issue, because there is a different architecture, and the test is failing with a different error. I opened a new issue #101065.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-crossgen2-coreclr disabled-test The test is disabled in source code against the issue
Projects
None yet
Development

No branches or pull requests

4 participants