System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod failing on arm64

[jaredpar]

System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod

Console Log Summary

* ExceptionTests - reported call stack:
   at System.Tests.ExceptionTests.ThrowException() in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 143
   at System.Tests.ExceptionTests.ThrowAndRethrowOtherMethod(ValueTuple`3& rethrownExceptionStackFrame) in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 130
   at System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod() in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 118
    System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod [FAIL]
      Assert.Equal() Failure
      Expected: 131
      Actual:   130
      Stack Trace:
        /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs(154,0): at System.Tests.ExceptionTests.VerifyCallStack(ValueTuple`3 expectedStackFrame, String reportedCallStack, Int32 skipFrames)
        /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs(122,0): at System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod()
  Finished:    System.Runtime.Tests

Builds

Build	Test Failure Count
#506244	2
#506364	2

Configurations

• netcoreapp5.0-Linux-Release-arm64-CoreCLR_release-(Alpine.38.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.8-helix-arm64v8-a45aeeb-20190620184035
• netcoreapp5.0-Linux-Release-arm64-CoreCLR_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-bfcd90a-20200127194925

Helix Logs

Build	Console	Core	Test Results
#506364	console.f84b1a44.log
#506244	console.9fde1ff7.log
#506364	console.3cbd2e2b.log		testResults.xml
#506244	console.0b83c09a.log		testResults.xml

[janvorli]

@jaredpar it actually shows a message indicating the problem:
_ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
See #1723 for more details.

[jaredpar]

@janvorli thanks! Missed that.

Does look like that is the same issue as #1723. Will keep this one open to track getting the CI runs back to green and have #1723 track fixing the bug + enabling the run again.

[safern]

I just merged: #2290 which fixes this, and now exposes a real test failure:

* ExceptionTests - reported call stack:
   at System.Tests.ExceptionTests.ThrowException() in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 143
   at System.Tests.ExceptionTests.ThrowAndRethrowOtherMethod(ValueTuple`3& rethrownExceptionStackFrame) in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 130
   at System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod() in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 118
    System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod [FAIL]
      Assert.Equal() Failure
      Expected: 131
      Actual:   130
      Stack Trace:
        /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs(154,0): at System.Tests.ExceptionTests.VerifyCallStack(ValueTuple`3 expectedStackFrame, String reportedCallStack, Int32 skipFrames)
        /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs(122,0): at System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod()
  Finished:    System.Runtime.Tests
=== TEST EXECUTION SUMMARY ===

@jaredpar do we want to open a follow up issue for that one, or should we just keep this one open and track this failure getting fixed as part of this issue.

[safern]

I updated the title and description for this issue. @jkotas @stephentoub do you have any idea what might be happening and what would the difference be on alpine arm64 vs amd64? Could it be a docker image setup? @janvorli @wfurt?

[stephentoub]

cc: @mikem8361

[safern]

Actually it seems like it is not only alpine, also Ubuntu.
https://helix.dot.net/api/2019-06-17/jobs/9983a98e-06ab-40bf-94f9-1a4216df4da0/workitems/System.Runtime.Tests/console

Which also runs in a docker container: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-bfcd90a-20200127194925

Both are arm64. Updating the title.

[mikem8361]

My recent PR ##2269 to fix the stack trace line numbers may have broken this test. Investigating.

[mikem8361]

Looks like PR #31696 didn't fix this problem. I'm going to have to figure out how to debug it on one of our local arm64 pi's.

It is weird that it only fails on linux arm64 not windows arm64 though I couldn't find a windows arm64 library test leg.

[jaredpar]

@mikem8361

Don't believe we run the libraries tests on windows arm64 at the moment. Here is the relevant entry from runtime.yml

- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/libraries/run-test-job.yml
    buildConfig: ${{ variables.debugOnPrReleaseOnRolling }}
    platforms:
    - Windows_NT_x64
    - OSX_x64
    - Linux_x64
    - Linux_musl_x64
    - ${{ if eq(variables['isFullMatrix'], true) }}:
      - Linux_arm
      - Linux_arm64
      - Linux_musl_arm64
    - ${{ if eq(variables['isFullMatrix'], false) }}:
      - Windows_NT_x86

@safern can confirm my reading here.

[safern]

Yes that is correct, it fails on Linux arm64 and Linux_musl_arm64. We're waiting for necessary resources in order to be able to run tests on windows arm64, @trylek knows more about windows arm/arm64.

[trylek]

In mid-December the DDFUN team reinstalled the queue Windows.10.Arm64.Open we had been using previously for Windows arm32 testing to a new chipset that doesn't support the arm32 subsystem. That nuked Windows arm32 testing that doesn't work to this day. Based on my discussions with @MattGal and other dnceng and DDFUN people I received repeated confirmation that the queue Windows.10.Arm64.Open should be sufficient for Windows arm64 testing of the runtime repo. Last week I managed to push the first successful run through the queue and I sent out a PR for adding the Windows arm64 job to the PR runs:

#2095

Once this gets approved and merged in, we'll start receiving Windows arm64 results in PR runs. Initially I don't want to add the arm64 job to outerloop runs as they are noisy enough already - let's monitor stability and pass rate of the arm64 runs say for a week and based on the results we can decide what other pipelines we want to add arm64 jobs to.

[safern]

I put up a PR to disable this tests for arm64 in the effort of getting the rolling builds green: #31735

[trylek]

Thank you, just approved that.

[tommcdon]

This is an off-by-one-line number issue. We suggest updating the tests to be resilient against off-by one line issues between architectures. This is a common issue on both native and managed code.

[safern]

@tommcdon the test is disabled, should we keep the issue opened until we add that resiliency and we re-enable the test?

[janvorli]

This is just a test issue, moving to future.

[jeffschwMSFT]

Linking the following customer reported issue to this issue: https://developercommunity.visualstudio.com/t/NET-SDK-on-Ubuntu-18046-libnethosta-/10269507

[jkotas]

@jeffschwMSFT This does not look like the right issue.

[jeffschwMSFT]

I will create a new one ;)