[9.0] Test assert failure in X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes

[carlossanlop]

Important: This was found in the 9.0 branch

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789147
Build error leg or test failing: System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(aiaResponseKind: Cert, mustIgnore: False)

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(aiaResponseKind: Cert, mustIgnore: False) [FAIL]",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789147&view=logs&j=07e4961f-4232-5b4a-5e6a-31092474840f&t=dfe85eac-3de9-57b6-7c34-e995be202bc7&l=55
Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-106995-merge-c7701f6f34bd4379a0/System.Security.Cryptography.Tests/1/console.2a4f1b53.log?helixlogtype=result
Output:

     System.AggregateException : One or more errors occurred. (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False) (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False) (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False) (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False) (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False)
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      Stack Trace:
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(54,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(79,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(AiaResponseKind aiaResponseKind, Boolean mustIgnore)
           at System.Object.InvokeStub_AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(Object , Span`1 )
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(136,0): at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
        ----- Inner Stack Trace #1 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #2 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #3 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #4 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #5 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=789147
Error message validated: [System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(aiaResponseKind: Cert, mustIgnore: False) [FAIL]]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/4/2024 5:20:44 PM UTC

Report

Build	Definition	Test	Pull Request
898564	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes	#110436
898034	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes	#108796
886541	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes
884679	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	2	4

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[bartonjs]

This failed when AIA was expected to work. Unfortunately, AIA tests can fail under high load (which RetryHelper hopes to outlast). Given that no other AIA tests failed it doesn't seem to be that AIA itself is not working, just that the requests timed out (or somehow the listener port got closed).