Test: System.Net.Http.Functional.Tests.ManagedHandler_HttpClientHandler_ServerCertificates_Test/UseCallback_ValidCertificate_ExpectedValuesDuringCallback failed with "Xunit.Sdk.EqualException"

Opened on behalf of @JiayiLi1

The test System.Net.Http.Functional.Tests.ManagedHandler_HttpClientHandler_ServerCertificates_Test/UseCallback_ValidCertificate_ExpectedValuesDuringCallback(url: https://corefx-net.cloudapp.net/Echo.as... has failed.

Assert.Equal() Failure
Expected: None
Actual: RemoteCertificateChainErrors

    Stack Trace:

       at System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.<>c__DisplayClass4_0.<UseCallback_ValidCertificate_ExpectedValuesDuringCallback>b__0(HttpRequestMessage request, X509Certificate2 cert, X509Chain chain, SslPolicyErrors errors) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.ServerCertificates.cs:line 132
   at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback, ProtocolToken& alertToken) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 1036
   at System.Net.Security.SslState.CompleteHandshake(ProtocolToken& alertToken) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 1022
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 817
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 801
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 970
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 873
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 830
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 801
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 970
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 1102
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result) in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 720
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpConnectionHandler.<EstablishSslConnection>d__9.MoveNext() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Http/src/System/Net/Http/Managed/HttpConnectionHandler.cs:line 89
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Threading.Tasks.Extensions/src/System/Runtime/CompilerServices/ConfiguredValueTaskAwaitable.cs:line 61
   at System.Net.Http.HttpConnectionHandler.<CreateConnection>d__10.MoveNext() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Http/src/System/Net/Http/Managed/HttpConnectionHandler.cs:line 105
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Threading.Tasks.Extensions/src/System/Runtime/CompilerServices/ConfiguredValueTaskAwaitable.cs:line 61
   at System.Net.Http.HttpConnectionHandler.<SendAsyncWithNewConnection>d__8.MoveNext() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Http/src/System/Net/Http/Managed/HttpConnectionHandler.cs:line 59
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.CookieHandler.<SendAsync>d__3.MoveNext() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Http/src/System/Net/Http/Managed/CookieHandler.cs:line 31
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.AutoRedirectHandler.<SendAsync>d__3.MoveNext() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Http/src/System/Net/Http/Managed/AutoRedirectHandler.cs:line 28
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 464
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.<UseCallback_ValidCertificate_ExpectedValuesDuringCallback>d__4.MoveNext() in /Users/buildagent/agent/_work/48/s/corefx/src/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.ServerCertificates.cs:line 138
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Build : Master - 20170706.01 (Core Tests)
Failing configurations:

• OSX.1012.Amd64-x64
  • Release

Detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170706.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.ManagedHandler_HttpClientHandler_ServerCertificates_Test~2FUseCallback_ValidCertificate_ExpectedValuesDuringCallback(url:%20https:~2F~2Fcorefx-net.cloudapp.net~2FEcho.as...

[v-haren]

failed again: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_osx_debug/110/testReport/System.Net.Http.Functional.Tests/ManagedHandler_HttpClientHandler_ServerCertificates_Test/UseCallback_ValidCertificate_ExpectedValuesDuringCallback_url__https___corefx_net_cloudapp_net_Echo_ashx__checkRevocation__True_/

[KristinXie1]

This issue is repro on OuterLoop_netcoreapp_osx10.12_debug, detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_osx_debug/115/testReport/System.Net.Http.Functional.Tests/ManagedHandler_HttpClientHandler_ServerCertificates_Test/UseCallback_ValidCertificate_ExpectedValuesDuringCallback_url__https___corefx_net_cloudapp_net_Echo_ashx__checkRevocation__True_/

[v-haren]

failed again: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_osx_debug/115/testReport/System.Net.Http.Functional.Tests/ManagedHandler_HttpClientHandler_ServerCertificates_Test/UseCallback_ValidCertificate_ExpectedValuesDuringCallback_url__https___corefx_net_cloudapp_net_Echo_ashx__checkRevocation__True_/

[KristinXie1]

Failed again on Core Tests (Master), detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170713.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.ManagedHandler_HttpClientHandler_ServerCertificates_Test~2FUseCallback_ValidCertificate_ExpectedValuesDuringCallback(url:%20https:~2F~2Fcorefx-net.cloudapp.net~2FEcho.as...

[davidsh]

Error:
Assert.Equal() Failure\nExpected: None\nActual: RemoteCertificateChainErrors

Note: This is a strange error because the SSL certificate is perfect for the corefx-net.cloudapp.net server. So, this looks like a bug in the OSX HTTP stack.

[v-haren]

failed again: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_osx_debug/120/testReport/System.Net.Http.Functional.Tests/ManagedHandler_HttpClientHandler_ServerCertificates_Test/UseCallback_ValidCertificate_ExpectedValuesDuringCallback_url__https___corefx_net_cloudapp_net_Echo_ashx__checkRevocation__True_/

[stephentoub]

@bartonjs, do we have any SslStream tests for cert revocation checking? This test UseCallback_ValidCertificate_ExpectedValuesDuringCallback is a theory, and it's the checkRevocation:true calls that are failing. I'm wondering if this is actually an SslStream bug related to revocation checking that we're only noticing now having turned on the ManagedHandler tests that use SslStream.

[bartonjs]

do we have any SslStream tests for cert revocation checking?

I'm not sure. I don't think I wrote one, but I'd like to think they were already there.

this looks like a bug in the OSX HTTP stack

I know that Apple does revocation checking differently. (Which reminds me of their "Think Different" slogan). They sometimes (for reasons I haven't been able to black-box deduce) don't bother checking CRLs, deciding that OCSP is the only way to go. Then they report RevocationStatusUnknown, because they realize "oh, yeah, I was supposed to check for revocation, but didn't bother".

So it's possible that for this cert chain Apple has decided that it's OCSP-o-nada, and that there's not an OCSP option.

That'd be mostly true if the test is failing deterministicly.

What would probably help is changing it up similar to how some of the cert chain tests are. Instead of Assert.Equal(SslPolicyErrors.None, errors) do

if ((SslPolicyErrors.RemoteCertificateChainErrors & errors) != 0)
{
    X509ChainStatusFlags flags = chain.ChainStatus.Aggregate(X509ChainStatusFlags.NoError, (cur, status) => cur | status);
    // throw, assert, whatever, to get the flags reported.
    // or use the ChainElements values to get the flags per level in the chain.
}

Assert.Equal(SslPolicyErrors.None, errors);

[stephentoub]

What would probably help is changing it up similar to how some of the cert chain tests are

dotnet/corefx#22303

[stephentoub]

@bartonjs, you were right about the status:

Expected None, got RemoteCertificateChainErrors with chain status RevocationStatusUnknown

Is there anything we can do about this in the product code? Or should I just change the test to skip the assert if revocation checking was enabled and it gets this error with this status and it's a mac? Or something else?