Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_StapledOcsp(offlineContext: False) #70322

Closed
VincentBu opened this issue Jun 7, 2022 · 5 comments · Fixed by #70413
Closed

Test failure System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_StapledOcsp(offlineContext: False) #70322

VincentBu opened this issue Jun 7, 2022 · 5 comments · Fixed by #70413
Assignees
@bartonjs
Labels
area-System.Net.Security blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs os-linux Linux OS (any supported distro)
Milestone
7.0.0

Comments

@VincentBu
Copy link
Contributor

Run: runtime-libraries-coreclr outerloop 20220606.1

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af

- System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_StapledOcsp(offlineContext: False)

net7.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

- System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_StapledOcsp(offlineContext: False)

Error message:

Assert.Contains() Failure
Not found: (filter expression)
In value:  X509ChainStatus[] [System.Security.Cryptography.X509Certificates.X509ChainStatus, System.Security.Cryptography.X509Certificates.X509ChainStatus]


Stack trace
   at System.Net.Security.Tests.CertificateValidationRemoteServer.<ConnectWithRevocation_WithCallback_Core>g__CertificateValidationCallback|6_0(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 255
   at System.Net.Security.SslStream.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs:line 985
   at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 514
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 538
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 356
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 128
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 92
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 198
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
--- End of stack trace from previous location ---
@VincentBu VincentBu added area-System.Net.Security os-linux Linux OS (any supported distro) arch-x64 blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels Jun 7, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 7, 2022
@ghost
Copy link

ghost commented Jun 7, 2022

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-libraries-coreclr outerloop 20220606.1

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af

- System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_StapledOcsp(offlineContext: False)

net7.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

- System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_StapledOcsp(offlineContext: False)

Error message:

Assert.Contains() Failure
Not found: (filter expression)
In value:  X509ChainStatus[] [System.Security.Cryptography.X509Certificates.X509ChainStatus, System.Security.Cryptography.X509Certificates.X509ChainStatus]


Stack trace
   at System.Net.Security.Tests.CertificateValidationRemoteServer.<ConnectWithRevocation_WithCallback_Core>g__CertificateValidationCallback|6_0(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 255
   at System.Net.Security.SslStream.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs:line 985
   at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 514
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 538
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 356
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 128
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 92
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 198
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
--- End of stack trace from previous location ---
Author: VincentBu
Assignees: -
Labels:

area-System.Net.Security, os-linux, arch-x64, blocking-outerloop
Milestone: -

@rzikm
Copy link
Member

rzikm commented Jun 7, 2022

@bartonjs you added this test recently, any thoughts why it may have failed?

@runfoapp runfoapp bot mentioned this issue Jun 7, 2022
Closed
@bartonjs
Copy link
Member

bartonjs commented Jun 7, 2022

any thoughts why it may have failed?

runtime/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs

Line 103 in 6a8f16e

[OuterLoop("Subject to system load race conditions")]

The 200ms sleep in the test wasn't sufficient to get the background OCSP download completed, probably due to system load.

That number was enough that I never saw failures on my machine. We can raise it, or do some magic with reflection to sleep in small increments until the backing field of the certificate context object has gone non-null (and fail it if too much total time has elapsed).

@karelz karelz added this to the 7.0.0 milestone Jun 7, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Jun 7, 2022
@karelz karelz removed the arch-x64 label Jun 7, 2022
@karelz
Copy link
Member

karelz commented Jun 7, 2022

Triage: New test, we should tweak it in 7.0. @bartonjs let us know if you need help here.

@bartonjs bartonjs mentioned this issue Jun 7, 2022
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jun 7, 2022
@VincentBu
Copy link
Contributor Author

Failed again in: runtime-libraries-coreclr outerloop 20220614.1

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

- System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_StapledOcsp(offlineContext: False)

net7.0-Linux-Release-x64-CoreCLR_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af

- System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_StapledOcsp(offlineContext: False)

Error message:

Assert.Contains() Failure
Not found: (filter expression)
In value:  X509ChainStatus[] [System.Security.Cryptography.X509Certificates.X509ChainStatus, System.Security.Cryptography.X509Certificates.X509ChainStatus]


Stack trace
   at System.Net.Security.Tests.CertificateValidationRemoteServer.<ConnectWithRevocation_WithCallback_Core>g__CertificateValidationCallback|6_0(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 255
   at System.Net.Security.SslStream.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs:line 984
   at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 519
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 543
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 359
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 128
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 92
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 198
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 199
--- End of stack trace from previous location ---

@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jun 15, 2022
@rzikm rzikm mentioned this issue Jun 21, 2022
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Jul 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@bartonjs bartonjs

Labels
area-System.Net.Security blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs os-linux Linux OS (any supported distro)
Projects
None yet
Milestone
7.0.0
Development

Successfully merging a pull request may close this issue.

Make ConnectWithRevocation_StapledOcsp more resilient to being slow bartonjs/runtime
4 participants
@karelz @bartonjs @rzikm @VincentBu