Unexpected HttpClient cert validation & timeout failures #805

geeknoid · 2019-12-12T16:08:12Z

Below is a simple program that tries to read the home page of a few common web sites. I'm persistently experiencing two types of errors:

For certain sites (e.g. www.apple.com), the GET request fails due to a cert validation error. Installing a cert validation callback hook lets me see that the validation error is due to RemoteCertificateNameMismatch.
For certain sites (e.g. www.opentable.com), something gets stuck for a long time (maybe 30s), and then fails due to a timeout.

Issuing simple curl www.apple.com or curl www.opentable.com commands on the same physical machine works flawlessly, as does using a web browser, yet my simple program is failing. This is happening on macOS 10.15.1 using .NET Core 3.0 from Visual Studio for Mac 8.3.11 (build 1).

Produced output is included below.

using System;
using System.Net.Http;
using System.Threading.Tasks;

namespace HttpClientBug
{
	class Program
	{
		static async Task Main(string[] args)
		{
			string[] uris = {
				"https://microsoft.com",
				"https://www.apple.com",
				"https://ibm.com",
				"https://amazon.com",
				"https://github.com",
				"https://cnn.com",
				"https://msnbc.com",
				"https://wikipedia.org",
				"https://cbs.com",
				"https://news.com",
				"https://engadget.com",
				"https://www.opentable.com",
				"https://yelp.com",
			};

			var handler = new HttpClientHandler();
			var httpClient = new HttpClient(handler);
			foreach (var uri in uris) {
				try {
					var msg = await httpClient.GetAsync(uri);
					Console.WriteLine("NORMAL: uri {0}, code {1}, phrase '{2}'", uri, msg.StatusCode, msg.ReasonPhrase);
				} catch (TaskCanceledException e) {
					Console.WriteLine("CANCELED: uri {0}\n{1}", uri, e);
				} catch (Exception e) {
					Console.WriteLine("FAILED: uri {0}\n{1}", uri, e);
				}
			}
		}
	}
}

/*
NORMAL: uri https://microsoft.com, code OK, phrase 'OK'
FAILED: uri https://apple.com
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HttpClientBug.Program.Main(String[] args) in /Users/martin/Projects/WebProofer/HttpClientBug/Program.cs:line 30
NORMAL: uri https://ibm.com, code OK, phrase 'OK'
NORMAL: uri https://amazon.com, code OK, phrase 'OK'
NORMAL: uri https://github.com, code OK, phrase 'OK'
NORMAL: uri https://cnn.com, code OK, phrase 'OK'
NORMAL: uri https://msnbc.com, code OK, phrase 'OK'
NORMAL: uri https://wikipedia.org, code OK, phrase 'OK'
NORMAL: uri https://cbs.com, code OK, phrase 'OK'
FAILED: uri https://news.com
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HttpClientBug.Program.Main(String[] args) in /Users/martin/Projects/WebProofer/HttpClientBug/Program.cs:line 30
NORMAL: uri https://engadget.com, code OK, phrase 'OK'
FAILED: uri https://opentable.com
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation timed out.
 ---> System.Net.Sockets.SocketException(60): Operation timed out
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HttpClientBug.Program.Main(String[] args) in /Users/martin/Projects/WebProofer/HttpClientBug/Program.cs:line 30
NORMAL: uri https://yelp.com, code OK, phrase 'OK'
*/

The text was updated successfully, but these errors were encountered:

scalablecory · 2019-12-12T17:31:39Z

news.com fails for me in browser, so I think that one is a legit cert issue.
opentable.com is performing browser detection and works when adding relevant headers.

apple.com works on Windows. @aik-jahoda can you try to repro in OSX?

JohnyL · 2019-12-12T17:39:08Z

I'd like to add that sometimes I experience Task was cancelled without any explanations and inner exceptions.

scalablecory · 2019-12-12T17:43:50Z

@JohnyL HttpClient.Timeout will manifest as an OperationCancelledException, this might be what you're seeing. If not and if you have more specifics, please open a separate issue so we can take a look.

JohnyL · 2019-12-12T17:57:18Z

@scalablecory It's a luck I have found that piece of code:

internal static async Task<string> GetEncodedStringAsync(this HttpClient client, string url, string postData = null, int codepage = 65001)
{
	int retryCount = 0;
RETRY_REQUEST:
	if (retryCount == 6)
	{
		Logger.AddEntry($"URL: [{url}]\r\nPost Data: [{postData}]\r\nError: Getting TaskCanceledException more than 5 times.");
		return null;
	}
	var encoding = Encoding.GetEncoding(codepage);
	var request = new HttpRequestMessage(HttpMethod.Post, url);
	if (postData.Length > 0)
	{
		request.Content = new FormUrlEncodedContent(new Dictionary<string, string> { ["id"] = postData });
	}
	HttpResponseMessage response = null;
	try
	{
		response = await client.SendAsync(request);
	}
	catch (Exception ex)
	{
		if (ex is TaskCanceledException)
		{
			++retryCount;
			goto RETRY_REQUEST;
		}
		else
		{
			string innerEx = ex.InnerException?.Message;
			Logger.AddEntry($"URL: [{url}]\r\nPost Data: [{postData}]" +
							$"\r\nError: {ex.Message}" +
							(innerEx.Length > 0 ? ($"\r\nInner Error:{innerEx}") : string.Empty));
		}
	}
	if (response == null)
	{
		return null;
	}
	response = response.EnsureSuccessStatusCode();
	var bytesArray = await response.Content.ReadAsByteArrayAsync();
	return encoding.GetString(bytesArray);
}

As you see, it's TaskCanceledException and it appeared randomly.

geeknoid · 2019-12-12T18:06:46Z

@scalablecory news.com works great for me in a browser on Mac and on Windows. It's a redirect to cnet.com/news. As I mentioned in my initial report, it also works with curl on the same machines.

If I directly invoke cnet.com instead, I get the same error. Their cert has an issuer name of *.cnet.com, so maybe there's a problem with wildcard handling?

geeknoid · 2019-12-12T18:08:36Z

@scalablecory Also, opentable.com works fine with curl with no special headers.

scalablecory · 2019-12-12T18:11:15Z

@JohnyL thanks, please move to separate issue so we can keep track of it. from what I can tell it is unrelated to this issue.

geeknoid · 2019-12-12T18:14:50Z

@scalablecory I added the User-Agent and Accept headers. With User-Agent set to my app name, it hangs. I then set it to curl, and it worked fine. So that one seems to be an oddity with opentable's handling of the user agent string.

That just leaves the cert problem with apple.com and news.com.

scalablecory · 2019-12-12T18:20:18Z

@geeknoid

news.com discrepancy is possibly the browser using a private cert store -- for me, it works on Chrome but fails on Edge. HttpClient will use the platform's cert store. @bartonjs @krwq do you have any tips to help diagnose that?

For opentable.com, I ran Fiddler to grab all the headers my browser generates and added them all to HttpClient.DefaultRequestHeaders. I did not perform further analysis to see what exactly they are looking for. They may be trying to prevent screen scraping of their data, so I would like to respect that and not go further into that here.

geeknoid · 2019-12-12T18:24:07Z

@scalablecory Presumably, curl would be using the platform's cert store, right? I would expect it to have the same behavior as HttpClient.

Note that I also tried replacing SocketsHttpHandler with HttpClientHandler for the HttpClient instance. It behaved in an identical way.

bartonjs · 2019-12-12T18:31:48Z

The hostname mismatch from macOS may be the same thing that's being looked at in #666. (We let the OS tell us if the hostname mismatches, but can only really do it as a side effect of a different chain build, and it seems that some ambient state is causing failures on newer macOS versions that never failed in the past).

scalablecory · 2019-12-12T18:48:48Z

Your curl may be compiled to use a different cert store (like openssl/libressl) rather than the macos cert store (Secure Transport) which HttpClient uses.

curl, on Windows and compiled to use the Windows cert store, fails in the same way that Edge and HttpClient does.

aik-jahoda · 2019-12-12T19:41:07Z

It is reproducible on my machine.

scalablecory · 2019-12-12T20:17:17Z

Thanks @aik-jahoda

wfurt · 2019-12-13T07:43:48Z

It seems like this is distinct issue from #666 even if observed behavior is same. #666 fails only on Catalina for me with internal error -2147408889 CSSMERR_APPLETP_INVALID_EXTENDED_KEY_USAGE (Invalid Extended Key Usage for policy).
However, access to www.apple.com fails with -2147408836 CSSMERR_APPLETP_CA_PIN_MISMATCH (Certificate authority pinning mismatch)
In both cases Apple's validation function returns kSecTrustResultRecoverableTrustFailure @bartonjs

It is curious that "apple.com" is special cased in tp_verifySslOpts() https://opensource.apple.com/source/Security/Security-58286.260.20/OSX/libsecurity_apple_x509_tp/lib/tpPolicies.cpp.auto.html
This is only place I found returning CSSMERR_APPLETP_CA_PIN_MISMATCH.

bartonjs · 2019-12-13T16:11:49Z

That's frustrating. I don't see an obvious way that we can get into the hostname verifier without the additional baggage. (I also don't see where the CSSMERR_APPLETP_INVALID_EXTENDED_KEY_USAGE error is coming from, since the SSL policy does have the "if it's not present, that's fine").

catlion · 2019-12-19T12:03:04Z

We have similar issue here with an intranet application calling internal API from different network. The internal API uses a certificate issued by internal CA which is behind firewall and not available for the API consumer. Here is the sample:

        using (var handler = new HttpClientHandler())
        {
            handler.CheckCertificateRevocationList = false;
            handler.ServerCertificateCustomValidationCallback =
                HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

            using (var client = new HttpClient(handler, true))
            {
                client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
                var content = new StringContent(
                    "client_id=...&grant_type=...", 
                    Encoding.UTF8,
                    "application/x-www-form-urlencoded");

                Console.WriteLine($"Sending request to {url}");
               try
                {
                    var response = await client.PostAsync(url, content);
                    response.EnsureSuccessStatusCode();
                    Console.WriteLine(response);
                }
                catch (Exception e)
                {
                    while (e.InnerException != null)
                    {
                        e = e.InnerException;
                    }

                    Console.WriteLine(e.ToString());
                }

We tried this on netcore2.2 and netcore3.1. Operating system is RHEL7, libcurl 7.29. Error is always the same:

Unhandled exception. System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOpt
ions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken
cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cance
llationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellati
onToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, Cancellat
ionToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenS
ource cts, Boolean disposeCts)
   at TestHttpClient.Program.Main(String[] args)
   at TestHttpClient.Program.<Main>(String[] args)

wfurt · 2020-01-23T21:24:49Z

Your issue is different IMHO @catlion. Can you open a separate issue and add packet capture? I think it will be better to keep this one specific to MacOS as the underlying reasons are different.

new.com fails in Safari and I think that is a legitimate failure.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            05:5d:35:21:6f:07:b0:ae:9d:85:6f:91:b0:97:35:5c
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert SHA2 High Assurance Server CA
        Validity
            Not Before: Aug 22 00:00:00 2017 GMT
            Not After : Aug 26 12:00:00 2020 GMT
        Subject: C=US, ST=California, L=San Francisco, O=CBS Interactive Inc., CN=*.cnet.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d2:a6:5d:c9:fe:b7:62:dd:18:6e:e9:c3:ae:ea:
                    57:a3:d0:f0:d9:e3:da:22:c3:0f:67:06:27:c0:ad:
                    be:82:46:ab:1a:93:af:2f:ed:cb:6e:dd:9a:a6:b4:
                    81:6c:f7:85:34:65:20:99:42:49:e0:02:7d:13:a6:
                    4c:f0:c1:c6:cf:81:d3:7b:70:88:d9:62:68:57:6a:
                    a2:bb:b1:fd:85:8a:17:24:11:52:33:40:69:c9:7e:
                    99:2b:39:d4:0a:6a:2d:df:26:d7:73:0e:d7:d5:3a:
                    68:a1:b7:ef:06:24:48:d2:48:e7:a1:70:22:52:c2:
                    4c:69:65:72:53:51:56:67:eb:ef:c5:46:96:e0:70:
                    e4:05:c6:e0:c1:93:92:7b:38:54:df:ef:6c:8f:fc:
                    33:4d:74:e6:a5:f3:a3:f2:80:99:18:8c:e5:a3:1a:
                    d5:ba:0f:8e:7f:c0:7a:eb:71:82:1e:fb:57:45:34:
                    07:db:3d:36:3c:97:2d:88:96:3b:b2:71:31:c0:4f:
                    3f:5b:67:00:43:27:ae:59:f9:c9:9c:30:b6:10:10:
                    9a:0a:31:a2:3c:b2:5e:12:d8:73:a6:4b:b1:52:4b:
                    29:ba:47:be:02:70:f3:3a:7b:f5:03:35:4c:1f:13:
                    79:4a:50:64:83:5d:49:bf:66:09:6a:37:b7:95:7c:
                    3d:33
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:51:68:FF:90:AF:02:07:75:3C:CC:D9:65:64:62:A2:12:B8:59:72:3B

            X509v3 Subject Key Identifier:
                EF:52:A8:07:0A:01:E9:F4:CE:C0:7B:4B:8E:BE:B5:E3:AC:17:DA:88
            X509v3 Subject Alternative Name:
                DNS:*.cnet.com, DNS:cnet.com
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl3.digicert.com/sha2-ha-server-g1.crl

                Full Name:
                  URI:http://crl4.digicert.com/sha2-ha-server-g1.crl

            X509v3 Certificate Policies:
                Policy: 2.16.840.1.114412.1.1
                  CPS: https://www.digicert.com/CPS
                Policy: 2.23.140.1.2.2

            Authority Information Access:
                OCSP - URI:http://ocsp.digicert.com
                CA Issuers - URI:http://cacerts.digicert.com/DigiCertSHA2HighAssuranceServerCA.crt

            X509v3 Basic Constraints: critical
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption

There is nothing in the certificate about news.com.

That would leave apple.com last from the list. (Works from Linux and fails on MacOS) Can you confirm that @geeknoid?

rdasan · 2020-05-26T07:04:08Z

We have similar issue here with an intranet application calling internal API from different network. The internal API uses a certificate issued by internal CA which is behind firewall and not available for the API consumer. Here is the sample:

        using (var handler = new HttpClientHandler())
        {
            handler.CheckCertificateRevocationList = false;
            handler.ServerCertificateCustomValidationCallback =
                HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

            using (var client = new HttpClient(handler, true))
            {
                client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
                var content = new StringContent(
                    "client_id=...&grant_type=...", 
                    Encoding.UTF8,
                    "application/x-www-form-urlencoded");

                Console.WriteLine($"Sending request to {url}");
               try
                {
                    var response = await client.PostAsync(url, content);
                    response.EnsureSuccessStatusCode();
                    Console.WriteLine(response);
                }
                catch (Exception e)
                {
                    while (e.InnerException != null)
                    {
                        e = e.InnerException;
                    }

                    Console.WriteLine(e.ToString());
                }

We tried this on netcore2.2 and netcore3.1. Operating system is RHEL7, libcurl 7.29. Error is always the same:

Unhandled exception. System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOpt
ions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken
cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cance
llationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellati
onToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, Cancellat
ionToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenS
ource cts, Boolean disposeCts)
   at TestHttpClient.Program.Main(String[] args)
   at TestHttpClient.Program.<Main>(String[] args)

@catlion Did you ever figure out what the issue was or did you open up another issue for it? We are running into similar problems intermttantly on our internal intranet sites. Very rarely though

catlion · 2020-05-28T09:47:31Z

@rdasan we just allowed access from our server to the certificate authority through the firewall. We couldn't figure how to disable this check completely, unfortunately.

wfurt · 2020-05-28T18:10:00Z

It should be sufficient to have all intermediate certs in your X509Certificate store @catlion. That avoids needs to fetch them as needed.

danmoseley transferred this issue from dotnet/corefx Dec 12, 2019

Dotnet-GitSync-Bot added area-System.Net.Http untriaged New issue has not been triaged by the area owner labels Dec 12, 2019

davidsh removed the untriaged New issue has not been triaged by the area owner label Dec 12, 2019

scalablecory added the os-mac-os-x macOS aka OSX label Dec 12, 2019

wfurt mentioned this issue Jan 9, 2020

improve cert validation diagnostic on OSX #1513

Merged

wfurt mentioned this issue Jan 24, 2020

improve handling of kSecTrustResultRecoverableTrustFailure #2135

Merged

wfurt closed this as completed in #2135 Jan 31, 2020

wfurt mentioned this issue Apr 13, 2020

Unable to ignore SSL certificate validation, request in curl works fine #34858

Closed

karelz added this to the 5.0.0 milestone Aug 18, 2020

ghost locked as resolved and limited conversation to collaborators Dec 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unexpected HttpClient cert validation & timeout failures #805

Unexpected HttpClient cert validation & timeout failures #805

geeknoid commented Dec 12, 2019

scalablecory commented Dec 12, 2019

JohnyL commented Dec 12, 2019

scalablecory commented Dec 12, 2019

JohnyL commented Dec 12, 2019

geeknoid commented Dec 12, 2019 •

edited

Loading

geeknoid commented Dec 12, 2019

scalablecory commented Dec 12, 2019

geeknoid commented Dec 12, 2019 •

edited

Loading

scalablecory commented Dec 12, 2019

geeknoid commented Dec 12, 2019

bartonjs commented Dec 12, 2019

scalablecory commented Dec 12, 2019 •

edited

Loading

aik-jahoda commented Dec 12, 2019

scalablecory commented Dec 12, 2019

wfurt commented Dec 13, 2019

bartonjs commented Dec 13, 2019

catlion commented Dec 19, 2019 •

edited

Loading

wfurt commented Jan 23, 2020

rdasan commented May 26, 2020

catlion commented May 28, 2020

wfurt commented May 28, 2020

Unexpected HttpClient cert validation & timeout failures #805

Unexpected HttpClient cert validation & timeout failures #805

Comments

geeknoid commented Dec 12, 2019

scalablecory commented Dec 12, 2019

JohnyL commented Dec 12, 2019

scalablecory commented Dec 12, 2019

JohnyL commented Dec 12, 2019

geeknoid commented Dec 12, 2019 • edited Loading

geeknoid commented Dec 12, 2019

scalablecory commented Dec 12, 2019

geeknoid commented Dec 12, 2019 • edited Loading

scalablecory commented Dec 12, 2019

geeknoid commented Dec 12, 2019

bartonjs commented Dec 12, 2019

scalablecory commented Dec 12, 2019 • edited Loading

aik-jahoda commented Dec 12, 2019

scalablecory commented Dec 12, 2019

wfurt commented Dec 13, 2019

bartonjs commented Dec 13, 2019

catlion commented Dec 19, 2019 • edited Loading

wfurt commented Jan 23, 2020

rdasan commented May 26, 2020

catlion commented May 28, 2020

wfurt commented May 28, 2020

geeknoid commented Dec 12, 2019 •

edited

Loading

geeknoid commented Dec 12, 2019 •

edited

Loading

scalablecory commented Dec 12, 2019 •

edited

Loading

catlion commented Dec 19, 2019 •

edited

Loading