AuthenticationException on some Linux distros and root cert setups starting 3.0

[arunjvs]

Description

This is a brief summary of a longer internal conversation I had with @bartonjs after I saw outbound SSL (and therefore all HTTP) connections suddenly fail with this exception in some customer machines, after our product upgraded from 2.2 to 3.1, as a result of which, this has now been documented as a breaking change here. But this doesn't solve the problem. A result of this change is, beyond the regression itself from 2.2, is that while all non .NET processes work fine with SSL, including curl, Python and GoLang, .NET fails, depending on the configuration mentioned below, and whether the process is running as root or otherwise.

Configuration

• .NET 3.0+
• SLES 12 SP2+. This could happen with any distro in similar situations as below.
• x64
• Necessary root certs only have the BEGIN TRUSTED CERTIFICATE version. Typically in SLES, SSL_CERT_DIR = /var/lib/ca-certificates/openssl has this version.
• Necessary root certs have a BEGIN CERTIFICATE version, but it's only copy is in /etc/ssl/certs, which is not seen by .NET as it's not SSL_CERT_DIR on SLES. GoLang has done a fix to explicitly look there for SLES. This could be the only BEGIN CERTIFICATE copy since customers may not have run the distro specific sync script which updates the concatenated SSL_CERT_FILE. At least in the case of SLES, there's not a strong case to mandate this, as the script puts this notice in the generated file:

# automatically created by /usr/lib/ca-certificates/update.d/99certbundle.run. Do not edit!
#
# Use of this file is deprecated and should only be used as last
# resort by applications that do not support p11-kit or reading /etc/ssl/certs.
# You should avoid hardcoding any paths in applications anyways though. Use
# functions that know the operating system defaults instead:
#
# - openssl: SSL_CTX_set_default_verify_paths()
# - gnutls: gnutls_certificate_set_x509_system_trust(cred)
#

• Necessary root certs have a BEGIN CERTIFICATE version in the SSL_CERT_FILE, but it's not accessible to non-root users, perhaps because customers have a umask with o-rwx as a matter of security hardening the root user before they run the distro specific sync script. In case of SLES, SSL_CERT_FILE = /var/lib/ca-certificates/ca-bundle.pem and the script is update-ca-certificates.

So effectively on SLES, we are dependent solely on the concatenated bundle file's BEGIN CERTIFICATE entries, and ignoring both /etc/ssl/certs and the BEGIN TRUSTED CERTIFICATE versions in the SSL_CERT_DIR.

Regression?

Yes, this is a problem starting .NET Core 3.0.

Other information

Multiple fixes are possible:

• Optimal: Generalize chain building. Chain will build from leaf to root cert by maximizing the purpose of the overall chain, first trying a general cert and then a more specific one, while keeping track of the intersection of purposes as the effective chain purpose. Users of the chain may use it or not based on purpose. IMHO, this should not break either the purpose behind this regression as well as the cases discussed in this issue.
• Spill: Another modification to above option is to allow users to specify a set of purposes for every chain building run with a default retaining current behavior. Http/Ssl layers can override it.
• Low-hanging: Explicitly add /etc/ssl/certs to the lookup directory, on top of SSL_CERT_DIR.

It appears the low-hanging fix is a reasonable one to take. It's also the least risk IMO, as the deprecation notice in ca-bundle.pem in the case of SLES is explicitly telling you to read that directory; as are other language runtimes.

Sample exception causing HTTP failures:

 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
Notify danmosemsft if you want to be subscribed.

[bartonjs]

Suggested fix:

In CachedSystemStoreProvider, change ProcessFile to return a bool. True means that we ever saw TryReadX509PemNoAux/TryReadX509Der return true (we made it into the while loop), false means we never did.

In LoadMachineStores replace

            if (rootStorePath != null && rootStorePath.Exists)
            {
                newDirTime = rootStorePath.LastWriteTimeUtc;
                foreach (FileInfo file in rootStorePath.EnumerateFiles())
                {
                    ProcessFile(file);
                }
            }

with something like

            if (rootStorePath != null && rootStorePath.Exists)
            {
                newDirTime = rootStorePath.LastWriteTimeUtc;
                bool hadCerts = false;
                bool hadFiles = false;

                foreach (FileInfo file in rootStorePath.EnumerateFiles())
                {
                    hadCerts |= ProcessFile(file);
                    hadFiles = true;
                }

               if (hadFiles && !hadCerts && SSL_CERT_DIR isn't set)
               {
                    DirectoryInfo etcSslCerts = new DirectoryInfo("/etc/ssl/certs");
                   
                    if (etcSslCerts.Exists())
                    {
                        foreach (FileInfo file in rootStorePath.EnumerateFiles())
                        {
                            hadCerts |= ProcessFile(file);
                        }

                        if (hadCerts)
                        {
                            promote etcSslCerts as the replacement directory
                        }
                    }
            }

It needs to be a little different, since it should only do the directory replacement on first load. Which probably means that SafeOpenRootDirectoryInfo method needs to be where the sanity test happens. And rather than duplicate the definition of the environment variable that gets used, change CryptoNative_GetX509RootStorePath to out a value that indicates if it was from OPENSSL_DIR or the environment variable, and plumb that through.

Bonus points for checking if /etc/ssl/certs and the CryptoNative_GetX509RootStorePath value are the same thing after symlink resolution, and avoiding doing the work/repointing.

[arunjvs]

Why hadFiles and why !hadCerts, before you see /etc/ssl/certs? Won't a union irrespective, just work? What is GoLang/Python doing for instance?

There may always be some usable (i.e. BEGIN CERTIFICATE) certs in OPENSSL_DIR and some usable in /etc/ssl/certs. In the case of SLES perhaps almost all usable are in /etc/ssl/certs. Would just one valid cert in OPENSSL_DIR stop us from loading /etc/ssl/certs?

[bartonjs]

• If the environment variable is set, we won't do anything other than what the variable says to do (it may be intentionally reducing the support list).
• If the directory that the library was told to use is empty, then it was told to be empty and we should accept that.
• If the distro-specific tooling is being followed, there shouldn't be a mix of styles, so a directory either contains usable contents or it doesn't.

What is GoLang/Python doing for instance?

GoLang hard-codes /etc/ssl/certs, but that's always what they've done. We've always used whatever directory OpenSSL was told to use, by asking the library what it was configured to do; but since that directory started being incompatible with our logic on some distros we're looking for the least surprising change. So the platform then changes to "if the default OpenSSL trusted authorities directory only contains files in an unsupported format, we fall back to the fixed path /etc/ssl/certs for determining the system trust list."

[arunjvs]

I agree with point 1 and point 3, but point 2 seems a bit weak, unless that empty directory is from the environmental variable (which is point 1).

How is an empty directory really different from a single junk file or a single BEGIN TRUSTED CERTIFICATE file present in the directory for the purpose of this decision? And what if someone just keeps this directory empty (perhaps he considers them redundant)?

At this level it appears to be an obscure assumption making the runtime look apparently non-deterministic and perhaps hard to debug.

[arunjvs]

Also, I just found this clear statement from this SUSE KB. Quoting:

The last ca certificate in chain needs to be in /etc/ssl/certs-> /var/lib/ca-certificates/pem/ symlinked by his hash or base64 encoded in bundle file /var/lib/openssl/ca-certificates which is deprecated.

Oddly the path to the OpenSSL bundle file/directory is wrong there. I've sent them a note.

[jeffhandley]

I agree with point 1 and point 3, but point 2 seems a bit weak, unless that empty directory is from the environmental variable (which is point 1).

From what I understand, this detail won't interfere with the reported scenario that was leading to unexpected behavior, and this would align with the principle of least astonishment.

Oddly the path to the OpenSSL bundle file/directory is wrong there. I've sent them a note.

Thanks for doing that, @arunjvs.