Cannot connect to AWS RDS with new CA certs using VerifyFull

[sp-jcberleur]

Software versions
MySqlConnector version: 2.3.6
Server type (MySQL, MariaDB, Aurora, etc.) and version: MariaDB 10.6.17
.NET version: .net6
(Optional) ORM NuGet packages and versions:

Describe the bug
I tried to update your RDS instance from the old rds-ca-2019 CA to the new rds-ca-ecc384-g1 CA

Unfortunately it did not work, it gives the exception The remote certificate was rejected by the provided RemoteCertificateValidationCallback

Exception

[ERROR]	MySqlConnection	Session 1.1 couldn't initialize TLS connection
System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, MySqlConnection connection, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1471
[DEBUG]	MySqlConnection	Session 1.1 closing stream/socket
[DEBUG]	MySqlConnection	Session 1.1 closing stream/socket
[ERROR]	Program	Connection failed
MySqlConnector.MySqlException (0x80004005): SSL Authentication Error
 ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, MySqlConnection connection, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1471
   at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, MySqlConnection connection, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1509
   at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 523
   at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 428
   at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 433
   at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 111
   at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144
   at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1054
   at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 554

var cs = new MySqlConnector.MySqlConnectionStringBuilder
{
    Server = "XXXXXX.YYYYYYY.us-west-2.rds.amazonaws.com"
    UserID = "user",
    Password = "password",
    TlsVersion = "TLS 1.1,TLS 1.2,TLS 1.3",
    SslMode = MySqlSslMode.VerifyFull,
    SslCa = Environment.GetEnvironmentVariable("MYSQL_CA_CERTIFICATE_FILE") ?? "",
};
await using (var connection = new MySqlConnection(cs.ConnectionString))
{
        await connection.OpenAsync();
}

Expected behavior
I expect that the certificate present by AWS is validated by the CA bundle provided by AWS and the remote name matches

Additional context
RDS bundle https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

I performed some debugging using my own RemoteCertificateValidationCallback and the current error seems to be because of X509RevocationMode.Online

RemoteCertificateValidationCallback rcbPolicyErrors = RemoteCertificateChainErrors
rcbChain is not null && caCertificateChain is not null
caCertificateChain.Build(rcbChain.ChainElements[^1].Certificate) True
caCertificateChain.ChainStatus.Length 0
rcbChain.ChainStatus.Length = 2
rcbChain.ChainStatus[0].Status = RevocationStatusUnknown
rcbChain.ChainStatus[1].Status = OfflineRevocation
rcbChain.ChainElements[0].SubjectName C=US, S=Washington, L=Seattle, O=Amazon.com, OU=RDS, CN=XXXXXX.YYYYYYY.us-west-2.rds.amazonaws.com
rcbChain.ChainElements[0].IssuerName L=Seattle, CN=Amazon RDS us-west-2 Subordinate CA ECC384 G1.A.1, S=WA, OU=Amazon RDS, O="Amazon Web Services, Inc.", C=US
rcbChain.ChainElements[1].SubjectName L=Seattle, CN=Amazon RDS us-west-2 Subordinate CA ECC384 G1.A.1, S=WA, OU=Amazon RDS, O="Amazon Web Services, Inc.", C=US
rcbChain.ChainElements[1].IssuerName L=Seattle, CN=Amazon RDS us-west-2 Root CA ECC384 G1, S=WA, OU=Amazon RDS, O="Amazon Web Services, Inc.", C=US
rcbChain.ChainElements[2].SubjectName L=Seattle, CN=Amazon RDS us-west-2 Root CA ECC384 G1, S=WA, OU=Amazon RDS, O="Amazon Web Services, Inc.", C=US
rcbChain.ChainElements[2].IssuerName L=Seattle, CN=Amazon RDS us-west-2 Root CA ECC384 G1, S=WA, OU=Amazon RDS, O="Amazon Web Services, Inc.", C=US
RemoteCertificateValidationCallback rcbPolicyErrors = RemoteCertificateChainErrors
RemoteCertificateValidationCallback result False

[bgrainger]

Does the problem only happen when using the "ECC384" certs, not "RSA2048", or is that irrelevant to this issue?

Are you requesting that MySqlConnector set X509Chain.ChainPolicy.RevocationMode to X509RevocationMode.NoCheck (or .Offline) to work around this issue?

If so, I can't decide if that should be:

• default behavior
• opt-in with a new connection string option
• NoCheck for MySqlSslMode.VerifyCA, but Online for VerifyFull
• NoCheck if SslCA is specified (under the theory that clients should remove revoked certificates from the list of provided CA certificates and if they're providing a revoked cert, it's intentional)
• something else

[sp-jcberleur]

it happens with both the ecc384 and rsa2048 CAs

after more testing the behavior seems to only apply when you add the RDS CA certs to the global trust store (on linux update-ca-certificates)
probably because if (caCertificateChain.ChainStatus[0].Status == X509ChainStatusFlags.UntrustedRoot) returns false in this case

[bgrainger]

Correct: RemoteCertificateChainErrors is only cleared if UntrustedRoot is the only error, and if we can manually clear that ourselves by verifying the root.

Or are you saying that when that happens, rcbChain.ChainStatus.Length is 3, the first error is UntrustedRoot, that's cleared, and the other two problems are ignored? That would be unintentional and undesirable.

[sp-jcberleur]

when the RDS CA certs are added to the global trust store caCertificateChain.ChainStatus is empty (the root is trusted), but rcbPolicyErrors is still RemoteCertificateChainErrors because of the revocation status

rcbChain.ChainStatus.Length = 2
rcbChain.ChainStatus[0].Status = RevocationStatusUnknown
rcbChain.ChainStatus[1].Status = OfflineRevocation