Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yaSSL/Schannel TLS 1.2 workaround causing unintended TLS downgrade #1132

Closed
lostatredrock opened this issue Feb 9, 2022 · 2 comments
Closed
Assignees
Labels

Comments

@lostatredrock
Copy link
Contributor

lostatredrock commented Feb 9, 2022

Software versions
MySqlConnector version: 1.3.14 (issue also exists in latest version 2.1.5)
Server type (MySQL, MariaDB, Aurora, etc.) and version: MySQL 8.0.27
.NET version: 5.0.14
(Optional) ORM NuGet packages and versions: Pomelo.EntityFrameworkCore.MySql 5.0.3

Describe the bug
We have been experiencing connectivity issues in which our application server looses the ability to establish new connections to the database server. Reviewing the logs we see repeats of the message: "The server doesn't support the client's specified TLS versions."

Stepping back to the start of this connectivity issue we are seeing the message "Session1.944 failed negotiating TLS; falling back to TLS 1.1" logged as a warning.

Looking into the connector code I see that this is part of a work around to an issue establishing TLS 1.2 connections between a windows server a yaSSL based MySQL instance (comment). In this case our MySQL instance is OpenSSL based and we are experiencing an IOException due to an unknown cause (we are still investigating the cause of the IO Exception).

Stepping into the connect code the crtiteria for retry is here and the downgrade code is here . The effect of this is that if you are running a windows server and experience an IO Exception when establishing a new database connection your pool will permenatly fall back to TLS 1.1. As far as I can see there is no way to override this fallback. Specifying TLS 1.2 in the connection string will get overidden and specifying nothign (and leaving it to the OS) will also make the application subject to this downgrade.

In our case the downgrade crashed the application until restart because we disallow TLS 1.0 and 1.1 connections at the OS level on the app and db servers.

Exception

The fallback is initiated when this exception occurs:

`System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine..
---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory1 buffer, SocketFlags socketFlags, CancellationToken cancellationToken)
at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory1 buffer, CancellationToken cancellationToken)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)
at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1338
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken)
at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 470
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetExistingTaskResult(Task1 task, TResult result)
at MySqlConnector.Core.ServerSession.OpenTcpSocketAsync(ConnectionSettings cs, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1045
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetExistingTaskResult(Task1 task, TResult result)
at System.Net.Sockets.TcpClient.CompleteConnectAsync(Task task)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.TrySetResult()
at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pNativeOverlapped)
--- End of stack trace from previous location ---

--- End of inner exception stack trace ---
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1346
at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 470`

which then leads to this exception occuring on all future connection attempts:

Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: The maximum number of retries (2) was exceeded while executing database operations with 'MySqlRetryingExecutionStrategy'. See the inner exception for the most recent failure.
---> MySqlConnector.MySqlException (0x80004005): The server doesn't support the client's specified TLS versions.
---> System.ComponentModel.Win32Exception (0x80090331): The client and server cannot communicate, because they do not possess a common algorithm.
at System.Net.SSPIWrapper.AcquireCredentialsHandle(ISSPIInterface secModule, String package, CredentialUse intent, SCHANNEL_CRED* scc)
at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCHANNEL_CRED* secureCredential)
at System.Net.Security.SslStreamPal.AcquireCredentialsHandleSchannelCred(X509Certificate certificate, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer)
at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint)
at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output)
at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan`1 incomingBuffer)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1346
at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1388
at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 470
at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(String logMessage, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 435
at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 139
at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 139
at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int32 startTickCount, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 902
at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 435
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlRelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)

snipped to exclude internal code which can be anything requesting a DB connection

Expected behavior
There should be a way to override this behavior. I would think a seperate NotUsingYASSL flag would probably be the best way to acccomplish this without breaking the work around for users who do need the fallback.

Additional context
We are looking into the source of the IO Exception to try and work around the problem. But a single random IO Exception (which is not unheard of) should not disable TLS 1.2 permenantly.

@bgrainger
Copy link
Member

As far as I can see there is no way to override this fallback. Specifying TLS 1.2 in the connection string will get overidden

That is definitely not intentional.

The intended way to avoid TLS fallback would be to specify TlsVersion=Tls12,Tls13 (or similar) in the connection string. If TLS 1.1 is disallowed, MySqlConnector should not use it as a fallback.

@bgrainger bgrainger self-assigned this Feb 9, 2022
@bgrainger bgrainger added the bug label Feb 9, 2022
lostatredrock added a commit to lostatredrock/MySqlConnector that referenced this issue Feb 10, 2022
Change the logic used to determine when a connection attempt is failing
due to a lack of support for TLS 1.2+ in yaSSL-based MySQL Server so
that a downgrade no longer occurs if the user has explicitly configured
supported TLS Version(s) in the connection string.

Signed-off-by: Andrew Nagel <andrew.nagel@gmail.com>
lostatredrock added a commit to lostatredrock/MySqlConnector that referenced this issue Feb 10, 2022
Change the logic used to determine when a connection attempt is failing
due to a lack of support for TLS 1.2+ in yaSSL-based MySQL Server so
that a downgrade no longer occurs if the user has explicitly configured
supported TLS Version(s) in the connection string.

Signed-off-by: Andrew Nagel <andrew.nagel@gmail.com>
Signed-off-by: Andrew Nagel <andrew.nagel.ctr@dot.gov>
@bgrainger
Copy link
Member

Fixed in 2.1.6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

2 participants