Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problems connecting to SQL server when using some versions of Microsoft.Data.SqlClient library (e.g. 4.1.0) #7650

Open
1 task done
patrick-smergut-octopus opened this issue Jul 5, 2022 · 1 comment
Open
1 task done

Problems connecting to SQL server when using some versions of Microsoft.Data.SqlClient library (e.g. 4.1.0) #7650

patrick-smergut-octopus opened this issue Jul 5, 2022 · 1 comment
Labels
kind/bug This issue represents a verified problem we are committed to solving

Comments

@patrick-smergut-octopus
Copy link

Team

  • I've assigned a team label to this issue

Severity

One customer impacted so far, but could impact others. Workaround isn't good because it means vulnerability to MITM attacks..

Version

Found in 2022.2.6971 after upgrading from 2022.1.2232

Latest Version

No response

What happened?

Octopus was upgraded from 2022.1.2232 to 2022.2.6971 using the Docker base image from Docker Hub. Octopus image runs in ECS in AWS and connects to an RDS SQL Server Instance. This uses the rds-ca-2019 Certificate Authority for encryption.

Connection string prior to the upgrade had ;Encrypt=True;TrustServerCertificate=False.

Unable to start Octopus 2022.2.6971 with the error: System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.

It looks like this may have to do with the SqlClient library changing versions from 2.1.0 to 4.1.0, and there is likely a bug in 4.1.0 which was fixed in 5.0.0: dotnet/SqlClient#1402

Reproduction

Reported by user [internal]: https://octopus.zendesk.com/agent/tickets/92367

Error and Stacktrace

EVENTS 1656420612905                 Connection test to "master" timed out after trying 60 times over 00:01:00.3592593.                1656420612737
EVENTS 1656420612905                 Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 35 - An internal exception was caught)   1656420612737
EVENTS 1656420612905                 ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.          1656420612737
EVENTS 1656420612905                    at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)            1656420612737
EVENTS 1656420612905                    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)     1656420612738
EVENTS 1656420612905                    at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)     1656420612738
EVENTS 1656420612905                    at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)                1656420612738
EVENTS 1656420612905                    at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)                1656420612738
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SNI.SNITCPHandle.EnableSsl(UInt32 options)                1656420612738
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) 1656420612738
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)        1656420612738
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)       1656420612738
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)                1656420612738
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)          1656420612738
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)                1656420612738
EVENTS 1656420612905                    at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in ./source/Octopus.Core/RelationalStorage/SqlServerManager.cs:line 360 1656420612738
EVENTS 1656420612905                 --- End of stack trace from previous location ---   1656420612738
EVENTS 1656420612905                    at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in ./source/Octopus.Core/RelationalStorage/SqlServerManager.cs:line 384 1656420612738
EVENTS 1656420612905                    at Octopus.Server.Commands.DatabaseCommand.Start() in ./source/Octopus.Server/Commands/DatabaseCommand.cs:line 113       1656420612738
EVENTS 1656420612905                 ClientConnectionId:1b6a04c6-18fd-4b9f-907f-d2b4a3af2adc        1656420612738
EVENTS 1656420612905                 Error Number:-2146893019,State:0,Class:20        1656420612738
EVENTS 1656420612905                 This error could happen when the SQL Server certificate is not trusted. Try setting TrustServerCertificate=true on the connection string: https://github.com/dotnet/SqlClient/issues/1402                1656420612738
EVENTS 1656420612905                 We are having trouble working with your SQL Database. This will usually be something you can fix. If not please contact support@octopus.com for help. Here are the error details: SQL Error -2146893019 - A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 35 - An internal exception was caught)      1656420612741
EVENTS 1656420612905                 Microsoft.Data.SqlClient.SqlException    1656420612741
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
              1656420612741
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
              1656420612741
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
              1656420612741
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
              1656420612741
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
              1656420612741
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
              1656420612741
EVENTS 1656420612905                    at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in SqlServerManager.cs:line 360         1656420612741
EVENTS 1656420612905                    at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in SqlServerManager.cs:line 384         1656420612741
EVENTS 1656420612905                    at Octopus.Server.Commands.DatabaseCommand.Start() in DatabaseCommand.cs:line 113  1656420612741
EVENTS 1656420612905                 --Inner Exception--          1656420612741
EVENTS 1656420612905                 The remote certificate was rejected by the provided RemoteCertificateValidationCallback.     1656420612741
EVENTS 1656420612905                 System.Security.Authentication.AuthenticationException              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)
              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
              1656420612742
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SNI.SNITCPHandle.EnableSsl(UInt32 options)                1656420612742
EVENTS 1656420612905                 Octopus.Shared.ControlledFailureException: We are having trouble working with your SQL Database. This will usually be something you can fix. If not please contact support@octopus.com for help. Here are the error details: SQL Error -2146893019 - A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 35 - An internal exception was caught)                1656420612742
EVENTS 1656420612905                 Microsoft.Data.SqlClient.SqlException    1656420612742
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
              1656420612742
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
              1656420612742
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
              1656420612742
EVENTS 1656420612905                    at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
              1656420612742
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
              1656420612742
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
              1656420612742
EVENTS 1656420612905                    at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in SqlServerManager.cs:line 360         1656420612742
EVENTS 1656420612905                    at Octopus.Core.RelationalStorage.SqlServerManager.TestConnection(Boolean throwException, String databaseName, Nullable`1 timeout) in SqlServerManager.cs:line 384         1656420612742
EVENTS 1656420612905                    at Octopus.Server.Commands.DatabaseCommand.Start() in DatabaseCommand.cs:line 113  1656420612742
EVENTS 1656420612905                 --Inner Exception--          1656420612742
EVENTS 1656420612905                 The remote certificate was rejected by the provided RemoteCertificateValidationCallback.     1656420612742
EVENTS 1656420612905                 System.Security.Authentication.AuthenticationException              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)
              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
              1656420612742
EVENTS 1656420612905                    at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
              1656420612742
EVENTS 1656420612905                    at Microsoft.Data.SqlClient.SNI.SNITCPHandle.EnableSsl(UInt32 options)                1656420612742
EVENTS 1656420612905                    at Octopus.Server.Commands.DatabaseCommand.Start() in ./source/Octopus.Server/Commands/DatabaseCommand.cs:line 234       1656420612742
EVENTS 1656420612905                    at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in ./source/Octopus.Shared/Startup/AbstractCommand.cs:line 100                1656420612742
EVENTS 1656420612905                    at Octopus.Shared.Startup.OctopusProgram.Start(ICommandRuntime commandRuntime) in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 500       1656420612742
EVENTS 1656420612905                    at Octopus.Shared.Startup.ConsoleHost.Run(Action`1 start, Action shutdown) in ./source/Octopus.Shared/Startup/ConsoleHost.cs:line 34              1656420612742
EVENTS 1656420612905                    at Octopus.Shared.Startup.OctopusProgram.RunHost(ICommandHost host) in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 217   1656420612742
EVENTS 1656420612905                    at Octopus.Shared.Startup.OctopusProgram.Run() in ./source/Octopus.Shared/Startup/OctopusProgram.cs:line 168   1656420612742

More Information

It looks like the version of Nevermore and therefore the SqlClient were changed between Octopus 2022.1.2232 and 2022.2.6971:

2022.1.2322 - 16.3.5 Nevermore, 2.1.4 SqlClient
2022.2.6971 - 18.1.0 Nevermore, 4.1.0 SqlClient

Workaround

Modify connection string - set TrustServerCertificate=True
@patrick-smergut-octopus patrick-smergut-octopus added kind/bug This issue represents a verified problem we are committed to solving state/triage labels Jul 5, 2022
@cheenamalhotra
Copy link

Have you tried upgrading to v4.1.1 that includes fix for this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug This issue represents a verified problem we are committed to solving
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@droyad @cheenamalhotra @patrick-smergut-octopus