Error tls: handshake failure on try to connect to mysql 5.7 with SSL client certs

[randreev1321]

Issue description

Failed to connect to mysql server version 5.7.44 using SSL client certificates. An error is returned. Everything works correctly with mysql server version 8.4.2.

Example code

	rootCertPool := x509.NewCertPool()
	pem, err := os.ReadFile("certs/ca-cert.pem")
	if err != nil {
		log.Fatal(err)
	}
	if ok := rootCertPool.AppendCertsFromPEM(pem); !ok {
		log.Fatal("Failed to append PEM.")
	}
	clientCert := make([]tls.Certificate, 0, 1)
	certs, err := tls.LoadX509KeyPair("certs/client-cert.pem", "certs/client-key.pem")
	if err != nil {
		log.Fatal(err)
	}
	clientCert = append(clientCert, certs)
	mysql.RegisterTLSConfig("custom", &tls.Config{
		RootCAs:      rootCertPool,
		Certificates: clientCert,
		InsecureSkipVerify: true,
	})
	db, err := sql.Open("mysql", "user@tcp(localhost:3306)/test?tls=custom")
	if err != nil {
		log.Fatal(err)
	}

Error log

MySQL connect error: remote error: tls: handshake failure

Configuration

v1.8.1

Go version: go version go1.22.6 darwin/arm64

Server version: MySQL 5.7.44

Server OS: oraclelinux7

[methane]

https://dev.mysql.com/blog-archive/ssltls-improvements-in-mysql-5-7-10/

MySQL 5.7 may not support TLS v1.2. You should check your MySQL configuration and tweak MinVersion if TLS v1.2 is not supported.

[randreev1321]

Support of TLS v1.2 in mysql 5.7.44 enabled by default.

I tried specifying MinVersion TLS v1.2, it didn't do anything.

[methane]

How did you confirm it on your MySQL server?
Supported TLS version is depends on how the server binary is built. See the link I wrote above.

[randreev1321]

Here is my connection from console

mysql -u root -p -h 192.168.8.53 --ssl-ca=ca.pem --ssl-cert=client-cert.pem --ssl-key=client-key.pem
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.44 MySQL Community Server (GPL)

Here is server settings and my connection status:

mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+------------------------------+
| Variable_name           | Value                        |
+-------------------------+------------------------------+
| innodb_version          | 5.7.44                       |
| protocol_version        | 10                           |
| slave_type_conversions  |                              |
| tls_version             | TLSv1,TLSv1.1,TLSv1.2        |
| version                 | 5.7.44                       |
| version_comment         | MySQL Community Server (GPL) |
| version_compile_machine | x86_64                       |
| version_compile_os      | Linux                        |
+-------------------------+------------------------------+
8 rows in set (0.03 sec)

mysql> SELECT 
    -> variable_value AS tls_version,
    -> processlist_user AS user,
    -> processlist_host AS host
    -> FROM performance_schema.status_by_thread sbt
    -> JOIN performance_schema.threads t
    -> ON (t.thread_id = sbt.thread_id)
    -> WHERE variable_name = 'Ssl_version'
    ->  ORDER BY tls_version;
+-------------+------+------------+
| tls_version | user | host       |
+-------------+------+------------+
| TLSv1.2     | root | 172.18.0.1 |
+-------------+------+------------+
1 row in set (0.02 sec)

[methane]

The next suspect is the cipher suite.
Compare MySQL's ssl_cipher_list with Go's default cipher suites.

[methane]

No information about cipher suites?
I will close this issue if not enough information to investigate or reproduce is provided.

[evanelias]

I'm experiencing the same problem. Still investigating, but so far it appears to be caused by Go 1.22+ default removing the cipher suites which used RSA based key exchange. (See https://go.dev/doc/go1.22#library and scroll to crypto/tls: "By default, cipher suites without ECDHE support are no longer offered by either clients or servers during pre-TLS 1.3 handshakes")

Here's what I've found so far when testing against a stock "mysql:5.7" Docker image:

With a Go 1.22+ client program and a MySQL 5.7 server, the "tls: handshake failure" seems to happen when simply using tls=preferred in DSN. In other words, a custom TLS config isn't even necessary to repro this. (For background: upon database server initialization, MySQL 5.7+ automatically creates a self-signed cert and supports TLS out of the box. But MySQL 8.0+ uses a better/newer list of cipher suites, so it is not affected by this problem. And MySQL 5.6 and older did not support TLS without manual configuration.)

Using tls=preferred,allowFallbackToPlaintext=true in DSN does not appear to fix it. This still gives "tls: handshake failure", suggesting fallback to plaintext may not be working if there's no matching cipher suite? (Also possible something else in my program is breaking this, I will try to confirm in a more isolated test/reproduction soon.)

Setting env var GODEBUG='tlsrsakex=1' in the environment of the client program does fix the problem. See https://go.dev/doc/godebug#go-122

Alternatively, using "go 1.21" in go.mod and recompiling also fixes it, since this reverts the standard library.

Percona Server for MySQL 5.7 does not exhibit this error at all, presumably because they use a better cipher suite list.

Given all the above findings, I expect this is also fixable by setting tls.Config.CipherSuites to include the IDs from tls.CipherSuites() plus the four removed from there in 1.22: tls.TLS_RSA_WITH_AES_128_CBC_SHA, tls.TLS_RSA_WITH_AES_256_CBC_SHA, tls.TLS_RSA_WITH_AES_128_GCM_SHA256, tls.TLS_RSA_WITH_AES_256_GCM_SHA384. I will test this soon to confirm.

That all said, it would be good if there was a clean way for tls=preferred to continue working as-is with MySQL 5.7, without needing GODEBUG or a custom tls.Config. And/or ideally allowFallbackToPlaintext=true could solve this.

[evanelias]

I did more research and testing across different MySQL and MariaDB versions.

1. The cipher suite problem with MySQL 5.7 depends on how the server binary is built, in terms of OpenSSL version.

• In the "official" mysql:5.7 Docker image, OpenSSL 1.0.2k is used, and elliptic curve cipher suites do not work with any client (including the official mysql CLI client).
• This is likely the case with all precompiled 5.7 builds from Oracle, judging by the verified bug and Percona Server's fix.
• Meanwhile go-sql-driver/mysql's test suite on GitHub Actions uses actions-setup-mysql, which has custom-compiled server binaries using OpenSSL 1.1.1. That seems to fix the elliptic curve ciphers, which explains why the tests here aren't failing on 5.7. The tests successfully connect using ECDHE-RSA-AES128-GCM-SHA256.
• It's possible that some Linux distributions/packages also compile MySQL 5.7 in a similar way that fixes this problem. But my assumption would be that a large portion of 5.7 installations do use Oracle's builds, meaning this is likely to be a somewhat prevalent issue among MySQL 5.7 holdouts.

2. In the standard mysql CLI client, --ssl-mode=preferred actually does not solve issues with cipher suite mismatches, nor TLS version mismatches. In those cases, a hard error is simply returned, and there's no automatic retry/fallback to a plaintext connection. It appears that --ssl-mode=preferred just means "if the server isn't configured to use TLS at all, use a plaintext connection."

• go-sql-driver/mysql's behavior with tls=preferred matches that of the standard mysql CLI client: in the event of a cipher suite mismatch or TLS version mismatch, you just get an error.
• I hadn't realized that the driver's tls=preferred automatically sets allowFallbackToPlaintext=true too. The README was a bit confusing here on that and the description of Add API to express like a --ssl-mode=PREFERRED MySQL client #1370 implied it was more extensive fallback logic for TLS version, when it isn't. But I now understand that this all just matches the mysql CLI behavior, which is reasonable.
• For MySQL 5.7 specifically, "preferred" is rarely helpful, because the server in 5.7+ automatically supports TLS by using a self-signed cert when the server is first initialized. So unless the server was specifically configured not to use TLS, "preferred" will attempt TLS in 5.7. And then in the event of a cipher suite mismatch, you just get a hard error.

3. When testing MariaDB, the cipher suite issue only affects only MariaDB 10.1 (EOL 4+ years) or older.

• Prebuilt MariaDB 10.1 server binaries use OpenSSL 1.0.2n which is old.
• MariaDB 10.2+ uses 1.1.1+ which is fine.
• MariaDB only added automatic server-side TLS via self-signed certs very recently, in MariaDB 11.4+ earlier this year. So "preferred" mode is much more useful with MariaDB, as most MariaDB servers don't have TLS enabled unless the administrator specifically configured it.

Summary: In a sense, there's no "bug" here. It is arguably the calling code's responsibility to set cipher suites and minimum TLS version. Instead just some potential feature requests:

• Ability to configure cipher suites and TLS version in DSN. But I understand there may be a desire to move away from DSNs anyway, so maybe not worthwhile.

• Automatic plaintext fallback logic in cases of cipher suite or TLS version related errors. But that's a fair amount of engineering effort just to support some old EOL database server versions.

One easy win could be some minor documentation improvements, describing how it's up to the caller to configure cipher suites and min TLS version if connecting to older servers. I'm happy to submit a PR next week, which adds a couple sentences to the README, if this would be helpful.