Breakage due to enabling TLS 1.3 by default in 3.6.0

[gilles-peskine-arm]

In Mbed TLS 3.6.0, we enabled TLS 1.3 support in by default. Unfortunately, this broke many applications that were working fine with Mbed TLS ≤3.5.x in the default configuration. The goal of this issue is to discuss how we'll resolve this in Mbed TLS 3.6.1. Continue reading for a list of known problems and workarounds for users of Mbed TLS 3.6.0.

Known problems

General workaround: you can disable TLS 1.3 support at compile time by commenting out or removing #define MBEDTLS_SSL_PROTO_TLS1_3 from include/mbedtls/mbedtls_config.h.

General workaround: you can disable TLS 1.3 support at runtime by calling mbedtls_ssl_conf_max_tls_version(ssl_config, MBEDTLS_SSL_VERSION_TLS1_2) before mbedtls_ssl_setup.

PSA initialization

Symptom: mbedtls_ssl_handshake fails with an internal error.

Cause: the TLS 1.3 code uses the PSA crypto subsystem, which needs to be initialized with psa_crypto_init.

Workaround: call psa_crypto_init before starting a TLS handshake.

You only need to call psa_crypto_init() once for the whole application; calling it a second time has no effect. If you are using a memory leak detector, call mbedtls_psa_crypto_free() in your cleanup code to clear the resources allocated by psa_crypto_init().

Further discussion: #9072, #9210

Mandatory server authentication

Symptom: In a TLS client, mbedtls_ssl_conf_authmode(ssl_config, MBEDTLS_SSL_VERIFY_NONE) or mbedtls_ssl_conf_authmode(ssl_config, MBEDTLS_SSL_VERIFY_OPTIONAL) works only for TLS 1.2 connections, not for TLS 1.3 connections.

Cause: server authentication is mandatory in TLS 1.3 clients.

Workaround: none

See also: #9072, #9079, #7075, #9210

No CA callback support

Symptom: mbedtls_ssl_conf_ca_cb has no effect with TLS 1.3 connections.

Cause: missing feature.

Workaround: none

Further discussion: #7079

Key store size

Symptom: Opening many TLS connections fails with an out-of-memory error, despite having not running out of RAM.

Cause: TLS 1.3 uses the PSA crypto subsystem, which places keys in a fixed-size array.

Workaround: set the compile-time option MBEDTLS_PSA_KEY_SLOT_COUNT to at least the number of simultaneous handshakes (in the default configuration — activating MBEDTLS_USE_PSA_CRYPTO consumes more keys, even after the handshake).

Further discussion: #9216

MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET error code

Symptom: When a client established a TLS 1.3 session, post-handshake, the server may send NewSessionTicket messages. The reception of such message is notified to the client application by TLS APIs (mostly mbedtls_ssl_read/write()) returning the MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET error code. This error code is a "non-fatal" error code but a <=3.5 working application client is likely to handle it as a fatal error and discard the connection.

Cause: Establishment of a TLS 1.3 session instead of a TLS 1.2 session due to TLS 1.3 being enabled is the default configuration.

Workaround: Handle MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET as a non-fatal error code like MBEDTLS_ERR_SSL_WANT_READ or MBEDLTS_ERR_SSL_WANT_WRITE.

Further discussion: #8749

Some TLS 1.2 clients cannot conect to an Mbed TLS 3.6.0 server

Symptom: TLS 1.2-only clients that send a ClientHello message with legacy_compression_methods work with Mbed TLS 3.5, which rejects compression, but understands the offer). With Mbed TLS 3.6.0 in the default configuration, the connection fails because the server rejects the ClientHello packet as malformed for TLS 1.3 in a way that bypasses the fallback to TLS 1.2.

Workaround: configure clients not to offer compression methods.

Further discussion: #9243

[gilles-peskine-arm]

Proposal for 3.6.1: Make it all work

• Call psa_crypto_init when starting a TLS 1.3 handshake.
  • Workload: trivial.
  • Downside: breaks applications that insist on freeing all memory before they exit: they will now have to call mbedtls_psa_crypto_free. I think that's an acceptable thing to document.
• Implement server authentication bypass.
  • Workload: S?
  • Downside: gives users a way of shooting themselves in the foot, which the TLS 1.3 ecosystem tries to avoid. That's unfortunate, but I think this should not depend on the TLS version, and instead we should wait until 4.0 to remove client-side authmode. You can make authentication optional by writing a callback (except in 3.6.0 because it's not implemented yet) if you know what you're doing.
• Implement the CA callback.
  • Workload: S?
• Dynamic PSA key store
  • Workload: S-M

[mpg]

All the issues causing the failures mentioned here are addressed in Mbed TLS 3.6.1 which as now been release. When further work remains, it is tracked in separate issues. So, I'm closing this one.