[mbedtls3] TLS 1.3 connection is unstable and sometime break in multi-thread env

[Taowyoo]

Description

TLS 1.3 connection is unstable(sometimes break) in multi-thread env:

1. run server in one thread/process
2. spawn multiple thread/process of client to connect server continuously
3. Sometimes, the TLS connection will break, the error is random

Reproduce error

This could be reproduced on both Rust side and C side.

Rust

1. Checkout code at https://github.com/fortanix/rust-mbedtls/tree/yx/debug_tls13_multithread_bug , this is based on https://github.com/fortanix/rust-mbedtls/tree/mbedtls_v0.11.0 .
2. Run bash script multithread_test_rust.sh

C

This not only could be reproduce at https://github.com/fortanix/rust-mbedtls/tree/yx/debug_tls13_multithread_bug but also on latest upstream version https://github.com/Mbed-TLS/mbedtls/tree/17526a0d168276aa3ba5833cbb1f8fcd69d688ba ( at the time of creating this issue )

1. Set config

   scripts/config.py set MBEDTLS_SSL_PROTO_TLS1_3
   scripts/config.py set MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE
   scripts/config.py set MBEDTLS_THREADING_PTHREAD
   scripts/config.py set MBEDTLS_THREADING_C

2. Update ./mbedtls-sys/vendor/programs/ssl/ssl_pthread_server.c with https://gist.github.com/Taowyoo/e2a90ed25bf299500074fdf03e67a050#file-ssl_pthread_server-c

3. Build and run example, example start a server with 5 threads and spawn 5 threads of clients to connect it.

   mkdir -p build
   cmake -S . -B ./build -DENABLE_PROGRAMS=ON -DENABLE_TESTING=OFF && cmake --build ./build
   ./build/programs/ssl/ssl_pthread_server

Error

The error is random, including:

• -0x0001 - ERROR - Generic error
• -0x006e - ERROR - This is a bug in the library
• -0x7280 - SSL - The connection indicated an EOF
• -0x6e00 - SSL - The handshake negotiation failed

Note, the script above can run successfully with TLS 1.2 for serval time, but for TLS 1.3 the script failed in most cases.