Heap buffer overflow somewhere in SendTls13CertificateVerify with 8k server private key

[vaintroub]

How to reproduce:

• checkout latest wolfssl

• build example programs server and client with ASAN :

  mkdir build
  cd build
  cmake .. -DCMAKE_C_FLAGS=-fsanitize=asan -DCMAKE_BUILD_TYPE=Debug
  make client server
  

• get server certificate and key from https://github.com/MariaDB/server/blob/10.4/mysql-test/std_data/server8k-cert.pem and
  https://github.com/MariaDB/server/blob/10.4/mysql-test/std_data/server8k-key.pem

• start server with these key/certificate and tls1_3

  examples/server/server -v4 -p 3306 -c server8key-cert.pem -k server8key-key.pem &
  

• try to connect to server using

  examples/client/client -v4 -p 3306
  

result :

depth=0 C==14793==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61700000046c at pc 0x7f38386b977a bp 0x7ffffa4ad510 sp 0x7ffffa4accb8
= FWRITE of size 32 at 0x61700000046c thread T0
#0 0x7f38386b9779 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x79779)
depth=0 #1 0x7f3839c78bd8 in wc_Sha256Final /home/wlad/wolfssl/wolfcrypt/src/sha256.c:1297
C #2 0x7f3839c6fc8a in wc_Sha256Hash /home/wlad/wolfssl/wolfcrypt/src/hash.c:1137
= #3 0x7f3839c6ef5a in wc_Hash /home/wlad/wolfssl/wolfcrypt/src/hash.c:487
F #4 0x7f3839c7b08d in RsaPad_PSS /home/wlad/wolfssl/wolfcrypt/src/rsa.c:1299
I #5 0x7f3839c7b6ee in wc_RsaPad_ex /home/wlad/wolfssl/wolfcrypt/src/rsa.c:1405
, #6 0x7f3839c7d69d in RsaPublicEncryptEx /home/wlad/wolfssl/wolfcrypt/src/rsa.c:2992
ST #7 0x7f3839c7efd9 in wc_RsaPSS_Sign_ex /home/wlad/wolfssl/wolfcrypt/src/rsa.c:3774
= #8 0x7f3839c7ef7e in wc_RsaPSS_Sign /home/wlad/wolfssl/wolfcrypt/src/rsa.c:3750
H #9 0x7f3839d01c3b in RsaSign /home/wlad/wolfssl/src/internal.c:4070
e #10 0x7f3839c64161 in SendTls13CertificateVerify /home/wlad/wolfssl/src/tls13.c:6243
l #11 0x7f3839c6bfb2 in wolfSSL_accept_TLSv13 /home/wlad/wolfssl/src/tls13.c:9396
s #12 0x7f3839c381b2 in wolfSSL_accept /home/wlad/wolfssl/src/ssl.c:13568
i #13 0x7f3839c27ee8 in server_test /home/wlad/wolfssl/examples/server/server.c:2746
n #14 0x7f3839c28b2f in main /home/wlad/wolfssl/examples/server/server.c:3149
k #15 0x7f3837ec1bf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
i #16 0x7f3839c22159 in _start (/home/wlad/wolfssl/examples/server/server+0x22159)
,
LAddress 0x61700000046c is a wild pointer.
= SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x79779)
HShadow bytes around the buggy address:
....

Heap overflow seems to be dependent on the math library in use, i.e in MariaDB we get it only on 32bit ,where we compiler WolfSSL with integer.c Elsewhere we compile with tfm.c and FP_MAX_BITS=16384, so that even 8 keys work fine.

I'd be if you could propose a workaround for the issue. tfm.c does not compile on our CI, on 32bit Linuxen, which are using gcc 4.8.5. I tried yet another alternative sp_int.c and while this does not show heap overflows, it does not load the 8key either - it might still be better to have that than having a crash caused by some client in TLSv1.3 handshake

MariaDB bug : https://jira.mariadb.org/browse/MDEV-26236
MariaDB crashes in CI : http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-quantal-x86/builds/27418/steps/mtr/logs/stdio (look for ssl_8k_test in file)
Note : those tests passed when we did not compile TLS1.3

[kareem-wolfssl]

Hi @vaintroub ,

Thank you for reaching out to us. I will work on reproducing your issue here.

Thanks,
Kareem

[elms]

@vaintroub I was able to reproduce I'm looking for a root cause and solution. Thanks for the report.

[elms]

@vaintroub I'm working on a PR to avoid the overflow and will update when I post it.

The PR for this configuration and use will generate an error. In order to support these keys you will need to add -DFP_MAX_BITS=16384 to the DCMAKE_C_FLAGS.

[vaintroub]

@elms , to avoid misunderstanding, the commit above are MariaDB commits that reference this bug . We do use FP_MAX_BITS=16384 in user_settings.h, and the last commit successfully workarounds the issue for us, almost everywhere (i.e except clang-cl, for which it is not yet possible to build tfm.c at all)

[elms]

@vaintroub Thanks for the clarification.

Is there a ticket for clang-cl? I am not able to test at the moment, but using NO_64BIT may allow that to compile but may also impact performance.

[elms]

@vaintroub #4267 addresses this issue. Thanks for the report!

[vaintroub]

@elms , thanks. There is no ticket for clang-cl, I will file one. I tried a couple of ideas, and nothing worked so far