TLS handshake fails for 4096 bit CA on ESP32s3

[PaulMartinsen]

Version

5.5.4

Description

wolfSSL_connect fails connecting to a server that supplies a certificate signed by a certificate authority that use a 4096 bit RSA key when running on ESP32s3 hardware (no hardware acceleration). The same connection completes successfully when running on:

• Windows 11 x64,
• ESP32 using a certificate signed by a certificate authority that uses a 2048 bit key.

The failure occurs when ParseCertRelative in asn.c calls ConfirmSignature, which returns ASN_SIG_CONFIRM_E along the following path:

1. ConfirmSignature →
2. wc_RsaSSL_VerifyInline →
3. RsaPrivateDecryptEx →
4. wc_RsaFunction_ex →
5. wc_RsaFunctionSync →
6. wc_RsaFunctionSync (line 2797 in case RSA_PUBLIC_DECRYPT, RSA_PUBLIC_ENCRYPT) when it calls mp_exptmod_nct.

wc_RsaFunctionSync returns MP_EXPTMOD_E when mp_exptmod_nct fails, which gets translated into ASN_SIG_CONFIRM_E further up the call-stack.

mp_exptmod_nct returns MP_VAL when it fails the test with m->used = 128 and r->size = 193:

int sp_exptmod_nct(const sp_int* b, const sp_int* e, const sp_int* m, sp_int* r)
{
    // ....
    else if (m->used * 2 >= r->size) {
        err = MP_VAL;
    // ....
   return err;
}

When running on Windows with the same trust chain, m->used = 128 and r->size = 257 (so it doesn't fail).

I attached my user_settings.zip configuration file. Notably, I'm not using hardware acceleration for this test.

Is there some configuration I need to enable to support 4096 bit keys on embedded devices, or is this a bug or a known library limitation?

@gojimmypi

[kareem-wolfssl]

Hi @PaulMartinsen ,

Does defining WOLFSSL_SP_4096 help at all?

Thanks,
Kareem

[dgarske]

Also make sure SP_INT_BITS is 4096.

[PaulMartinsen]

Thanks for the suggestions @kareem-wolfssl & @dgarske . To my user_settings.h I added each in turn:

// Enable RSA/RH 4096-bit support
// See https://github.com/wolfSSL/wolfssl/issues/6108
// See https://www.wolfssl.com/documentation/manuals/wolfssl/chapter02.html#wolfssl_sp_4096
#define WOLFSSL_SP_4096 

// Following https://github.com/wolfSSL/wolfssl/issues/6108
#define SP_INT_BITS (4096)

Unfortunately the call to wolfSSL_connect fails with the same error with just WOLFSSL_SP_4096 defined and with WOLFSSL_SP_4096 defined and SP_INT_BITS set to 4096.

Incidentally, when I have the hardware acceleration on, it fails in tfm.c::fp_expmod_nct instead at the line:

int fp_exptmod_nct(fp_int * G, fp_int * X, fp_int * P, fp_int * Y)
{
   // :
   /* prevent overflows */
   if (P->used > (FP_SIZE/2)) { // P->used = 128; FP_SIZE = 136
     printf("FP_SIZE=%d\n", FP_SIZE);
      return FP_VAL;
   }
  // :
}

[kareem-wolfssl]

Hi @PaulMartinsen ,

Looks like ESP32 hardware acceleration is forcing on fast math. For fast math to support 4096-bit keys, please #define FP_MAX_BITS 8192. If you run into any crashes or errors, make sure you're giving wolfSSL sufficient stack (we usually need 25-30KB) and heap.

[PaulMartinsen]

Connection succeeded to a server with a certificate signed by a 4096 bit CA when:

• I turned hardware acceleration (ESP32s3, PR#5950) off again, and
• defined both WOLFSSL_SP_4096 and SP_INT_BITS

I will find a ESP32 to see what happens with the hw acceleration there.

[PaulMartinsen]

Oh. Thanks @kareem-wolfssl . I'll try that first.

[PaulMartinsen]

After adding #define FP_MAX_BITS 8192 and turning hardware acceleration back on (still on ESP32s3), mp_exptmod_nct succeeds, but RsaPrivateDecryptEx fails further on when it calls wc_RsaUnPad_ex. wc_RsaUnPad_ex return -201 ( RSA_PAD_E = -201, /* RSA Padding Error */)

[PaulMartinsen]

I tested on the ESP32 and was able to connect using both software and hardware acceleration. I didn't need SP_INT_BITS in user_settings.h for it to work, but I did need:

#define WOLFSSL_SP_4096 
#define FP_MAX_BITS (8192)

I couldn't find any documentation for SP_INT_BITS in the manual, so was wondering what it does and if I should be including it in user_settings.h even though it seemed, from the outside, unnecessary?

This also provided the clue to the underlying problem with the ESP32s3. On the ESP32, I received the warning: W (28839) wolfssl_mp: exceeds max bit length(2048) in esp_mp_mul, which ultimately led to discovery of a stupid bug in PR #5950 where that implementation checks hardware capabilities. I'll sort that out shortly.

This bug wasn't detected by tests on that PR. So two extra questions:

• Would it make sense to add a test case for this issue to wolfssl\wolfcrypt\test\test.c?
• The problem was discovered in a SSL handshake, but it seems a simpler test would be more suitable. Could you offer some guidance or point to a similar test example so I can add one as part of that PR? I couldn't immediately see a way forward in the 40,000 line test code file.

[gojimmypi]

@PaulMartinsen That's excellent you were able to resolve this. Good suggestion on the new test case. Perhaps that belongs in a separate issue?

As for the handshake, do you actually mean SSL, or TLS (or TLS1.3) ? We have some client and server examples that could be used as a starting point. There are also some heavily-commented examples over in wolfssl-examples/ESP32, such as this enc28j60 wired ethernet.

[PaulMartinsen]

@gojimmypi , i meant that I found out about the bug by calling wolfSSL_connect to establish a secure TLSv1.3 connection. This TLS business is quite new to me; what's that normally called?

But the problem was caused by trying to use the hardware acceleration for keys larger than it supported. So a test case for this bug could just do that, without needing a client/server rig. Agree about making it a separate issue.

I also found SP_INT_BITS appears to be used by WOLFSSL_SP_MATH_ALL. WOLFSSL_SP_MATH_ALL has the side effect of slowing down the "handshake" too. Useful to know, but I think I won't need it.

Thanks for your help @kareem-wolfssl , @dgarske , @gojimmypi , @BrianAker . Much appreciated.

[gojimmypi]

Hi @PaulMartinsen regarding your question:

connection ... what's that normally called?

I suppose everyone has their own preference, but I would just call it a "TLS1.3 connection", and in a technical written document, might even go as far as explicitly state what cipher suite(s) are used.

Regarding #5950, note that I have another relatively large change in the components/wolfssl CMake file coming soon. Specifically project-specific user_settings.h rather than placed in the wolfSSL directory. I also moved in some of the git hash version detection code that will be used in version.h as noted in #6149. Thanks for the reminder there are the explicit quotes needed for possible embedded spaces in the path as noted in #5154.

One of the tips @dgarske gave me was regarding TLS1.3: if you can use that everywhere and exclusively, there would be benefits to turning off everything else.

If you could please, go ahead and open a new issue regarding keys larger than supported. I see you also opened #6148 today, I'll take a look at that soon.

Thanks for all your help and feedback. Cheers