MBEDTLS_ERR_RSA_BAD_INPUT_DATA while parsing RSA key for CA signed certificate

[Somritag]

HI,

I am using latest MbedTLS library version 3.6.2 LTS with Libwebsocket version 4.3-stable. Earlier I was using older version of MbedTLS library 3.4.1.

Recently after porting I am facing failure while I try to use SSL with TLS using RSA key , the RSA key size is 1675 bytes initially (info.server_ssl_private_key_mem_len)

In case of EC key even though the keylength mismatch is there the parser api is not checking the condition (end != p+len)
pk_parse_key_sec1_der() in pkparse.c (MBedTLS libversion 3.6.2) so we are not failing further ahead.

while in case of RSA key the api

SSL_CTX_use_PrivateKey_ASN1 -> d2i_PrivateKey() (ssl_pkey.c) points to pkey_pm_load() (ssl_pm.c) and in mbedtls_pk_parse_key()

mbedtls_rsa_parse_key() has been used rather pk_parse_key_pkcs1_der() (reffer old version)

and post decoding the keyLength and len is not matching

so the check -> (end != p+len) is failing.

For time being i have bypassed the check and kept it similar to old approach.

Is there a better way to tackle this issue?
Thanks,
Som