Pushing NetworkClientSecure to PSRAM

[nseidle]

Using NetworkClientSecure has obvious benefits, but it's consuming a lot of RAM. Is there any way to push NetworkClientSecure (aka TLS) to PSRAM?

[PaulZC]

I think there's a way to redefine MBEDTLS_PLATFORM_STD_CALLOC and MBEDTLS_PLATFORM_STD_FREE
(or possibly MBEDTLS_PLATFORM_CALLOC_MACRO and MBEDTLS_PLATFORM_FREE_MACRO)
with ps_malloc and free

In:

AppData\Local\Arduino15\packages\esp32\tools\esp32-arduino-libs\idf-release_v5.1-442a798083\esp32/include/mbedtls/port/include/mbedtls/esp_config.h

/** Override calloc(), free() except for case where memory allocation scheme is not set to custom */
#ifndef CONFIG_MBEDTLS_CUSTOM_MEM_ALLOC
#include "esp_mem.h"
#define MBEDTLS_PLATFORM_STD_CALLOC		esp_mbedtls_mem_calloc
#define MBEDTLS_PLATFORM_STD_FREE		esp_mbedtls_mem_free
#endif

[PaulZC]

Bingo bongo!

Looks like
\"-DCONFIG_MBEDTLS_CUSTOM_MEM_ALLOC\" \"-DMBEDTLS_PLATFORM_STD_CALLOC=ps_malloc\" \"-DMBEDTLS_PLATFORM_STD_FREE=free\"
does the trick:

FreeHeap: 84060 / HeapLowestPoint: 82492 / LargestBlock: 4128756 / Used PSRAM: 63140
STATE_ROVER_NO_FIX --> STATE_ROVER_FIX, 2024-06-20 16:50:28.124
Network starting user MQTT Client on Active
Network request to start Active
Network layer starting Active
Network State: NETWORK_STATE_OFF --> NETWORK_STATE_DELAY
L-Band Eb/N0[dB] (>9 is good): 9.62
*WIFI_STATE_OFF
networkTypeUpdate, network->type: Active --> WiFi
networkTypeUpdate, network->requestedNetwork: Active --> WiFi
Network starting WiFi
Starting WiFi
WIFI_STATE_CONNECTING
FreeHeap: 84060 / HeapLowestPoint: 82492 / LargestBlock: 4128756 / Used PSRAM: 63140
Network State: NETWORK_STATE_DELAY --> NETWORK_STATE_CONNECTING
Connecting WiFi... [ 15094][I][WiFiMulti.cpp:86] addAP(): [WIFI][APlistAdd] add SSID: VM4924742
[ 26072][I][WiFiMulti.cpp:129] run(): [WIFI] scan done
[ 26077][I][WiFiMulti.cpp:134] run(): [WIFI] 16 networks found
[ 26083][D][WiFiMulti.cpp:228] run():  --->   0: [11][18:35:D1:9D:ED:09] VM4924742 (-56) (*) (visible)
[ 26092][D][WiFiMulti.cpp:233] run():         1: [6][58:CB:52:D8:5E:1E] VM6184319 (-71) (*) (visible)
[ 26101][D][WiFiMulti.cpp:233] run():         2: [6][40:0D:10:48:45:51] VM6184319 (-73) (*) (visible)
[ 26110][D][WiFiMulti.cpp:233] run():         3: [1][58:CB:52:D8:49:F7] VM6184319 (-78) (*) (visible)
[ 26119][D][WiFiMulti.cpp:233] run():         4: [11][8C:83:94:6B:6D:FB] BT-XPFH75 (-78) (*) (visible)
[ 26129][D][WiFiMulti.cpp:233] run():         5: [11][7A:83:94:6B:6D:FC] EE WiFi (-79) ( ) (visible)
[ 26138][D][WiFiMulti.cpp:233] run():         6: [6][C0:05:C2:46:C3:59] VM1245360 (-85) (*) (visible)
[ 26147][D][WiFiMulti.cpp:233] run():         7: [1][9C:31:C3:09:B0:42] SKYCEIY6 (-88) (*) (visible)
[ 26156][D][WiFiMulti.cpp:233] run():         8: [11][6C:A0:B4:84:68:2A] SKYCEIY6 (-88) (*) (visible)
[ 26165][D][WiFiMulti.cpp:233] run():         9: [11][18:35:D1:71:EE:D1] VM2041210 (-88) (*) (visible)
[ 26174][D][WiFiMulti.cpp:233] run():         10: [1][08:B4:B1:66:EF:FD] VM6184319 (-89) (*) (visible)
[ 26183][D][WiFiMulti.cpp:233] run():         11: [6][04:95:E6:62:FF:D9] NOVA (-90) (*) (visible)
[ 26192][D][WiFiMulti.cpp:233] run():         12: [1][D4:35:1D:17:49:49] vodafone174949 (-91) (*) (visible)
[ 26201][D][WiFiMulti.cpp:233] run():         13: [1][40:0D:10:E2:46:89] VM3885188 (-92) (*) (visible)
[ 26211][D][WiFiMulti.cpp:233] run():         14: [1][08:B4:B1:66:ED:5E] VM6184319 (-92) (*) (visible)
[ 26220][D][WiFiMulti.cpp:233] run():         15: [1][74:A5:28:F1:6E:84] TALKTALKF16E7C (-93) (*) (visible)
[ 26230][I][WiFiMulti.cpp:249] run(): [WIFI] Connecting BSSID: 18:35:D1:9D:ED:09 SSID: VM4924742 Channel: 11 (-56)
[ 26240][W][STA.cpp:533] disconnect(): STA already disconnected.
[ 27569][I][WiFiMulti.cpp:272] run(): [WIFI] Connecting done.
[ 27575][D][WiFiMulti.cpp:273] run(): [WIFI] SSID: VM4924742
[ 27580][D][WiFiMulti.cpp:274] run(): [WIFI] IP: 192.168.0.90
[ 27586][D][WiFiMulti.cpp:275] run(): [WIFI] MAC: 18:35:D1:9D:ED:09
[ 27592][D][WiFiMulti.cpp:276] run(): [WIFI] Channel: 11
[ 27598][D][WiFiMulti.cpp:334] resetFails(): [WIFI] Resetting failure flags

WIFI_STATE_CONNECTED
Network connected to WiFi
Network State: NETWORK_STATE_CONNECTING --> NETWORK_STATE_IN_USE
WiFi 'VM4924742' IP address: 192.168.0.90, RSSI: -54
Rover Accuracy (m): 0.316, SIV: 29 GNSS State: 3D Fix
FreeHeap: 61460 / HeapLowestPoint: 59556 / LargestBlock: 4063220 / Used PSRAM: 79960
Log file size: 47722 - Generation rate: 8.7kB/s
[ 27729][D][NetworkManager.cpp:83] hostByName(): Clearing DNS cache
[ 27782][D][NetworkManager.cpp:123] hostByName(): DNS found IPv4 52.215.95.108
[ 31673][D][ssl_client.cpp:322] ssl_starttls_handshake(): Protocol is TLSv1.2 Ciphersuite is TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
[ 31684][D][ssl_client.cpp:324] ssl_starttls_handshake(): Record expansion is 29
FreeHeap: 15792 / HeapLowestPoint: 1424 / LargestBlock: 4063220 / Used PSRAM: 87040
Rover Accuracy (m): 0.316, SIV: 29 GNSS State: 3D Fix
*WIFI_STATE_CONNECTED
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
ZED UBLOX_CFG_SPARTN_USE_SOURCE changed to 0
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 89 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 277 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 128 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 193 bytes from /pp/Lb/eu topic to GNSS
Pushing 512 bytes from /pp/Lb/eu topic to GNSS
Pushing 338 bytes from /pp/Lb/eu topic to GNSS
Log file size: 64809 - Generation rate: 3.4kB/s
FreeHeap: 15748 / HeapLowestPoint: 1424 / LargestBlock: 4063220 / Used PSRAM: 87040

[nseidle]

Nice! Please re-run from defaults so we can compare numbers (sorry, my original post should have had that).

I see that your changes moved (potentially) 87 - 79 = ~8k of RAM to PSRAM when TLS starts. That's good, but there's still a big hit of 61 - 15 = 46k to the heap when TLS is started. Any other tricks up your sleeve?

[PaulZC]

Nothing in life is ever as simple as it first appears...

mbedtls allows the definition of custom alloc and free macros. In ESP-IDF, these are esp_mbedtls_mem_calloc and esp_mbedtls_mem_free. And you'd think it would be possible to use them in arduino-esp32 too. Except it isn't. arduino-esp32 includes a compiled version of mbedtls which is hardwired to INTERNAL heap. To use EXTERNAL PSRAM, you need to (re)compile libmbedtls (actually libmbedcrypto) and patch the core with the updated library .a files...

After much headbanging, I've managed to do that:

WIFI_STATE_CONNECTED
Network connected to WiFi
Network State: NETWORK_STATE_CONNECTING --> NETWORK_STATE_IN_USE
WiFi 'VM4924742' IP address: 192.168.0.90, RSSI: -62
Rover Accuracy (m): 0.215, SIV: 31 GNSS State: 3D Fix
FreeHeap: 61568 / HeapLowestPoint: 59940 / LargestBlock: 4063220 / Used PSRAM: 79968
Log file size: 75511 - Generation rate: 14.0kB/s
*WIFI_STATE_CONNECTED
[ 36516][D][NetworkManager.cpp:83] hostByName(): Clearing DNS cache
[ 36577][D][NetworkManager.cpp:123] hostByName(): DNS found IPv4 54.220.186.214
[ 36584][V][ssl_client.cpp:68] start_ssl_client(): Free internal heap before TLS 61212
[ 36592][V][ssl_client.cpp:75] start_ssl_client(): Starting socket (domain 2)
[ 36761][V][ssl_client.cpp:166] start_ssl_client(): Seeding the random number generator
[ 36770][V][ssl_client.cpp:174] start_ssl_client(): Setting up the SSL/TLS structure...
[ 36778][V][ssl_client.cpp:194] start_ssl_client(): Loading CA cert
[ 36789][V][ssl_client.cpp:265] start_ssl_client(): Loading CRT cert
[ 36798][V][ssl_client.cpp:274] start_ssl_client(): Loading private key
[ 36810][V][ssl_client.cpp:288] start_ssl_client(): Setting hostname for TLS session...
[ 36820][V][ssl_client.cpp:301] start_ssl_client(): Free internal heap after mbedtls_ssl_setup 61064
[ 36829][V][ssl_client.cpp:311] ssl_starttls_handshake(): Performing the SSL/TLS handshake...
[ 37984][D][ssl_client.cpp:324] ssl_starttls_handshake(): Protocol is TLSv1.2 Ciphersuite is TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
[ 37995][D][ssl_client.cpp:326] ssl_starttls_handshake(): Record expansion is 29
[ 38003][V][ssl_client.cpp:332] ssl_starttls_handshake(): Verifying peer X.509 certificate...
[ 38011][V][ssl_client.cpp:340] ssl_starttls_handshake(): Certificate verified.
[ 38019][V][ssl_client.cpp:356] ssl_starttls_handshake(): Free internal heap after TLS 60828
FreeHeap: 60868 / HeapLowestPoint: 58480 / LargestBlock: 3997684 / Used PSRAM: 131956
Pushing 528 bytes of RXM-PMP data to GNSS
Rover Accuracy (m): 0.163, SIV: 31 GNSS State: 3D Fix
FreeHeap: 60800 / HeapLowestPoint: 58480 / LargestBlock: 3997684 / Used PSRAM: 131956

The full write-up is here.

More commits to follow - including updated workflows.

[PaulZC]

Fixed in #390

[LeeLeahy2]

Hi Paul,

Instead of using MACROs, is it possible create a new module that includes WEAK routines for these operations. This way the code can be pre-compiled into a library and distributed. When the user links against the library everything resolves since the weak routines are the only ones available.

When users need to replace the routines, they just implement the routine that they need and link to the library. Since their code implements one or more of the routines, the code links against the new routines and ignores the weak routines. Any routines not implemented are resolved by the weak routines.

[PaulZC]

Hi Lee (@LeeLeahy2 ),

Great suggestion! I'd forgotten about weak attributes. I tried it, but unfortunately it causes a crash...

In mbedtls:
I reverted esp_mem.c back to the original code.
I added the weak attributes in esp_mem.h

void * esp_mbedtls_mem_calloc(size_t n, size_t size) __attribute__((weak));
void esp_mbedtls_mem_free(void *ptr) __attribute__((weak));

The code compiled OK. But when I try to use the new libmbedcrypto.a, I get a crash - even if I don't override the alloc and free functions:

[ 11894][I][WiFiMulti.cpp:249] run(): [WIFI] Connecting BSSID: 18:35:D1:9D:ED:09 SSID: VM4924742 Channel: 11 (-57)
[ 11904][W][STA.cpp:533] disconnect(): STA already disconnected.
Guru Meditation Error: Core  0 panic'ed (InstrFetchProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x00000000  PS      : 0x00060f30  A0      : 0x80236e87  A1      : 0x3ffd98d0
A2      : 0x00000001  A3      : 0x00000060  A4      : 0x0000000c  A5      : 0x3ffd9920
A6      : 0x3ffcb6f8  A7      : 0x00000000  A8      : 0x80238700  A9      : 0x3ffd9b60
A10     : 0x00000001  A11     : 0x00000060  A12     : 0x3ffca44c  A13     : 0x00000000
A14     : 0x00000030  A15     : 0x00000002  SAR     : 0x00000018  EXCCAUSE: 0x00000014
EXCVADDR: 0x00000000  LBEG    : 0x40093870  LEND    : 0x4009387b  LCOUNT  : 0xffffffff


Backtrace: 0xfffffffd:0x3ffd98d0 0x40236e84:0x3ffd98f0 0x4017c45f:0x3ffd9910 0x4017c575:0x3ffd9940 0x4017afd3:0x3ffd9970 0x401839f1:0x3ffd99e0 0x401849c5:0x3ffd9ad0 0x401856e9:0x3ffd9b60 0x402ecc21:0x3ffd9c30 0x402ee621:0x3ffd9c60 0x402ee7cd:0x3ffd9cb0 0x402f08d5:0x3ffd9cd0 0x402ebbd0:0x3ffd9cf0




ELF file SHA256: 68002e3a7b227ede

Rebooting...

Let me know if I missed something. But unfortunately I think this is a bust...

Paul

[PaulZC]

The updated files are in the Revisit_#367 branch if you want to experiment:

https://github.com/sparkfun/SparkFun_RTK_Everywhere_Firmware/tree/Revisit_%23367

[nseidle]

This is in release v1.3.