MQTT Connection with BearSSL and TLS 1.2 results in exception 28

[vishalkothari]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP-12
• Core Version: 2.5.2
• Development Env: Arduino IDE
• Operating System: Windows

Settings in IDE

• Module: Nodemcu 1.0
• Flash Size: 4MB
• lwip Variant: v2 Lower Memory
• CPU Frequency: 80Mhz
• Upload Speed: 115200

Problem Description

I am trying to connect to MQTT over TLS 1.2 using BearSSL. I am getting exception 28 when trying to connect.
I tried using web sockets API instead MQTT and it seems to work after some retries with exception 28.

I looked at #5347, and my rootCert seems to be ok as Websocket connection works.
I also looked at #4134, but latest version of umm_malloc doesnt seem to be compatible with ESP core 2.5.2

MCVE Sketch

#include <ESP8266HTTPClient.h>
#include <time.h>
#include <sys/time.h>                   // struct timeval
#include <coredecls.h>
#include <PubSubClient.h> //from https://github.com/Imroy/pubsubclient

#define TZ              5.5       // (utc+) TZ in hours
#define DST_MN          0      // use 60mn for summer time in some countries
#define TZ_MN           ((TZ)*60)
#define TZ_SEC          ((TZ)*3600)
#define DST_SEC         ((DST_MN)*60)

const char *ssid = "<ssid>";
const char *pass = "<password>";

const char *   host = "<broker>.amazonaws.com";
const uint16_t  port = 8883;

BearSSL::WiFiClientSecure wifiClient;
PubSubClient pubsubclient(wifiClient, host, port); 

// Set time via HTTP GET service, as required for x.509 validation
time_t setClockHttp() {
  HTTPClient http;
  time_t now;
  //.....
  //Get epoch from http service
  //....

  timeval tv = { now, 0 };
  timezone tz = { TZ_MN + DST_MN, 0 };

  settimeofday(&tv, &tz);
  gettimeofday(&cbtime, NULL);
  now = cbtime.tv_sec;
  return now;
}

void fetchCertAuthority() {

static const char ca_cert[] PROGMEM = R"EOF(
-----BEGIN CERTIFICATE-----
MIIE0zCCA7ugAwIBAgIQGNrRniZ96LtKIVjNzGs7SjANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJp
....
4fQRbxC1lfznQgUy286dUV4otp6F01vvpX1FQHKOtw5rDgb7MzVIcbidJ4vEZV8N
hnacRHr2lVz2XTIIM6RUthg/aFzyQkqFOFSDX9HoLPKsEdao7WNq
-----END CERTIFICATE-----
)EOF";

static const char client_cert[] PROGMEM = R"EOF(
-----BEGIN CERTIFICATE-----
MIIDWjCCAkKgAwIBAgIVAOFlUZIDMrUht5zbLrjKog1uWrUKMA0GCSqGSIb3DQEB
CwUAME0xSzBJBgNVBAsMQkFtYXpvbiBXZWIgU2VydmljZXMgTz1BbWF6b24uY29t
...
mJKLnqz8EkxQB6qd2/7XimHrmYoo/DI1KjHKfxEFxUnwkOp1wC6sh78bPXordDTL
gBWeKhwdZHZ3d6TdkY+tpmbSE13+n1+4kUit901F4NIDecdMlIN3zJqQwTPsyw==
-----END CERTIFICATE-----
)EOF";

static const char client_key[] PROGMEM = R"KEY(
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAvdXSakGZNJoGlnY7B/Q7FZNRToZzl5QUAYaOvxZRawnKWfd0
b70e7rSSphbf+GNtvKM7S2iXtMH9qz3Vmv7qVM0CxfwzFi//z3m3ncmrqOSkLSuT
....
ut8HxDMV+AmxMDPTnC+5dEhduIw8r1mMBiYr7TXcu5vnDOgzlCq+S6RyWuOVcF2m
YJ8BWgR5mp4KsuPj/eczZLnKgtDzVLoGjg5VE4dPYKypBYHzeeqX
-----END RSA PRIVATE KEY-----
)KEY";

  BearSSL::X509List cert(ca_cert);
  wifiClient.setTrustAnchors(&cert);  
  BearSSL::X509List client_crt(client_cert);
  BearSSL::PrivateKey key(client_key);
  wifiClient.setClientRSACert(&client_crt, &key);
  setClockHttp();
  Serial.printf("settings heap size2: %u\n", ESP.getFreeHeap());
  if (pubsubclient.connect("client1")) {
      Serial.println("connected");
      pubsubclient.publish("queue1","hello from esp");
  }
  Serial.printf("settings heap size3: %u\n", ESP.getFreeHeap());
  Serial.println(pubsubclient.connected());
}

void setup(){
  Serial.begin(115200);

  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, pass);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected");
  Serial.printf("settings heap size1: %u\n", ESP.getFreeHeap());
  fetchCertAuthority();
}

void loop() {
  if (WiFi.status() != WL_CONNECTED) {
    WiFi.begin(ssid, pass);
    if (WiFi.waitForConnectResult() != WL_CONNECTED)
      return;
    Serial.println("WiFi connected");
  }

  if (WiFi.status() == WL_CONNECTED) {
    if (!pubsubclient.connected()) {
      if (pubsubclient.connect("client1")) {
        pubsubclient.publish("queue1","hello from esp");
      }
    }
    else 
      pubsubclient.loop();
  }
}

Debug Messages

WiFi connected
IP address: 
192.168.0.104
settings heap size1: 44256
settings heap size2: 40064
settings heap size3: 38272
0
Connecting to Vishal kothari...
WiFi connected

Exception (28):
epc1=0x4020ab40 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00fe8524 depc=0x00000000
>>>stack>>>
ctx: cont
sp: 3ffffca0 end: 3fffffc0 offset: 01a0
3ffffe40:  00000000 3ffffea0 40208b6c 3fffefa0  
3ffffe50:  00000000 00000000 00000000 4020914b  
3ffffe60:  00000000 3fff189c 3ffeeb78 402033a1  
3ffffe70:  000022b3 00000d50 3ffeeb78 3ffeeb40  
3ffffe80:  000022b3 3ffeeb78 3fff0fbc 3ffeeb40  
3ffffe90:  000022b3 3ffeeb78 3fff0fbc 4020485d  
3ffffea0:  4020b580 3e367803 4020b580 3e367803  
3ffffeb0:  3fffff00 3ffeeb78 3ffeeb38 40206ed4  
3ffffec0:  3ffe87c1 00000000 3fffff54 40207cbc  
3ffffed0:  3ffe87c1 00000000 3fffff00 40207e70  
3ffffee0:  68736956 3ffeec44 3ffeeb38 3ffeeda8  
3ffffef0:  3ffe8524 3ffeec44 3ffeeb38 4020710d  
3fffff00:  4020b458 00000001 0000eb00 40203e00  
3fffff10:  00000000 00000000 3ffe85e1 00000000  
3fffff20:  3ffe8524 3ffeeb01 30303030 00000031  
3fffff30:  00fe85e3 00000000 00000000 00207c1e  
3fffff40:  00000000 00000000 00000000 00000000  
3fffff50:  00fe8524 00000000 00000000 00207cbc  
3fffff60:  3ffe000f 00000000 00000000 ff20af95  
3fffff70:  3ffe8524 3ffeec44 3ffeeb38 402010bc  
3fffff80:  30303030 00000031 00feec7c 40201486  
3fffff90:  4020b580 6800a8c0 feefeffe feefeffe  
3fffffa0:  3fffdad0 00000000 3ffeed78 40208c1c  
3fffffb0:  feefeffe feefeffe 3ffe8554 401004f5  
<<<stack<<<
 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v8b899c12
~ld

OR

WiFi connected
IP address: 
192.168.0.104
settings heap size1: 44256
settings heap size2: 40064

Exception (28):
epc1=0x4023304a epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000014 depc=0x00000000

>>>stack>>>

ctx: sys
sp: 3fffed50 end: 3fffffb0 offset: 01a0
3fffeef0:  3ffef8e4 40239df7 3ffe9b50 3ffe9b5c  
3fffef00:  3ffe9b5c 00000276 00000000 00000013  
3fffef10:  00000002 0000001a 40243763 3ffecce8  
3fffef20:  3ffe9b50 3fffdcc0 3ffe92e8 3ffe92e8  
3fffef30:  00000080 3ffecce8 3fffdab0 00000000  
3fffef40:  40243023 3fffdab0 00000000 00000001  
3fffef50:  3ffe92e8 40000f49 3fffdab0 40000f49  
3fffef60:  40000e19 40001878 00000002 3fffffc0  
3fffef70:  3fffff10 aa55aa55 000000cb 40104278  
3fffef80:  4010427e 00000002 3fffffc0 7fff7fff  
3fffef90:  4010000d 7fff7fff 7fff7fff 07ff7fff  
3fffefa0:  40100530 3fffef3c 401004dd 3ffffd48  
3fffefb0:  3fffffc0 00000000 00000000 feefeffe  
3fffefc0:  feefeffe feefeffe feefeffe feefeffe  
3fffefd0:  feefeffe feefeffe feefeffe feefeffe  
3fffefe0:  feefeffe feefeffe feefeffe feefeffe  
3fffeff0:  feefeffe feefeffe feefeffe feefeffe  
3ffff000:  feefeffe feefeffe feefeffe feefeffe  
3ffff010:  feefeffe feefeffe feefeffe feefeffe  
....
3fffff90:  4020b580 6800a8c0 feefeffe feefeffe  
3fffffa0:  3fffdad0 00000000 3ffeed78 40208c14  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v8b899c12
~ld

Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
Decoding 84 results
0x4023304a: ieee80211_crypto_decap at ?? line ?
0x40239df7: sta_input at ?? line ?
0x40243763: pp_tx_idle_timeout at ?? line ?
0x40243023: ppPeocessRxPktHdr at ?? line ?
0x40104278: call_user_start_local at ?? line ?
0x4010427e: call_user_start_local at ?? line ?
0x4010000d: call_user_start at ?? line ?
0x40100530: cont_ret at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/cont.S line 142
0x401004dd: cont_continue at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/cont.S line 51
0x4025b790: node_remove_from_list at ?? line ?
0x401030bd: lmacProcessTXStartData at ?? line ?
0x401030ba: lmacProcessTXStartData at ?? line ?
0x4021fa30: sha2big_update at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2big.c line 165
:  (inlined by) br_sha384_update at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2big.c line 207
0x40101fe6: wDev_ProcessFiq at ?? line ?
0x4022029b: sha2small_out at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2small.c line 249
0x40101e3c: wDev_ProcessFiq at ?? line ?
0x40236596: ieee80211_output_pbuf at ?? line ?
0x40244193: pp_attach at ?? line ?
0x402441e2: pp_attach at ?? line ?
0x402442ee: pp_attach at ?? line ?
0x40100d72: pp_post at ?? line ?
0x4024328b: ppTxPkt at ?? line ?
0x40236643: ieee80211_output_pbuf at ?? line ?

OR

Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
Decoding 19 results
0x4020ab40: BearSSL::PrivateKey::isRSA() const at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/BearSSLHelpers.cpp line 728 (discriminator 1)
0x40208b6c: esp_yield at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/core_esp8266_main.cpp line 91
0x4020914b: delay at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/core_esp8266_wiring.cpp line 54
0x402033a1: WiFiClient::connect(IPAddress, unsigned short) at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/include/ClientContext.h line 136
:  (inlined by) WiFiClient::connect(IPAddress, unsigned short) at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/WiFiClient.cpp line 170
0x4020485d: BearSSL::WiFiClientSecure::connect(char const*, unsigned short) at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/WiFiClientSecureBearSSL.cpp line 231
0x4020b580: std::function ::swap(std::function &) at ?? line ?
0x4020b580: std::function ::swap(std::function &) at ?? line ?
0x40206ed4: PubSubClient::connect(MQTT::Connect&) at C:\vishal data\arduino-1.8.3\libraries\pubsubclient-master\src/PubSubClient.cpp line 185
0x40207cbc: String::String(char const*) at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/WString.cpp line 36
0x40207e70: String::String(String const&) at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/WString.cpp line 41
0x4020710d: PubSubClient::connect(String) at C:\vishal data\arduino-1.8.3\libraries\pubsubclient-master\src/PubSubClient.cpp line 168
0x4020b458: std::function ::swap(std::function &) at ?? line ?
0x40203e00: BearSSL::WiFiClientSecure::_installClientX509Validator() at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/WiFiClientSecureBearSSL.cpp line 927
0x402010bc: loop at C:\vishal data\iot\esp code\esp-awsiot-bearssl/esp-awsiot-bearssl.ino line 235

Any inputs on this will be very helpful. Thanks.

[earlephilhower]

The problem looks to be here:

**  BearSSL::X509List cert(ca_cert);
  wifiClient.setTrustAnchors(&cert);  
**  BearSSL::X509List client_crt(client_cert);
**  BearSSL::PrivateKey key(client_key);
  wifiClient.setClientRSACert(&client_crt, &key);

You're passing in stack local variables to the persistent connection. Those variables (cert, client_crt, key) cease to be valid the moment you exit the function, and they're needed later on as part of handshaking so that causes very bad things to happen.

Move them to global space and all should be well.

[vishalkothari]

Thanks for your feedback @earlephilhower.
I made the changes you mentioned and modified sketch looks like below

#include <ESP8266HTTPClient.h>
#include <time.h>
#include <sys/time.h>                   // struct timeval
#include <coredecls.h>
#include <PubSubClient.h> //from https://github.com/Imroy/pubsubclient

#define TZ              5.5       // (utc+) TZ in hours
#define DST_MN          0      // use 60mn for summer time in some countries
#define TZ_MN           ((TZ)*60)
#define TZ_SEC          ((TZ)*3600)
#define DST_SEC         ((DST_MN)*60)

const char *ssid = "<ssid>";
const char *pass = "<password>";

const char *   host = "<broker>.amazonaws.com";
const uint16_t  port = 8883;


static const char ca_cert[] PROGMEM = R"EOF(
-----BEGIN CERTIFICATE-----
MIIE0zCCA7ugAwIBAgIQGNrRniZ96LtKIVjNzGs7SjANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJp
....
4fQRbxC1lfznQgUy286dUV4otp6F01vvpX1FQHKOtw5rDgb7MzVIcbidJ4vEZV8N
hnacRHr2lVz2XTIIM6RUthg/aFzyQkqFOFSDX9HoLPKsEdao7WNq
-----END CERTIFICATE-----
)EOF";

static const char client_cert[] PROGMEM = R"EOF(
-----BEGIN CERTIFICATE-----
MIIDWjCCAkKgAwIBAgIVAOFlUZIDMrUht5zbLrjKog1uWrUKMA0GCSqGSIb3DQEB
CwUAME0xSzBJBgNVBAsMQkFtYXpvbiBXZWIgU2VydmljZXMgTz1BbWF6b24uY29t
...
mJKLnqz8EkxQB6qd2/7XimHrmYoo/DI1KjHKfxEFxUnwkOp1wC6sh78bPXordDTL
gBWeKhwdZHZ3d6TdkY+tpmbSE13+n1+4kUit901F4NIDecdMlIN3zJqQwTPsyw==
-----END CERTIFICATE-----
)EOF";

static const char client_key[] PROGMEM = R"KEY(
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAvdXSakGZNJoGlnY7B/Q7FZNRToZzl5QUAYaOvxZRawnKWfd0
b70e7rSSphbf+GNtvKM7S2iXtMH9qz3Vmv7qVM0CxfwzFi//z3m3ncmrqOSkLSuT
....
ut8HxDMV+AmxMDPTnC+5dEhduIw8r1mMBiYr7TXcu5vnDOgzlCq+S6RyWuOVcF2m
YJ8BWgR5mp4KsuPj/eczZLnKgtDzVLoGjg5VE4dPYKypBYHzeeqX
-----END RSA PRIVATE KEY-----
)KEY";

BearSSL::WiFiClientSecure wifiClient;
PubSubClient pubsubclient(wifiClient, host, port); 
BearSSL::X509List cert(ca_cert);
BearSSL::X509List client_crt(client_cert);
BearSSL::PrivateKey key(client_key);

// Set time via HTTP GET service, as required for x.509 validation
time_t setClockHttp() {
  HTTPClient http;
  time_t now;
  //.....
  //Get epoch from http service
  //....

  timeval tv = { now, 0 };
  timezone tz = { TZ_MN + DST_MN, 0 };

  settimeofday(&tv, &tz);
  gettimeofday(&cbtime, NULL);
  now = cbtime.tv_sec;
  return now;
}

void fetchCertAuthority() {
  setClockHttp();
  wifiClient.setTrustAnchors(&cert);  
  wifiClient.setClientRSACert(&client_crt, &key);
  Serial.printf("settings heap size2: %u\n", ESP.getFreeHeap());
  if (pubsubclient.connect("client1")) {
      Serial.println("connected");
      pubsubclient.publish("queue1","hello from esp");
  }
  Serial.printf("settings heap size3: %u\n", ESP.getFreeHeap());
  Serial.println(pubsubclient.connected());
}

void setup(){
  Serial.begin(115200);

  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, pass);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected");
  Serial.printf("settings heap size1: %u\n", ESP.getFreeHeap());
  fetchCertAuthority();
}

void loop() {
  if (WiFi.status() != WL_CONNECTED) {
    WiFi.begin(ssid, pass);
    if (WiFi.waitForConnectResult() != WL_CONNECTED)
      return;
    Serial.println("WiFi connected");
  }

  if (WiFi.status() == WL_CONNECTED) {
    if (!pubsubclient.connected()) {
        fetchCertAuthority();
    }
    else 
      pubsubclient.loop();
  }
}

After this change, I get below errors

Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
Decoding 80 results
0x40232fe2: ieee80211_crypto_decap at ?? line ?
0x40239d8f: sta_input at ?? line ?
0x402436fb: pp_tx_idle_timeout at ?? line ?
0x40242fbb: ppPeocessRxPktHdr at ?? line ?
0x40104278: call_user_start_local at ?? line ?
0x4010427e: call_user_start_local at ?? line ?
0x4010000d: call_user_start at ?? line ?
0x40100530: cont_ret at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/cont.S line 142
0x401004dd: cont_continue at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/cont.S line 51
0x4025b720: node_remove_from_list at ?? line ?
0x4021f9c8: sha2big_update at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2big.c line 165
:  (inlined by) br_sha384_update at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2big.c line 207
0x40220233: sha2small_out at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2small.c line 249
0x4025b464: node_remove_from_list at ?? line ?
0x4022cfe1: br_multihash_update at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/multihash.c line 141
0x4024412b: pp_attach at ?? line ?
0x4024417a: pp_attach at ?? line ?
0x40244286: pp_attach at ?? line ?
0x40243223: ppTxPkt at ?? line ?
0x402365db: ieee80211_output_pbuf at ?? line ?
0x401047f7: wdt_feed at ?? line ?
0x40100d72: pp_post at ?? line ?
0x40213999: glue2esp_linkoutput at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c line 299
0x40213c17: new_linkoutput at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/glue-lwip/lwip-git.c line 235
0x40214008: ethernet_output at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/netif/ethernet.c line 312
0x40213999: glue2esp_linkoutput at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c line 299
0x4021b53c: etharp_output_to_arp_index at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/etharp.c line 770
0x40213c17: new_linkoutput at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/glue-lwip/lwip-git.c line 235
0x40100d72: pp_post at ?? line ?
0x40104154: lmacTxFrame at ?? line ?
0x4021b790: etharp_output_LWIP2 at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/etharp.c line 885
0x40100000: _stext at ?? line ?
0x40101c62: trc_NeedRTS at ?? line ?
0x4021c1a4: ip4_output_if_opt_src at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/ip4.c line 1007
0x402095a1: malloc at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/heap.cpp line 95
0x4020a770: umm_malloc at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266\umm_malloc/umm_malloc.cpp line 1677
0x4021cc6c: mem_malloc at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/mem.c line 210
0x4021c1ec: ip4_output_if_opt at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/ip4.c line 820
0x40221a49: sendrec_buf at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/ssl/ssl_engine.c line 892
:  (inlined by) br_ssl_engine_sendrec_buf at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/ssl/ssl_engine.c line 1160
:  (inlined by) br_ssl_engine_current_state at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/ssl/ssl_engine.c line 1267
0x4021c212: ip4_output_if at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/ip4.c line 793
....

OR

Exception 0: Illegal instruction
Decoding 78 results
0x40239d8f: sta_input at ?? line ?
0x402436fb: pp_tx_idle_timeout at ?? line ?
0x40242fbb: ppPeocessRxPktHdr at ?? line ?
0x40104278: call_user_start_local at ?? line ?
0x4010427e: call_user_start_local at ?? line ?
0x4010000d: call_user_start at ?? line ?
0x40100530: cont_ret at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/cont.S line 142
0x401004dd: cont_continue at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/cont.S line 51
0x4025b720: node_remove_from_list at ?? line ?
0x401020a2: wDev_ProcessFiq at ?? line ?
0x4021f9c8: sha2big_update at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2big.c line 165
:  (inlined by) br_sha384_update at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2big.c line 207
0x40220233: sha2small_out at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/sha2small.c line 249
0x4025b464: node_remove_from_list at ?? line ?
0x4022cfe1: br_multihash_update at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl/src/hash/multihash.c line 141
0x4024412b: pp_attach at ?? line ?
0x4024417a: pp_attach at ?? line ?
0x40244286: pp_attach at ?? line ?
0x40243223: ppTxPkt at ?? line ?
0x402365db: ieee80211_output_pbuf at ?? line ?
0x401047f7: wdt_feed at ?? line ?
0x40100d72: pp_post at ?? line ?
0x40213999: glue2esp_linkoutput at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c line 299
0x40100d72: pp_post at ?? line ?
....

Please let me know your inputs on this.

[vishalkothari]

@earlephilhower, I also tested this on core version 2.5.0 and I get below error

Exception 9: LoadStoreAlignmentCause: Load or store to an unaligned address
Decoding 92 results
0x40231f4e: ieee80211_crypto_decap at ?? line ?
0x4010471f: chm_get_current_channel at ?? line ?
0x40238ccb: sta_input at ?? line ?
0x40242757: pp_tx_idle_timeout at ?? line ?
0x40242017: ppPeocessRxPktHdr at ?? line ?
0x402098ff: loop_task(ETSEventTag*) at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0\cores\esp8266/core_esp8266_main.cpp line 133
0x40104814: call_user_start_local at ?? line ?
0x4010481a: call_user_start_local at ?? line ?
0x4010000d: call_user_start at ?? line ?
0x40100a5c: cont_ret at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0\cores\esp8266/cont.S line 142
0x40100a09: cont_continue at C:\Users\vikothar\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0\cores\esp8266/cont.S line 51
0x4025aa10: node_remove_from_list at ?? line ?
0x40104656: lmacTxFrame at ?? line ?
0x4010425b: lmacMSDUAged at ?? line ?
0x40103bce: lmacRecycleMPDU at ?? line ?
0x40103659: lmacProcessTXStartData at ?? line ?
0x40103656: lmacProcessTXStartData at ?? line ?
0x40102576: wDev_ProcessFiq at ?? line ?
0x40104420: lmacProcessAckTimeout at ?? line ?
0x401023cc: wDev_ProcessFiq at ?? line ?
0x40104656: lmacTxFrame at ?? line ?
0x4010425b: lmacMSDUAged at ?? line ?
0x40103bce: lmacRecycleMPDU at ?? line ?
0x40104165: lmacMSDUAged at ?? line ?
0x40104420: lmacProcessAckTimeout at ?? line ?
0x4010254b: wDev_ProcessFiq at ?? line ?
0x401023cc: wDev_ProcessFiq at ?

[earlephilhower]

@vishalkothari, None of the traces you attached seem to have crashes in BearSSL. It looks like memory corruption somewhere, as you see lots of ROM code in the backtraces.

I can't use the MCVE as it's obviously missing a valid keypair (you don't want to publish your AWS keys online!). Are there any public MQTT services you could connect to and report the error with (and share the pub/priv keys and cert)?

[vishalkothari]

@earlephilhower, Thanks. I tried to run the code against broker test.mosquitto.org with self signed certificates. Surprisingly, it worked well. I switched AWS rootCA from Verisign certificate which is marked as Legacy to new AWS Root CA 1 and regenerated Client cert and keys. This resulted in successful connection. Please note that verisign root CA still works ok with other MQTT clients like MQTT.fx but not from ESP.

I am still getting disconnected from AWS every 20 seconds or so. But, since the connection is working fine on test.mosquitto.org, I will close this issue.