Crashing on some large API Requests

[jLynx]

Basic Infos

Platform

• Hardware: NodeMcu
• Core Version: 2.6.3
• Development Env: Arduino IDE
• Operating System: Windows

Settings in IDE

• Module: Nodemcu
• Flash Mode: [qio|dio|other] (Unsure)
• Flash Size: 4MB
• lwip Variant: v2 Lower Memory
• Reset Method: [ck|nodemcu] (Unsure)
• Flash Frequency: [40Mhz]
• CPU Frequency: 160MHz
• Upload Using: SERIAL
• Upload Speed: 921600 (serial upload only)

Problem Description

Crashing on some API POST calls.
I can make my first call which logs into the site and returns a JSON object containing a token
Next I need to use that token as a cookie to get the real data I need from the API, the issue is it crashes here and I am not quite sure why.

In this example I have removed getting the token and just hard coded it in as it still crashes when I do that.

MCVE Sketch

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

const char* ssid     = "MyWiFi";
const char* password = "shhhhDontTellAnyoneThisPassword";

void setup() {
  Serial.begin(115200);
  delay(10);

  Serial.println();
  Serial.println("Connecting to WiFi");
  
  WiFi.begin(ssid, password);
  
  while (WiFi.status() != WL_CONNECTED) {
    delay(100);
    Serial.print(".");
  }

  Serial.println();
  Serial.print("WiFi connected with ip ");  
  Serial.println(WiFi.localIP());
  delay(1000);
}

void loop() { 
  getResults();
  delay(1000);
}

void getResults() {
  if (WiFi.status() == WL_CONNECTED)
  {
    HTTPClient http;
    
    http.begin("http://192.168.1.1/appGet.cgi"); 
    http.setUserAgent("asusrouter-Android-DUTUtil-1.0.0.3.58-163"); 
    http.addHeader("Content-Type", "application/x-www-form-urlencoded");
    http.addHeader("Cookie", "asus_token=SqHnRcNSSM8Yr56LRBwiUUDUEmhhQXp");

    Serial.println("debug! 1");
    int httpCode = http.POST("hook=get_clientlist%28appobj%29%3Bwl_sta_list_2g%28appobj%29%3Bwl_sta_list_5g%28appobj%29%3Bwl_sta_list_5g_2%28appobj%29%3Bnvram_get%28custom_clientlist%29");
    Serial.println("debug! 2");
    String payload = http.getString();
    Serial.println("debug! 3");
    Serial.println("Result: " + payload + " - " + httpCode);
    Serial.println("debug! 4");
    
    http.end();
  }
}

Debug Messages

Connecting to WiFi
........................................
WiFi connected with ip 192.168.1.245
debug! 1
debug! 2

Exception (3):
epc1=0x40100718 epc2=0x00000000 epc3=0x00000000 excvaddr=0x4001f6d9 depc=0x00000000

>>>stack>>>

ctx: cont
sp: 3ffffb20 end: 3fffffc0 offset: 01a0
3ffffcc0:  3fff0078 3ffefdf8 3ffeea58 40216b54  
3ffffcd0:  00007fff 005d97f8 4010262e 00000050  
3ffffce0:  0000005c 00000046 00000020 40100908  
3ffffcf0:  00000006 3ffee9d8 00000000 00000030  
3ffffd00:  3ffe948c 3fffc6fc 00000014 40217650  
3ffffd10:  3ffefdf8 3fff0034 3ffefdf4 40211594  
3ffffd20:  3ffeea58 00000000 00000000 ef2571ed  
3ffffd30:  4020408c 00000030 00000010 00000050  
3ffffd40:  351b0000 00000000 00000000 40212045  
3ffffd50:  ef256fd5 00000010 00000860 0000eeb6  
3ffffd60:  3ffeea58 3ffe8b34 00000000 3ffefdf4  
3ffffd70:  00000000 00000000 3ffefdf4 40212109  
3ffffd80:  00000010 00000860 64dd2f1a 3ffefdf4  
3ffffd90:  3fff0034 00000000 3ffefdf4 40212a32  
3ffffda0:  00000000 00000001 00000860 40212b4a  
3ffffdb0:  007a1200 5d2a64fe 4bc6a700 ffffffff  
3ffffdc0:  00007970 4bc6a7f0 94bc6a7e 00000000  
3ffffdd0:  00000000 00000000 4bc6a7f0 00000000  
3ffffde0:  007a1200 5d2a688f 40100200 ffffffff  
3ffffdf0:  3fff00f4 00000001 3ffefdf4 4020ddbd  
3ffffe00:  001bbc0b 3ffeff3c 3ffef9c4 402024e8  
3ffffe10:  3fff00f4 3ffefc84 00000065 4020255e  
3ffffe20:  3fff00f4 3ffefc84 3ffefc84 402040a0  
3ffffe30:  80000000 6e6e6f00 000006f4 402040ef  
3ffffe40:  3ffefca4 3ffefca4 00000020 ffffffff  
3ffffe50:  3ffefca4 ffffffff 3fff00f4 40206d34  
3ffffe60:  3ffe881a 000000a0 3fff00f4 40202fc0  
3ffffe70:  00000000 000005b4 00000000 3ffffef0  
3ffffe80:  00001130 00000000 65640032 40206189  
3ffffe90:  3ffe879d 80000029 3ffffef0 3ffffef0  
3ffffea0:  3ffefca4 3ffefca4 3ffffef0 40203042  
3ffffeb0:  40203b64 3ffee538 3ffe87db 40203e49  
3ffffec0:  3fffdad0 00000008 00000000 3ffee5b4  
3ffffed0:  00000000 3ffefca4 3ffffef0 40203a66  
3ffffee0:  3fffdad0 000000c8 3ffee538 402011fb  
3ffffef0:  3ffefd04 3ffefc84 3ffefc84 3ffef95c  
3fffff00:  000b000f 0000002e 3f010050 40001388  
3fffff10:  3ffef974 000b000f 0068000a 70747468  
3fffff20:  3ffe8700 84000003 3ffefeb4 0000007f  
3fffff30:  007a1200 3ffef98c 0029002f 00fee5b4  
3fffff40:  40105000 001ccb6a 80fee610 00000000  
3fffff50:  00000000 000000c8 ffffffff 00000000  
3fffff60:  3fff000a 3ffee600 000003e8 802051b7  
3fffff70:  00000000 3ffefca4 3ffefb00 009b009f  
3fffff80:  80ffdad0 6b6f6f00 00006569 802010b9  
3fffff90:  402071e0 00000000 3ffee574 40201275  
3fffffa0:  feefeffe 00000000 3ffee574 402052cc  
3fffffb0:  feefeffe feefeffe 3ffe84f0 40100b95  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1392, room 16 
tail 0
chksum 0xd0
csum 0xd0
v3d128e5c
~ld

Decoded:

Exception 3: LoadStoreError: Processor internal physical address or data error during load or store
PC: 0x40100718: umm_malloc_core(size_t) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266\umm_malloc\umm_malloc.cpp line 397
EXCVADDR: 0x4001f6d9

Decoding stack results
0x40216b54: ip4_output_if_opt_src at core/ipv4/ip4.c line 1764
0x40100908: malloc(size_t) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266\umm_malloc\umm_malloc.cpp line 511
0x40217650: mem_malloc at core/mem.c line 210
0x40211594: pbuf_alloc_LWIP2 at core/pbuf.c line 284
0x4020408c: Stream::timedRead() at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266\Stream.cpp line 29
0x40212045: tcp_output_alloc_header_common at core/tcp_out.c line 1825
0x40212109: tcp_output_alloc_header at core/tcp_out.c line 1863
0x40212a32: tcp_send_empty_ack at core/tcp_out.c line 2042
0x40212b4a: tcp_output at core/tcp_out.c line 1319
0x40100200: millis() at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266\core_esp8266_wiring.cpp line 188
0x4020ddbd: tcp_recved at core/tcp.c line 1003
0x402024e8: ClientContext::_consume(unsigned int) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\libraries\ESP8266WiFi\src/include/ClientContext.h line 557
0x4020255e: WiFiClient::read() at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\libraries\ESP8266WiFi\src/include/ClientContext.h line 239
0x402040a0: Stream::timedRead() at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266\Stream.cpp line 34
0x402040ef: Stream::readBytes(char*, unsigned int) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266\Stream.cpp line 215
0x40206d34: Stream::readBytes(unsigned char*, unsigned int) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266/Stream.h line 92
0x40202fc0: HTTPClient::writeToStreamDataBlock(Stream*, int) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\libraries\ESP8266HTTPClient\src\ESP8266HTTPClient.cpp line 1414
0x40206189: uart_write(uart_t*, char const*, size_t) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266\uart.cpp line 509
0x40203042: HTTPClient::writeToStream(Stream*) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\libraries\ESP8266HTTPClient\src\ESP8266HTTPClient.cpp line 947
0x40203b64: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266/HardwareSerial.h line 164
0x40203e49: Print::write(char const*) at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266/Print.h line 60
0x40203a66: HTTPClient::getString() at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\libraries\ESP8266HTTPClient\src\ESP8266HTTPClient.cpp line 1032
0x402011fb: getResults() at A:\Users\jLynx\Documents\Code\Arduino\HTTP_POST_Crash/HTTP_POST_Crash.ino line 47
0x40201275: loop() at A:\Users\jLynx\Documents\Code\Arduino\HTTP_POST_Crash/HTTP_POST_Crash.ino line 31
0x402052cc: loop_wrapper() at C:\Users\jLynx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.6.3\cores\esp8266\core_esp8266_main.cpp line 180

I have also tried doing what was suggested on #6811

But I was not able to get a read out as it would just crash

Note: When I give it an incorrect cookie, it works fine and dispalys and error saying I am not logged in, when I give it a valid cookie it will crash. This leaves me to beleive that the response it too long for it to handle maybe?

The length of the content I should be getting back will vary but is currently: 15,287 (Worked this out by calling the API on my desktop)

[jLynx]

Update:

If I change my POST data to return less informaion, it wont crash. But the issue is I need all the informaion in the request.

I tried updating #define _stackSize (5900/4) to #define _stackSize (15900/4) But sadly this did not fix the issue

[jLynx]

I have done a work around by call calling the API from another server and creating a smaller API from that responce containing only the informaion I need that the ESP then calls

[Misiu]

@jLynx what was the original returned data size? Use Fiddler or Wireshark to catch it.

[jLynx]

@Misiu as mentioned above, it's 15,287

[Misiu]

@jLynx sorry, I missed that

[earlephilhower]

Multiply that by 2x since there will be copying involved if you're using String and it's obvious that you're out of memory (or there is no contiguous block of RAM big enough to fit it). The 8266 has 40-50k free to start with, and when you start using WiFi and other things that reduces fast.

Your change to StackThunk has no effect on HTTP requests. You will need to use a different, non-buffering way to access the data, and it will probably need to be parsed as it comes in, not buffered until completely received.