[BUG] assert failed: tcp_update_rcv_ann_wnd IDF/components/lwip/lwip/src/core/tcp.c:951 (new_rcv_ann_wnd <= 0xffff) ONLY when connected in AP

[jordanfuerst]

Description

I have a web page (19KB) on a esp32dev (esp32-wroom-32E) with some text and two png files (4KB and 7KB) . The page works flawlessly for days when the esp32 is connected as a station and is accessed via wireless router, but if I connect to it with it acting like an AP in either WIFI_MODE_APSTA or WIFI_MODE_AP mode I get the following crash nearly, if not, 100% of the time:

assert failed: tcp_update_rcv_ann_wnd IDF/components/lwip/lwip/src/core/tcp.c:951 (new_rcv_ann_wnd <= 0xffff)
Backtrace: 0x400837b9:0x3ffb2cc0 0x4008d8bd:0x3ffb2ce0 0x40092f15:0x3ffb2d00 0x400fc3ee:0x3ffb2e30 0x400fc49c:0x3ffb2e50 0x401593c5:0x3ffb2e70 0x400f9268:0x3ffb2e90

I've made the following recommended changes:
-D CONFIG_ASYNC_TCP_MAX_ACK_TIME=3000
-D CONFIG_ASYNC_TCP_PRIORITY=10
-D CONFIG_ASYNC_TCP_QUEUE_SIZE=128
-D CONFIG_ASYNC_TCP_RUNNING_CORE=1
-D CONFIG_ASYNC_TCP_STACK_SIZE=4096
-D WS_MAX_QUEUED_MESSAGES=128

Additional notes

I think that the web page text visually loads, its not till the images are downloaded that it crashes (I THINK - not sure really how to verify this 100%). The index.html is loaded from PROGMEM while the images are loaded from littlefs

It makes no difference if in pure AP mode or APSTA and it makes no difference if it is connected as a station or not during the crash in APSTA. Client connected web page access is rock solid even in APSTA mode.

I see that there was a fix in v3.2.4 (fix#14 and #18) the fix must somehow only work in client connections, also this fix is included in 3.2.5 (I manually verified just to be safe)

I'm using latest AsyncTCP-3.2.5 and ESPAsyncWebServer-3.3.1 and ESPAsync_WiFiManager-1.15.1

I can duplicate this at will, and unfortunately the use case of this device is to connect directly to it to control a piece of equipment on my farm where there is no WiFi to rout it through.

If somehow useful: addr2line of backtrace gives:
panic_abort
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:408
esp_system_abort
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_system.c:137
__assert_func
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c:85
lwip_recvfrom_udp_raw
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/sockets.c:1196
get_socket
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/sockets.c:523
wr_rf_freq_mem
/home/cff/gittree/chip7.1_phy/chip_7.1/board_code/app_test/pp/phy/phy_chip_v7_ana.c:1208 (discriminator 3)
printf_decode
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/wpa_supplicant/src/utils/common.c:162

Thanks for any help or hopefully a fix by someone smarter than me...
-Jordan

[mathieucarbou]

Hello, for now with the info, it is not enough to trigger any action item, so I will let this issue opened in case you can find out more about it or other people.

In theory, AP mode should not differ (and the fix was reproduced and tested in AP mode), by me and also other people from other projects.

I would suggest you work on a minimal reproductible use case : a new main file only involving AsyncTCP and ESPAsyncWS, no ESPAsync_WiFiManager (which is bringing its own outdated dependencies by the way).

It really feels that the wrong library or wrong version is linked...

[jordanfuerst]

Okay, I stripped it down to bare bones and removed all the ajax and javascript so the page loads just a static page. The problem still happens, but not as often. I need to refresh the screen a handful of times, but it still does it. Tried on a second board with a different, but older micro, and it does it too. I acknowledge that probably everybody is a better programmer than my hobbyist style, so here is a link to my vscode project if i'm just doing something wrong scold me and hopefully nobody wastes time.

Libraries:
|-- AsyncTCP @ 3.2.5
|-- ElegantOTA @ 3.1.5
|-- ESPAsyncWebServer @ 3.3.1
|-- LittleFS @ 2.0.0
|-- Preferences @ 2.0.0

I went and moved the images into progmem so littlefs is not involved and the problem is still there.

Also i ran the esp32_exception_decoder filter this time:
assert failed: tcp_update_rcv_ann_wnd IDF/components/lwip/lwip/src/core/tcp.c:951 (new_rcv_ann_wnd <= 0xffff)

Backtrace: 0x400837b9:0x3ffb2cc0 0x4008d8bd:0x3ffb2ce0 0x40092f15:0x3ffb2d00 0x40101c0a:0x3ffb2e30 0x40101cb8:0x3ffb2e50 0x4015f3a4:0x3ffb2e70 0x400fea84:0x3ffb2e90
  #0  0x400837b9 in panic_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:408
  #1  0x4008d8bd in esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_system.c:137
  #2  0x40092f15 in __assert_func at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c:85
  #3  0x40101c0a in tcp_update_rcv_ann_wnd at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c:951
      (inlined by) tcp_update_rcv_ann_wnd at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c:931
  #4  0x40101cb8 in tcp_recved at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c:991
  #5  0x4015f3a4 in _tcp_recved_api(tcpip_api_call_data*) at lib/AsyncTCP-3.2.5/src/AsyncTCP.cpp:450
  #6  0x400fea84 in tcpip_thread_handle_msg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:172
      (inlined by) tcpip_thread at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:154

[mathieucarbou]

ok, can you please try something...

Inside lib/AsyncTCP-3.2.5/src/AsyncTCP.cpp, line 448, replace:

if(msg->closed_slot != -1 && !_closed_slots[msg->closed_slot]) {

with:

if(msg->closed_slot == -1 || !_closed_slots[msg->closed_slot]) {

I've looked at your project and I only see the definition of your content which could be progmem to free space, but the way you use ESPAsycnWS you go through the progmem handler anyway.

[jordanfuerst]

It amazes me that you do what you do, how do you even know where to look?

I moved everything to progmem just to eliminate littlefs just-in-case. I can move everything back if it helps. I made the change and was able to still cause the error. Most times the page loads quickly/immediately. Occasionally it loads and then only half an image and then goes through a timeout or something and gives up without loading, and then less occasionally the same error as before and seems to fail immediately without the timeout event.

I also, I think, for the first time got a new error:

Guru Meditation Error: Core  0 panic'ed (StoreProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x40092472  PS      : 0x00060833  A0      : 0x80092946  A1      : 0x3ffb2c60
A2      : 0x3ffc8dc4  A3      : 0x3ffd237c  A4      : 0xa5a5a5a5  A5      : 0x00000014
A6      : 0x00060820  A7      : 0x00000001  A8      : 0x00000013  A9      : 0x0000000e
A10     : 0x3ffd23f0  A11     : 0x3ffc9310  A12     : 0x3ffc8de8  A13     : 0x3ffc8e18
A14     : 0x3ffd2384  A15     : 0x00000004  SAR     : 0x00000014  EXCCAUSE: 0x0000001d
EXCVADDR: 0xa5a5a5b1  LBEG    : 0x40089c98  LEND    : 0x40089cae  LCOUNT  : 0xffffffff


Backtrace: 0x4009246f:0x3ffb2c60 0x40092943:0x3ffb2c80 0x40092a80:0x3ffb2ca0 0x40083c7c:0x3ffb2cc0 0x40083c91:0x3ffb2cf0 0x40083cba:0x3ffb2d10 0x40092e35:0x3ffb2d30 0x400f7874:0x3ffb2d50 0x400f84eb:0x3ffb2d70 0x400fca53:0x3ffb2d90 0x400fcc4a:0x3ffb2dc0 0x400fd5a6:0x3ffb2df0 0x400fd63c:0x3ffb2e10 0x400f97c5:0x3ffb2e50 0x4014efa6:0x3ffb2e70 0x400f7580:0x3ffb2e90
  #0  0x4009246f in insert_free_block at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_tlsf.c:235
      (inlined by) block_insert at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_tlsf.c:263
      (inlined by) block_trim_free at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_tlsf.c:377
      (inlined by) block_prepare_used at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_tlsf.c:454
      (inlined by) tlsf_malloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_tlsf.c:851
  #1  0x40092943 in multi_heap_malloc_impl at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap.c:187
  #2  0x40092a80 in multi_heap_malloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c:234
      (inlined by) multi_heap_malloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c:223
  #3  0x40083c7c in heap_caps_malloc_base at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:175
      (inlined by) heap_caps_malloc_base at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:120
  #4  0x40083c91 in heap_caps_malloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:195
  #5  0x40083cba in heap_caps_malloc_default at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:220
  #6  0x40092e35 in malloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/heap.c:24
  #7  0x400f7874 in mem_malloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/mem.c:237
  #8  0x400f84eb in pbuf_alloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/pbuf.c:288
  #9  0x400fca53 in tcp_output_alloc_header_common at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_out.c:1825
  #10 0x400fcc4a in tcp_output_alloc_header at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_out.c:1860
  #11 0x400fd5a6 in tcp_send_empty_ack at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_out.c:2042 (discriminator 2)
  #12 0x400fd63c in tcp_output at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_out.c:1319
  #13 0x400f97c5 in tcp_recved at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c:999
  #14 0x4014efa6 in _tcp_recved_api(tcpip_api_call_data*) at lib/AsyncTCP-3.2.5/src/AsyncTCP.cpp:450
  #15 0x400f7580 in tcpip_thread_handle_msg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:172
      (inlined by) tcpip_thread at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:154

[mathieucarbou]

It amazes me that you do what you do, how do you even know where to look?

Not really: the issue you got looks exactly like this long-standing issue impacting also other forks and projects:

• assert failed: tcp_update_rcv_ann_wnd (needs help to test fix) #14 (comment)
• crashes especially after firmware update at tcp_update_rcv_ann_wnd tbnobody/OpenDTU#2180

The fix was merged : #18

That is why I really don't understand why you have it, and made you try to rollback the fix, just in case it has a side effect I don't yet know about.

The new issue you have is clearly an indication that you are running out of memory.

If you have some progmem data (const uint8_t HTML[xxx] = .... ), you can serve it with :

AsyncWebServerResponse *response = request->beginResponse(200, "text/html", HTML, sizeof(HTML));

if you have some files in the FileSystem, you instead this to serve all files from a root location for example (or one serveStatic per file)

webServer.serveStatic("/", SPIFFS, "/www/").setDefaultFile("index.html")

I would suggest that you rollback the change - make sure you run AsyncTCP without any modifications, then clean your code, then retry, and if it fails like before, retry the patch I gave you.

You could also serve the progmem data using chunk encoding if you wanted to, like in the project example:

  /*
    Chunked encoding test: sends 16k of characters.
    ❯ curl -N -v -X GET -H "origin: http://192.168.4.1" http://192.168.4.1/chunk
  */
  static const char characters[] = "1234567890";
  static size_t charactersIndex = 0;
  server.on("/chunk", HTTP_HEAD | HTTP_GET, [](AsyncWebServerRequest* request) {
    AsyncWebServerResponse* response = request->beginChunkedResponse("text/html", [](uint8_t* buffer, size_t maxLen, size_t index) -> size_t {
      if (index >= 16384)
        return 0;
      memset(buffer, characters[charactersIndex], maxLen);
      charactersIndex = (charactersIndex + 1) % 10;
      return maxLen;
    });
    request->send(response);
  });

I really don't understand yet why rolling back the patch would solve your issue if that's the case.

[mathieucarbou]

Also here are additional tests you can try:

• test with https://github.com/esphome/AsyncTCP (mathieucarbou/AsyncTCP is derived on it) - so if it works, it means a change in this version of AsyncTCP impacts you

• test with AsyncTCPSock which is an implementation of AsyncTCP on BSD sockets. But this lib is old, it might not compile on Arduino 3 maybe...

[mathieucarbou]

FYI @jordanfuerst : #24

I will merge and release soon.

If you happen to see the issue again with the new upcoming releases, please let me know / reopen this issue.