[Fixed] WLED 0.16 crashes when refresing the webpage

[Andro-Marian]

What happened?

When randomly requesting the web API json, or when refresing the webpage 2-3 times the wled crashes.
Sometimes takes a little longer 8-10 times, sometimes 2-3 times,

The esp settings can be the factory ones.

To Reproduce Bug

1. Flash the latest 0.16 beta.
2. Open the webpage.
3. Refresh the webpage continously, around a 1 second delay to refresh.

The 0.16 is crashing really fast. I don't see this behaviour in 0.14.4 that easy to crash, seams more impossible with 0.14.4, sometimes after like 10 or more refreshes the page is not loading but the esp is not crashing, waiting a little and works again.

Expected Behavior

To not crash.

Install Method

Binary from WLED.me or Self Build

What version of WLED?

Nightly Release 20250723

Which microcontroller/board are you seeing the problem on?

ESP32

Relevant log/trace output

CORRUPT HEAP: Bad head at 0x3ffe15fc. Expected 0xabba1234 got 0x3ffe1814
abort() was called at PC 0x4009020d on core 1

ELF file SHA256: 0000000000000000

Backtrace: 0x4008ab48:0x3ffdd600 0x4008aea5:0x3ffdd620 0x4009020d:0x3ffdd640 0x400862fa:0x3ffdd660 0x40086791:0x3ffdd680 0x4000bec7:0x3ffdd6a0 0x401be815:0x3ffdd6c0 0x401365c9:0x3ffdd6e0 0x401358a2:0x3ffdd700 0x40137ed5:0x3ffdd720 0x4013244f:0x3ffdd740 0x4013245d:0x3ffdd760 0x4012a9de:0x3ffdd780 0x4012abba:0x3ffdd7b0 0x4012acb1:0x3ffdd7e0 0x4012ad57:0x3ffdd800 0x4008c92e:0x3ffdd830
  #0  0x4008ab48:0x3ffdd600 in invoke_abort at /home/cschwinne/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:648
  #1  0x4008aea5:0x3ffdd620 in abort at /home/cschwinne/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:648
  #2  0x4009020d:0x3ffdd640 in multi_heap_free at /home/cschwinne/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c:305
  #3  0x400862fa:0x3ffdd660 in heap_caps_free at /home/cschwinne/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:354
  #4  0x40086791:0x3ffdd680 in _free_r at /home/cschwinne/esp32-arduino-lib-builder/esp-idf/components/newlib/syscalls.c:42
  #5  0x4000bec7:0x3ffdd6a0 in ?? ??:0
  #6  0x401be815:0x3ffdd6c0 in operator delete(void*) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/del_op.cc:46
  #7  0x401365c9:0x3ffdd6e0 in AsyncWebServerResponse::~AsyncWebServerResponse() at .pio\libdeps\esp32dev\ESPAsyncWebServerWLED\src/WebResponses.cpp:658
  #8  0x401358a2:0x3ffdd700 in AsyncWebServerRequest::~AsyncWebServerRequest() at .pio\libdeps\esp32dev\ESPAsyncWebServerWLED\src/WebRequest.cpp:1048
  #9  0x40137ed5:0x3ffdd720 in AsyncWebServer::_handleDisconnect(AsyncWebServerRequest*) at .pio\libdeps\esp32dev\ESPAsyncWebServerWLED\src/DynamicBuffer.h:259
  #10 0x4013244f:0x3ffdd740 in AsyncWebServerRequest::_onDisconnect() at .pio\libdeps\esp32dev\ESPAsyncWebServerWLED\src/WebRequest.cpp:1048
  #11 0x4013245d:0x3ffdd760 in std::_Function_handler<void (void*, AsyncClient*), AsyncWebServerRequest::AsyncWebServerRequest(AsyncWebServer*, AsyncClient*)::{lambda(void*, AsyncClient*)#3}>::_M_invoke(std::_Any_data const&, void*&&, AsyncClient*&&) at .pio\libdeps\esp32dev\ESPAsyncWebServerWLED\src/WebRequest.cpp:1048
      (inlined by) _M_invoke at c:\users\desktoptv\.platformio\packages\toolchain-xtensa32\xtensa-esp32-elf\include\c++\5.2.0/functional:1871
  #12 0x4012a9de:0x3ffdd780 in std::function<void (void*, AsyncClient*)>::operator()(void*, AsyncClient*) const at .pio\libdeps\esp32dev\AsyncTCP\src/AsyncTCP.cpp:1123
  #13 0x4012abba:0x3ffdd7b0 in AsyncClient::_error(signed char) at .pio\libdeps\esp32dev\AsyncTCP\src/AsyncTCP.cpp:1123
  #14 0x4012acb1:0x3ffdd7e0 in AsyncClient_detail::handle_async_event(AsyncClient_event_t*) at .pio\libdeps\esp32dev\AsyncTCP\src/AsyncTCP.cpp:1123
  #15 0x4012ad57:0x3ffdd800 in _async_service_task(void*) at .pio\libdeps\esp32dev\AsyncTCP\src/AsyncTCP.cpp:1123
  #16 0x4008c92e:0x3ffdd830 in vPortTaskWrapper at /home/cschwinne/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

Rebooting...

Anything else?

No response

Code of Conduct

Fix:

For now is to use and old version of AsyncTCP with 0.16
Instead of using: willmmiles/AsyncTCP @ 1.3.1
use: https://github.com/pbolduc/AsyncTCP.git @ 1.2.0

• I agree to follow this project's Code of Conduct

[blazoncek]

No matter how hard I hit it, I can't make it kneel.

[Andro-Marian]

https://github.com/user-attachments/assets/804e72eb-e701-4505-99a8-4429d31d06a2
Like this?

[DedeHai]

I can get it to crash if I violently hit the refresh button for about 10 seconds. The crash dump is completely different:

=== Memory Info ===
Free heap getFreeHeap(): 50196 bytes
DRAM 8-bit:    Free:  23776 bytes | Largest block:  11764 bytes
IRAM ram:  Free:  50420 bytes | Largest block:  11764 bytes
IRAM 32bit only ram:  Free:  51532 bytes | Largest block:  11764 bytes
RTC ram:  Free:      0 bytes | Largest block:      0 bytes
INTERNAL ram:  Free:  55848 bytes | Largest block:  11764 bytes
DEFAULT ram:  Free:  28660 bytes | Largest block:  11764 bytes
DRAM 32-bit:  Free:  55848 bytes | Largest block:  11764 bytes
PSRAM:        Free:      0 bytes | Largest block:      0 bytes
CORRUPT HEAP: Bad tail at 0x3ffda678. Expected 0xbaad5678 got 0x3ffda4b8
CORRUPT HEAP: Bad head at 0x3ffd8eac. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad tail at 0x3ffd92e0. Expected 0xbaad5678 got 0x3ffd8ea4
CORRUPT HEAP: Bad head at 0x3ffd8c0c. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad tail at 0x3ffd9988. Expected 0xbaad5678 got 0x3ffd96bc
CORRUPT HEAP: Bad tail at 0x3ffd92c8. Expected 0xbaad5678 got 0x3ffd9018
CORRUPT HEAP: Bad head at 0x3ffd8fdc. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad head at 0x3ffd8f04. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad tail at 0x3ffd9988. Expected 0xbaad5678 got 0x3ffd97a8
CORRUPT HEAP: Bad tail at 0x3ffd90e4. Expected 0xbaad5678 got 0x3ffd8efc
CORRUPT HEAP: Bad tail at 0x3ffdabec. Expected 0xbaad5678 got 0x3ffdab7c
CORRUPT HEAP: Bad tail at 0x3ffd97a8. Expected 0xbaad5678 got 0x3ffd95f0
=== Memory Info ===
Free heap getFreeHeap(): 55792 bytes
DRAM 8-bit:    Free:  28604 bytes | Largest block:  11764 bytes
IRAM ram:  Free:  48396 bytes | Largest block:  11764 bytes
IRAM 32bit only ram:  Free:  48396 bytes | Largest block:  11764 bytes
RTC ram:  Free:      0 bytes | Largest block:      0 bytes
INTERNAL ram:  Free:  55384 bytes | Largest block:  11764 bytes
DEFAULT ram:  Free:  28196 bytes | Largest block:  11764 bytes
DRAM 32-bit:  Free:  51948 bytes | Largest block:  11764 bytes
PSRAM:        Free:      0 bytes | Largest block:      0 bytes
CORRUPT HEAP: Bad tail at 0x3ffda010. Expected 0xbaad5678 got 0x3ffd9c78
CORRUPT HEAP: Bad tail at 0x3ffd9c78. Expected 0xbaad5678 got 0x3ffd96f0
CORRUPT HEAP: Bad tail at 0x3ffd9340. Expected 0xbaad5678 got 0x3ffd9178
CORRUPT HEAP: Bad head at 0x3ffd95f8. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad head at 0x3ffda640. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad head at 0x3ffd9180. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad head at 0x3ffd96e8. Expected 0xabba1234 got 0x401595ac
=> 0x401595ac: lwip_cyclic_timer at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/timeouts.c:243
CORRUPT HEAP: Bad head at 0x3ffd96e8. Expected 0xabba1234 got 0x401595ac
=> 0x401595ac: lwip_cyclic_timer at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/timeouts.c:243
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x401bbbfd  PS      : 0x00060e30  A0      : 0x80156b29  A1      : 0x3ffdf020
=> 0x401bbbfd: pbuf_clen at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/pbuf.c:825
A2      : 0xabba1234  A3      : 0x3ffd93cc  A4      : 0x00000000  A5      : 0x00000005
A6      : 0x00000000  A7      : 0x00000001  A8      : 0x00000002  A9      : 0x0000ff00  
A10     : 0x00ff0000  A11     : 0x00000000  A12     : 0x00000001  A13     : 0x00000001
A14     : 0x00000002  A15     : 0x3ffbf8b8  SAR     : 0x0000001c  EXCCAUSE: 0x0000001c
EXCVADDR: 0xabba1234  LBEG    : 0x400875f1  LEND    : 0x400875f9  LCOUNT  : 0x00000027
=> 0x400875f1: esp_timer_impl_get_counter_reg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer_impl_lac.c:118 (discriminator 2)
=> 0x400875f9: esp_timer_impl_get_counter_reg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer_impl_lac.c:128 (discriminator 1)


Backtrace: 0x401bbbfa:0x3ffdf020 0x40156b26:0x3ffdf040 0x40156fb4:0x3ffdf060 0x4015815e:0x3ffdf090 0x4015c9da:0x3ffdf100 0x40161622:0x3ffdf130 0x40152bcd:0x3ffdf150
=> 0x401bbbfa: pbuf_clen at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/pbuf.c:824
=> 0x40156b26: tcp_free_acked_segments at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:1113
=> 0x40156fb4: tcp_receive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:1289 (discriminator 4)
=> 0x4015815e: tcp_process at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:958
      (inlined by) tcp_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:438
=> 0x4015c9da: ip4_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/ip4.c:800
=> 0x40161622: ethernet_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/netif/ethernet.c:186
=> 0x40152bcd: tcpip_thread_handle_msg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:180
      (inlined by) tcpip_thread at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:154

@Andro-Marian did you erase the flash and start on a fresh install directly to latest nightly?

[Andro-Marian]

0.16 Crash.txt
Still can reproduce after a full clean.

[DedeHai]

need more details on how to reproduce, like exact LED setup

[Andro-Marian]

Nothing much.
Used the: python -m esptool --chip esp32 erase_flash
Then Build and upload the firmware with vscode
Then opening the webpage after connected to WiFi and pressing Ctrl+R to refresh the page until randomly crash

I think you need to refresh more violently.

Noticed this when requesting the "/json/si" then after "/presets.json". I think if they are to many requests fast can crash.

[blazoncek]

Do not forget: ESP32 is a microcontroller with limited resources. If you hit it too hard you may deplete its resources (which may cause undesired effects).

[Andro-Marian]

Yea. But should not crash if can't resolve the request. 0.14.4 is very stable or below 0.13. Not a single one crashed when requesting information and in same time as 0.16 ones.

[Andro-Marian]

I can get it to crash if I violently hit the refresh button for about 10 seconds. The crash dump is completely different:

=== Memory Info ===
Free heap getFreeHeap(): 50196 bytes
DRAM 8-bit:    Free:  23776 bytes | Largest block:  11764 bytes
IRAM ram:  Free:  50420 bytes | Largest block:  11764 bytes
IRAM 32bit only ram:  Free:  51532 bytes | Largest block:  11764 bytes
RTC ram:  Free:      0 bytes | Largest block:      0 bytes
INTERNAL ram:  Free:  55848 bytes | Largest block:  11764 bytes
DEFAULT ram:  Free:  28660 bytes | Largest block:  11764 bytes
DRAM 32-bit:  Free:  55848 bytes | Largest block:  11764 bytes
PSRAM:        Free:      0 bytes | Largest block:      0 bytes
CORRUPT HEAP: Bad tail at 0x3ffda678. Expected 0xbaad5678 got 0x3ffda4b8
CORRUPT HEAP: Bad head at 0x3ffd8eac. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad tail at 0x3ffd92e0. Expected 0xbaad5678 got 0x3ffd8ea4
CORRUPT HEAP: Bad head at 0x3ffd8c0c. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad tail at 0x3ffd9988. Expected 0xbaad5678 got 0x3ffd96bc
CORRUPT HEAP: Bad tail at 0x3ffd92c8. Expected 0xbaad5678 got 0x3ffd9018
CORRUPT HEAP: Bad head at 0x3ffd8fdc. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad head at 0x3ffd8f04. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad tail at 0x3ffd9988. Expected 0xbaad5678 got 0x3ffd97a8
CORRUPT HEAP: Bad tail at 0x3ffd90e4. Expected 0xbaad5678 got 0x3ffd8efc
CORRUPT HEAP: Bad tail at 0x3ffdabec. Expected 0xbaad5678 got 0x3ffdab7c
CORRUPT HEAP: Bad tail at 0x3ffd97a8. Expected 0xbaad5678 got 0x3ffd95f0
=== Memory Info ===
Free heap getFreeHeap(): 55792 bytes
DRAM 8-bit:    Free:  28604 bytes | Largest block:  11764 bytes
IRAM ram:  Free:  48396 bytes | Largest block:  11764 bytes
IRAM 32bit only ram:  Free:  48396 bytes | Largest block:  11764 bytes
RTC ram:  Free:      0 bytes | Largest block:      0 bytes
INTERNAL ram:  Free:  55384 bytes | Largest block:  11764 bytes
DEFAULT ram:  Free:  28196 bytes | Largest block:  11764 bytes
DRAM 32-bit:  Free:  51948 bytes | Largest block:  11764 bytes
PSRAM:        Free:      0 bytes | Largest block:      0 bytes
CORRUPT HEAP: Bad tail at 0x3ffda010. Expected 0xbaad5678 got 0x3ffd9c78
CORRUPT HEAP: Bad tail at 0x3ffd9c78. Expected 0xbaad5678 got 0x3ffd96f0
CORRUPT HEAP: Bad tail at 0x3ffd9340. Expected 0xbaad5678 got 0x3ffd9178
CORRUPT HEAP: Bad head at 0x3ffd95f8. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad head at 0x3ffda640. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad head at 0x3ffd9180. Expected 0xabba1234 got 0x3ffc9c24
CORRUPT HEAP: Bad head at 0x3ffd96e8. Expected 0xabba1234 got 0x401595ac
=> 0x401595ac: lwip_cyclic_timer at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/timeouts.c:243
CORRUPT HEAP: Bad head at 0x3ffd96e8. Expected 0xabba1234 got 0x401595ac
=> 0x401595ac: lwip_cyclic_timer at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/timeouts.c:243
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x401bbbfd  PS      : 0x00060e30  A0      : 0x80156b29  A1      : 0x3ffdf020
=> 0x401bbbfd: pbuf_clen at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/pbuf.c:825
A2      : 0xabba1234  A3      : 0x3ffd93cc  A4      : 0x00000000  A5      : 0x00000005
A6      : 0x00000000  A7      : 0x00000001  A8      : 0x00000002  A9      : 0x0000ff00  
A10     : 0x00ff0000  A11     : 0x00000000  A12     : 0x00000001  A13     : 0x00000001
A14     : 0x00000002  A15     : 0x3ffbf8b8  SAR     : 0x0000001c  EXCCAUSE: 0x0000001c
EXCVADDR: 0xabba1234  LBEG    : 0x400875f1  LEND    : 0x400875f9  LCOUNT  : 0x00000027
=> 0x400875f1: esp_timer_impl_get_counter_reg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer_impl_lac.c:118 (discriminator 2)
=> 0x400875f9: esp_timer_impl_get_counter_reg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer_impl_lac.c:128 (discriminator 1)


Backtrace: 0x401bbbfa:0x3ffdf020 0x40156b26:0x3ffdf040 0x40156fb4:0x3ffdf060 0x4015815e:0x3ffdf090 0x4015c9da:0x3ffdf100 0x40161622:0x3ffdf130 0x40152bcd:0x3ffdf150
=> 0x401bbbfa: pbuf_clen at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/pbuf.c:824
=> 0x40156b26: tcp_free_acked_segments at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:1113
=> 0x40156fb4: tcp_receive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:1289 (discriminator 4)
=> 0x4015815e: tcp_process at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:958
      (inlined by) tcp_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:438
=> 0x4015c9da: ip4_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/ip4.c:800
=> 0x40161622: ethernet_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/netif/ethernet.c:186
=> 0x40152bcd: tcpip_thread_handle_msg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:180
      (inlined by) tcpip_thread at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:154

@Andro-Marian did you erase the flash and start on a fresh install directly to latest nightly?

I don't have that "=== Memory Info ===" in the logs anywhere, what I am missing? Looks really different
Is not ESP32-DevKit-V4 with ESP32-WROOM-32E ?

[DedeHai]

Memory info is from my debug build.
If you overload the ESP with requests, it can crash. Nothing we can do about it, its a core function done by espressif.

[blazoncek]

One more thing. I can see you are using slow (but most compatible) flash mode of DOUT.
Try switching it for faster DIO or even QIO if your ESP supports it. And perhaps also 80MHz clock speed.
This will enable ESP to process web requests (and everything else) faster.

I.e.

CPU:   ESP32-D0WDQ6 rev.1, 2 core(s), 240 MHz.
FLASH: 4 MB, Mode 2 (DIO), speed 40 MHz.

EDIT: I propose to close as unresolved as this is something out of scope of WLED.

[Andro-Marian]

I try the DIO, 0.16 still crash very easily, event after 2-3 refreshes or requesting data from API after boot.
0.14 I can't crash even harder than trying on 0.16

Still looks like something is broken with 0.16. Stick to 0.14 for stability.

ShareX_hDim5OJgBQ.mp4