ESP32 crashes while using Events #932
Comments
|
update: replacing events.send() in that loop() with websockes 'globalClient->text(...)' leads to much less troubles and crashes are happening (randomly) after tens of webpage refreshes with the following dump:
Sometimes, there is also an error even without a page refresh, but after 1000 of sent messages int he form:
|
|
same |
|
same. It would seem a problem on the web part, I read that the problem should be in AsyncEventSource and AsyncTCP. We should wait for a more stable release of these libraries. Waiting for a solution |
|
I have the issue here as well ending in a: CORRUPT HEAP: Bad head at 0x3ffd8b28. Expected 0xabba1234 got 0x00000000 assertion I have a polling setup every 5 seconds. Crash happens often when the polling coincides with another request. |
|
Any leads on this? I'm using AsyncEventSource and am hitting these crashes in _runQueue. There appears to be plenty of memory. I'll continue to investigate if there is some property of particular events being sent, but I think the params are all reasonable and valid. |
|
Incidents in may case seem to increase greatly when more clients connect, i.e. connect to the ESP32 using several tabs in the browser and it crashes quickly. Memory usage increases a bit, but nothing that would seem fatal-- heap free ~183K, quickly reloading the page served by the ESP32 a few times usually results in a crash and restart (heap min free observed to drop to ~113K, page is sending several server-side events, crash is in AsyncEventSource _runQueue as above). |
|
I've managed to make it stable. Using: I've downgraded the ESP32 Arduino from 1.0.5 ( 1.0.6 has the same problems for me ) to 1.0.4 and all the problems went away. |
|
@Gamelauncher great, and thanks for the info! I'm using PlatformIO, but it sounds like a bug in the latest framework, or an incompatibility between the latest framework and one of the libs. I'll see if I can get some combination of these lib and framework versions working for me in the PlatformIO environment. |
Where is the patched AsyncTCP lib? |
|
https://github.com/ul-gh/AsyncTCP/tree/Fix_%23111 Some of the forks contain multiple additional fixes. I believe I used this one. |
|
@Gamelauncher which branch/version for ESPAsyncWebserver did you use (the "patched up version")? |
|
Try this one: https://github.com/ul-gh/ESPAsyncWebServer/tree/dev There are allot of pull requests open some people have actually merged them in the forks. |
|
Thank you. I will give it a try! |
|
These branches helped a lot. I have yet to verify 100% fix, but it's looking good. |
|
What are the odds of getting these changes merged back into the origins? Is me-no-dev still maintaining these libs? They're great, btw, not to complain. It would be good to have these fixes in the official libraries available in the library manager and PIO (or at least the origin git repo URL). |
|
Unfortunately i think me_no_dev has a lot going on with new esps integration and such :( The fix is crashed for me btw. My esp is port forwarded and if more than 3 clients wants to connect from outside the network, the esp crashes. I dont know if it is my fault on the routes or not, yet. |
|
I confirm the library fixes. |
|
For other PlatformIO users, these are the lib_deps I'm using for the patched branches (semicolon lines are commented out, replaced by ul-gh repo libs: lib_deps = ; me-no-dev/ESP Async WebServer@^1.2.3 If this is for a pre-existing project, you'll have to delete the lib_deps folder in the project so PlatformIO will download the patched libs. |
What do you mean by returning to the origin? To the original me_no_dev libraries? How is it possible if no fix was merged? |
Thank you! I will give this a try too. |
|
No, I intended to go back to my old unedited sketch, so I had problems before and now my old project has no more problems. |
Oh i see. I'am testing these two libs now: AsyncTCP by ul-gh I'll let it run for a couple of days and we will see. |
Wrapper class for PZEM004T/PZEM004Tv30 libs, controlled with USE_PZEMv3 definition Note: Async Server has lot's of issues under esp32 me-no-dev/ESPAsyncWebServer#876 me-no-dev/ESPAsyncWebServer#900 espressif/arduino-esp32#1101 me-no-dev/ESPAsyncWebServer#324 me-no-dev/ESPAsyncWebServer#932 Signed-off-by: Emil Muratov <gpm@hotplug.ru>
Still crashing but with other backtrace. :(It is crashing when i want to send a websocket text message like this: if(globalClient != NULL && globalClient->status() == WS_CONNECTED){
if (ws.count() > 0){
ws.textAll(msg); // msg is a String
}
}
} |
|
@zekageri I'm not using websockets directly, just heavily using AsyncEventSource, and I've encountered no such issues, so it might be specifically in the websockets portion of the code. |
|
Yeah, it seems to me that my crashes was because of low heap for me. Iam not sure yet, but i moved some big json Strings from internal to external ram and it seems stable for a day now. With fixed libs ofc |
|
The problem still there. Iam sending a not that big JSON String every two seconds and eventually it crashes on ws send and ws free. :( |
|
Could it be, no client is connected anymore when it crashes? Closing the connection doesn't work always. |
|
I have also crashes of an ESP32 with AsyncEventServer. for(auto i = _messageQueue.begin(); i != _messageQueue.end(); ++i) I also noticed, that sometimes the same event is sent two times, if more than one event.sent command follows the ping command without any delay, eg: In my opinion there must be accesses to the messageQueue from different tasks (of the 2 cores of ESP32), that are not locked. I think the crash happens, when an event is sent and the message is removed from the messageQueue, while the AsyncEventServer is in the for i loop. In this case the end value i != _messageQueue.end() is not updated in the loop and therefore an access to an removed message happens. |
|
[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions. |
|
[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions. |
I've got similar issue, when intensively switching on/off over MQTT (just a gpio switch), issue look very similar... did you find a cure finally? Tried the libs you mentioned - all the same... |
I can observe crashes with my code on ESP32 on a very irregular basis and I was wondering is I am doing something wrong with ESPAsyncWebServer. My code is quite long but I think the conceptual part matters more. I have a webserver on ESP32 what serves a couple of webpages. Those webpages listen to /events on ESP32 and plot some graph real time using Highcharts. The real-time data comes to the webpages via /events, which are sent from the loop() function inside the ESP32 c++ part. Inside the loop() some computations are done and then messages are send:
loop(){
// do some computations for 100-300 msec
events.send(msg, "fttick", millis());
}
Now, everything works nicely when I stay on the page in the browser and watch the data. As soon as I start jumping between other webpages(served by ESP32), in one out of 5-10 jumps I catch a crash (with the error dump below). The same happens when I just start refreshing the same page in the browser, every 1-2 seconds (so after getting 4-5 events from the ESP32). It feels like 'client' pointer is disappearing while the message is supposed to be sent and the page is reloading.... but what is the proper way to do that (to handle closing and opening webpages on ESP32 while /events are sent)? The error is not easily reproducible, and it looks like the problem is on the ESPAsyncWebServer side, with all that async behavior. Would switching to websockets be a better option? (even though I do not need the two way communication).
The stacktrace is most of the time the same:
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled. Core 1 register dump: PC : 0x401b0417 PS : 0x00060030 A0 : 0x800df59f A1 : 0x3ffb1c90 A2 : 0xbaad5678 A3 : 0x3f4059e3 A4 : 0x00000034 A5 : 0x00000001 A6 : 0x3ffd7478 A7 : 0x3ffb1c4c A8 : 0x800f4a28 A9 : 0x3ffb1c70 A10 : 0x00000002 A11 : 0x3f405d03 A12 : 0x80088db0 A13 : 0x3ffd64e0 A14 : 0x00000000 A15 : 0x3ffb0060 SAR : 0x0000000a EXCCAUSE: 0x0000001c EXCVADDR: 0xbaad5678 LBEG : 0x400014fd LEND : 0x4000150d LCOUNT : 0xffffffffBacktrace: 0x401b0417:0x3ffb1c90 0x400df59c:0x3ffb1cb0 0x400dfcea:0x3ffb1cd0 0x400dfd45:0x3ffb1cf0 0x400dfd67:0x3ffb1d10 0x400dfdbd:0x3ffb1d30 0x400dbf49:0x3ffb1d60 0x400f6d85:0x3ffb1fb0 0x40088b9d:0x3ffb1fd0PC: 0x401b0417: AsyncClient::space() at /Users/stealth/.platformio/lib/AsyncTCP_ID1826/src/AsyncTCP.cpp line 749 EXCVADDR: 0xbaad5678Decoding stack results 0x401b0417: AsyncClient::space() at /Users/stealth/.platformio/lib/AsyncTCP_ID1826/src/AsyncTCP.cpp line 749 0x400df59c: AsyncEventSourceMessage::send(AsyncClient*) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 141 0x400dfcea: AsyncEventSourceClient::_runQueue() at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 243 0x400dfd45: AsyncEventSourceClient::_queueMessage(AsyncEventSourceMessage*) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 191 0x400dfd67: AsyncEventSourceClient::write(char const*, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 226 0x400dfdbd: AsyncEventSource::send(char const*, char const*, unsigned int, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 318 0x400dbf49: loop() at src/main.cpp line 3691 0x400f6d85: loopTask(void*) at /Users/stealth/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp line 19 0x40088b9d: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143or it can also start without complaining about 'space()', something like this:
0x400dfcea: AsyncEventSourceClient::_runQueue() at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 243 0x400dfd45: AsyncEventSourceClient::_queueMessage(AsyncEventSourceMessage*) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 191 0x400dfd67: AsyncEventSourceClient::write(char const*, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 226 0x400dfdbd: AsyncEventSource::send(char const*, char const*, unsigned int, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 318 0x400dbf49: loop() at src/main.cpp line 3691 0x400f6d85: loopTask(void*) at /Users/stealth/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp line 19 0x40088b9d: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143The text was updated successfully, but these errors were encountered: