Using with ESPHome?

[stas-sl]

Hi, thanks for maintaining this library!

I'm wondering what the best way is to integrate it with ESPHome. I noticed some discussions about it, but I couldn't find a clear guide on what’s needed to get it working. Can it serve as a drop-in replacement for ESPAsyncWebServer-esphome?

I spent a few hours wrestling with PIO dependency management, and I think I've finally got it working. I'll share my solution here, but if there's a better approach, I’d love to know!

So I ended up with:

esphome:
  platformio_options:

    # that's important, as by default it is set to 'off', 
    # which can lead to missing dependencies.
    lib_ldf_mode: chain 

    # that's also important, as without it both libraries will be included, 
    # which will result in duplicate symbols
    lib_ignore:
      - ESPAsyncWebServer-esphome
  libraries:
    - mathieucarbou/ESPAsyncWebServer @ 3.3.7

Maybe there is a way to avoid setting lib_ldf_mode: chain? I'm unsure what potential downsides might be.

Solution that didn't work for me, though it is mentioned in ESPHome documentation as a way to override some libraries:

esphome:
  libraries:
    - ESPAsyncWebServer-esphome=mathieucarbou/ESPAsyncWebServer @ 3.3.7

I haven't thoroughly tested it yet, but it compiles and works so far. Hopefully, it will be more stable than ESPHome's fork.

[mathieucarbou]

Good question!

And I do not know the answer. Since ESPAsyncWebServer is used internally to make esphome work, I am not sure that you can replace it with the lib option (which is meant to add additional libraries).

But It should be compatible since the API is the same and based on the same original fork. If you happen to see some differences,, please let me know.

Also, pay attention that ESPAsyncWebServer is not the only dependency: ESPAsyncWebServer also have some dependencies (e.g. AsyncTCP for ESP32), and the AsyncTCP lib ESPHome is using also misses some bug-fixes.

[stas-sl]

Mathieu, thanks for your response.

I've found some discussions why they decided to turn off PIO's LDF and specify all dependencies manually instead. Setting lib_ldf_mode: chain worked for me, but maybe it could cause issues with other components, as mentioned in these discussions:
https://discord.com/channels/429907082951524364/1089875019326300250
esphome/esphome#2435
esphome/esphome#2608

It seems it's better to keep it turned off. Fortunately, making it work is simple once you understand that the order of specifying dependencies is crucial! If you disable LDF, you need to declare dependencies before the libraries that rely on them.

So, now my working config looks like this:

esphome:
  platformio_options:
    lib_ignore:
      - AsyncTCP-esphome
      - ESPAsyncWebServer-esphome
  libraries:
    - FS
    - WiFi
    - Update
    - mathieucarbou/ESPAsyncWebServer @ 3.3.7

Strangely, even after adding those libraries to lib_ignore, they're still being downloaded:

Library Manager: Installing esphome/AsyncTCP-esphome @ 2.1.4
INFO Installing esphome/AsyncTCP-esphome @ 2.1.4
Unpacking  [####################################]  100%
Library Manager: AsyncTCP-esphome@2.1.4 has been installed!
INFO AsyncTCP-esphome@2.1.4 has been installed!
Library Manager: Installing mathieucarbou/AsyncTCP @ 3.2.5
INFO Installing mathieucarbou/AsyncTCP @ 3.2.5
Unpacking  [####################################]  100%
Library Manager: AsyncTCP@3.2.5 has been installed!
INFO AsyncTCP@3.2.5 has been installed!
Library Manager: Installing mathieucarbou/ESPAsyncWebServer @ 3.3.7
INFO Installing mathieucarbou/ESPAsyncWebServer @ 3.3.7
Unpacking  [####################################]  100%
Library Manager: ESPAsyncWebServer@3.3.7 has been installed!
INFO ESPAsyncWebServer@3.3.7 has been installed!
Library Manager: Resolving dependencies...
INFO Resolving dependencies...
Library Manager: Installing esphome/ESPAsyncWebServer-esphome @ 3.2.2
INFO Installing esphome/ESPAsyncWebServer-esphome @ 3.2.2
Unpacking  [####################################]  100%
Library Manager: ESPAsyncWebServer-esphome@3.2.2 has been installed!
INFO ESPAsyncWebServer-esphome@3.2.2 has been installed!
Library Manager: Resolving dependencies...
INFO Resolving dependencies...
Library Manager: Installing bblanchon/ArduinoJson @ 6.18.5
INFO Installing bblanchon/ArduinoJson @ 6.18.5
Unpacking  [####################################]  100%
Library Manager: ArduinoJson@6.18.5 has been installed!
INFO ArduinoJson@6.18.5 has been installed!

However, they are not mentioned later in Dependency Graph:

Dependency Graph
|-- FS @ 2.0.0
|-- WiFi @ 2.0.0
|-- Update @ 2.0.0
|-- AsyncTCP @ 3.2.5
|-- ESPAsyncWebServer @ 3.3.7
|-- ESPmDNS @ 2.0.0
|-- Wire @ 2.0.0
|-- ArduinoJson @ 6.18.5
|-- WiFiClientSecure @ 2.0.0
|-- HTTPClient @ 2.0.0

In contrast to when they're not added to lib_ignore, both libraries are still mentioned, and I encounter multiple compilation errors.

Dependency Graph
|-- AsyncTCP-esphome @ 2.1.4
|-- FS @ 2.0.0
|-- WiFi @ 2.0.0
|-- Update @ 2.0.0
|-- AsyncTCP @ 3.2.5
|-- ESPAsyncWebServer @ 3.3.7
|-- ESPAsyncWebServer-esphome @ 3.2.2
|-- ESPmDNS @ 2.0.0
|-- Wire @ 2.0.0
|-- ArduinoJson @ 6.18.5
|-- WiFiClientSecure @ 2.0.0
|-- HTTPClient @ 2.0.0

As I mentioned, this version compiles and performs significantly better than the original ESPAsyncWebServer-esphome. I’ve tested it on two heavily loaded devices with similar configurations. Both of them run CPU-intensive audio processing that keeps a single core busy about 90% of the time. Though one of them besides it also has Bluetooth stack enabled, which leaves only 70-80 KB of free memory.

Previously, it was nearly impossible to open the ESPHome's Web server UI on either device because the server would initially send a lot of data, including all sensor information, and continue updating every second. Even if you managed to open it, refreshing the page a few times would inevitably crash the server.

Now, after switching to your fork, I haven’t experienced a single crash on the device without Bluetooth, which has more free RAM. However, I still get some crashes on the device with Bluetooth and less available memory, but it’s much more stable overall. Maybe I should tune some parameters you mentioned in the README, like CONFIG_ASYNC_TCP_STACK_SIZE or something similar. I wonder if 70 KB of free RAM is too low...

However, it's probably still too early to draw conclusions. I'll continue monitoring the stability and see how it performs over time.

[mathieucarbou]

Maybe because some components are adding them ?

• https://github.com/esphome/esphome/blob/dev/esphome/components/async_tcp/__init__.py
• https://github.com/esphome/esphome/blob/361b6ab961e12634fc040ec11e45ab5ae39359f6/esphome/components/web_server_base/__init__.py#L37

Thanks for your perf tests! These are really encouraging. A lot of things have been improved yes, and I am pretty sure that depending on how ESPHome is using the library, further improvements could be made.

Tweaking the stack size will only work if esphome correctly move tasks to execute outside of the async_tcp callback.

[mathieucarbou]

Here is the add_library code: https://github.com/esphome/esphome/blob/361b6ab961e12634fc040ec11e45ab5ae39359f6/esphome/core/__init__.py#L715

They are added because ESPHome renamed them, so they are considered as different.

A hack / workaround otherwise could be to create 2 empty modules called AsyncTCP-esphome and ESPAsyncWebServer-esphome, with only one header file for each AsyncTCP-esphome.h and ESPAsyncWebServer-esphome.h and of course a dummy library.json, and include AsyncTCP.h and ESPAsyncWebServer.h in those.

And then add them.

My hypothesis is that libraries parsed from the esphome yaml file are added AFTER the ones from the internal components, so the newly added AsyncTCP-esphome and ESPAsyncWebServer-esphome would then override the existing ones.... Hopefully! 😀

[stas-sl]

Maybe because some components are adding them?

Yes, they are adding them, that's why I have to "remove" them by including in lib_ignore and it seems to have effect, based on Dependency Graph and ability to compile.

Here is part of platformio.ini generated by ESPHome:

framework = arduino
lib_deps =
    esphome/AsyncTCP-esphome@2.1.4
    FS
    WiFi
    Update
    mathieucarbou/AsyncTCP @ 3.2.5
    mathieucarbou/ESPAsyncWebServer @ 3.3.7
    esphome/ESPAsyncWebServer-esphome@3.2.2
    ESPmDNS
    Wire
    bblanchon/ArduinoJson@6.18.5
    WiFiClientSecure
    HTTPClient
    ${common.lib_deps}
lib_ignore =
    AsyncTCP-esphome
    ESPAsyncWebServer-esphome
lib_ldf_mode = off

Here you can see both libraries: those added by components and the ones I've included in my config, along with my lib_ignore section. It seems that you can't remove libraries that have already been included by other components. The only available method is to exclude them using lib_ignore. However, it appears that even when "ignored", they are still downloaded, as they still listed under lib_deps, though they are not used.

[mathieucarbou]

and it seems to have effect, based on Dependency Graph and ability to compile.

Not sure: esphome/ESPAsyncWebServer-esphome@3.2.2 is still there esphome/AsyncTCP-esphome@2.1.4 also...

Note: I know the lib_ignore flag for having played a lot with it and I've only managed to make pio ignore local libraries installed in the project lib folder which are "final" dependencies not required by any other.

I've never managed to make lib_ignore work to exclude a dependency that is declared, or a dependency of a declared dependency.

It would be nice to have a recipe to make that work! Ability to use this ESPAsyncWS fork in ESPHome would be great indeed.

[stas-sl]

Not sure: esphome/ESPAsyncWebServer-esphome@3.2.2 is still there esphome/AsyncTCP-esphome@2.1.4 also...

Hmm, did you notice the difference? I posted above 2 versions of PIO's Dependency Graphs: first with lib_ignore section added (there are no references to ESPAsyncWebServer-esphome or AsyncTCP-esphome) and the second one without lib_ignore for comparison, which has references to both libraries. Maybe you looked at the latter?

I can guess how PIO library management works, though I’m not entirely sure. It likely operates in two stages: first, a straightforward one where it downloads everything listed under lib_deps, and second, a more intelligent stage where it applies the LDF and takes into account the lib_ignore section.

There is some relevant discussion here platformio/platformio-core#4823, though it not the same issue, but it explains that not all downloaded libraries are built:

We resolve dependencies BEFORE LDF starts (because we need all manifests). Please note that libraries installed/copied to the libdeps folder are not always built (LDF decides).

[mathieucarbou]

Ok !!! I didn't know that! Thanks !!