Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for BME680 #2295

Merged
merged 4 commits into from
Aug 2, 2020
Merged

Add support for BME680 #2295

merged 4 commits into from
Aug 2, 2020

Conversation

ruimarinho
Copy link
Contributor

@ruimarinho ruimarinho commented Jul 4, 2020

Add support for BME680 using BSEC's proprietary algorithms for precise Indoor Air Quality (IAQ) measurement. Unlike traditional CO2 sensors - and good ones are expensive - it measures nearly all VOCs compounds in the air (plus other gases) and compensates those measurements with its built-in temperature and humidity sensors to determine indoor air quality.

Overall, the CO2 and VOCs are correlated and both are in the human's exhaled breath. However, VOCs allow detecting other reasons for bad air, like smells or human metabolism.

image

Source: Bosch Sensortec BME680 Presentation

The BME680 is an expensive sensor (~€15-20) but it's quite amazing in what it does when compared to dedicated CO2 sensors like the SenseAir S8 (~€50).

The most difficult part until now is that to my knowledge, no open source firmware like Espurna, Tasmota or Esphome has ever shipped support for the only reliable IAQ calculation. While several attempts have been made to reverse engineer the formula behind gas resistance/temperature/humidity, none actually matches the quality of the algorithm built by Bosch's BSEC team.

Since there is now an open repository for Arduino directly from Bosch Sensortec on GitHub which includes the proprietary lib compiled for the BME680, in my understanding, as long as we don't deliberately remove any copyright notice (which 3-Clause BSD requires), then there should be no reason not to include it here. Platform.io manages this dependency, so the copyright notice is always downloaded and carried around alongside the compiled library.

This pull request should work a bit like magic - you don't need to edit any configuration to include additional paths for the compiled binary or even change the memory layout. It is all handed automatically for the end user for the best experience possible. Just enable support for this sensor with BME680_SUPPORT=1.

Fixes #1668.

Future improvements:

  • Save sensor state on EEPROM after IAQ accuracy >= 3.
  • Make sensor config init configurable.

code/espurna/sensor.cpp Outdated Show resolved Hide resolved
code/espurna/sensor.cpp Outdated Show resolved Hide resolved
// The maximum allowed time between two `bsec_sensor_control` calls depends on
// configuration profile `bsec_config_iaq` below. With these settings, the
// maximum allowed time is 3s.
#define SENSOR_READ_INTERVAL 1
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mcspr I don't really like this approach.. do you have better suggestions?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'Maximum allowed time' meaning we must call it at least every 3 seconds?
There is always tick(), which gets called independently of the reading and can be polled via millis() - last >= timeout_milliseconds / micros() - last >= timeout_microseconds

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. In Low Power (LP) mode, which is the default, a reading must happen at least every 3 seconds.

tick() is exactly what I was looking for. I'll make this modification.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://www.bosch-sensortec.com/media/boschsensortec/downloads/datasheets/bst-bme680-ds001.pdf

4.1 BSEC Software
...
Low power (LP) mode that is designed for interactive applications where the indoor-air-quality is tracked and
observed at a higher update rate of 3 seconds with a current consumption of <1 mA

Technically, we just loose last 3 seconds or readings if we don't? It is still useful though, as we don't have any new data with less than 3 seconds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my observations, if we don't call within 3 seconds, the sensor can never calibrate. The default was at 6 seconds and the readings were stalled. I believe it's part of the algorithm.

@ruimarinho ruimarinho force-pushed the enhancement/bme680 branch 2 times, most recently from 3ee3892 to f811b36 Compare July 4, 2020 17:15
@ruimarinho
Copy link
Contributor Author

@mcspr can you help me understand why the web interface is crashing?

@mcspr
Copy link
Collaborator

mcspr commented Jul 4, 2020

@mcspr can you help me understand why the web interface is crashing?

Check out the stack trace? platformio.ini needs espressif8266@2.5.3 as selected platform and you can use exception monitor via serial:
https://docs.platformio.org/en/latest/core/userguide/device/cmd_monitor.html#cmd-device-monitor-filters

Or decode manually from text file:
https://github.com/mcspr/esparduinoexceptiondecoder

@ruimarinho
Copy link
Contributor Author

I did that yes, but unfortunately I can’t figure out the issue. It was in ws.cpp so most likely related to the magnitude topic value. I’ll see if I can paste something here.

@ruimarinho
Copy link
Contributor Author

Can you try to compile locally and see if it loads for you, even if you don’t have the sensor? Maybe it’s a stupid mistake you can quickly spot :)

@mcspr
Copy link
Collaborator

mcspr commented Jul 4, 2020

I have built it locally, but not seeing the issue. And not really something obvious in the code... magnitude list seems in bounds.

@mcspr
Copy link
Collaborator

mcspr commented Jul 4, 2020

Are you building using latest Core version?

@ruimarinho
Copy link
Contributor Author

I’m using the default version as defined on platformio.ini to avoid any customization.

Your suggestions on the ld script are way beyond my comfort zone, but the concept looks amazing.. is there a way I can help given my hobbyist knowledge of embedded systems? :)

@mcspr
Copy link
Collaborator

mcspr commented Jul 4, 2020

Probably OOM, we are still using espasyncwebserver which buffers large chunks of memory on requests. I tried removing isSensorOk() calls and loaded up error-less sensor... but it still works.
I'd suggest to try building with -latest platform.

Also note about the script - everything is included with PlatformIO, as they ship pyelftools and toolchain is already there, too. We can still use extra script.
Basically, the issue is that we want every function in ROM, not RAM (IRAM, but nonetheless), but compiler will just dump things into default places (.text - RAM, .literal - RAM too).

@ruimarinho
Copy link
Contributor Author

Probably OOM, we are still using espasyncwebserver which buffers large chunks of memory on requests. I tried removing isSensorOk() calls and loaded up error-less sensor... but it still works.
I'd suggest to try building with -latest platform.

Will give it a try. Does it have more free memory?

Also note about the script - everything is included with PlatformIO, as they ship pyelftools and toolchain is already there, too. We can still use extra script.

I don't think pyelftools is available for me. I had to add it to requirements.txt locally. At least when using platform.io via home-brew on macOS.

Basically, the issue is that we want every function in ROM, not RAM (IRAM, but nonetheless), but compiler will just dump things into default places (.text - RAM, .literal - RAM too).

Understood. What would they have to change on their side to default to ROM instead?

@mcspr
Copy link
Collaborator

mcspr commented Jul 5, 2020

Will give it a try. Does it have more free memory?

4KB more, at least. You may also want to try to change lwip LOW_MEMORY setting in the .ini when using newer Core version. How much memory does it have though? heap command shows general info. I do remember alexa & mdns sometimes keeping a lot of memory in specific cases, but it is not really easily reproducible. But, can't really tell without debugging traces and conditions when they happen.

@ruimarinho
Copy link
Contributor Author

@mcspr quick update:

  1. I was able to get the WebUI working with platform_latest (espressif8266@2.5.3).

image

  1. In terms of RAM:
[000277] [MAIN] Heap  : 31776 bytes initially |  5672 bytes used (17%) | 26104 bytes free (82%)
[000286] [MAIN] Stack :  4096 bytes initially |  1232 bytes used (30%) |  2864 bytes free (69%)
…
[600005] [MAIN] Heap  : 31776 bytes initially | 21032 bytes used (66%) | 10744 bytes free (33%)
  1. I got a draft change set for the *.o to *.c.o rename but I will need your help figuring out the best way to structure it. I'll update this PR with this change.

@ruimarinho
Copy link
Contributor Author

@mcspr with platform_2_3_0, the WebUI still crashes. I've tried to use https://github.com/mcspr/esparduinoexceptiondecoder but it always returns ERROR: Parser not complete!. Using https://github.com/me21/EspArduinoExceptionDecoder, I am able to get a stack trace:

stack:
0x4023aa30: operator new(unsigned int) at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/cores/esp8266/abi.cpp:30
0x40230407: AsyncWebServerRequest::addInterestingHeader(String const&) at ~/Projects/github/espurna/code/.pio/libdeps/wemos-d1mini/ESP Async WebServer/src/StringArray.h:67
  \-> inlined by: AsyncWebServerRequest::addInterestingHeader(String const&) at ~/Projects/github/espurna/code/.pio/libdeps/wemos-d1mini/ESP Async WebServer/src/WebRequest.cpp:703
0x40232f35: AsyncWebServer::_attachHandler(AsyncWebServerRequest*) at ~/Projects/github/espurna/code/.pio/libdeps/wemos-d1mini/ESP Async WebServer/src/WebServer.cpp:122
0x40232e73: AsyncWebServer::_rewriteRequest(AsyncWebServerRequest*) at ~/Projects/github/espurna/code/.pio/libdeps/wemos-d1mini/ESP Async WebServer/src/WebServer.cpp:107
0x402310e2: AsyncWebServerRequest::_parseLine() at ~/Projects/github/espurna/code/.pio/libdeps/wemos-d1mini/ESP Async WebServer/src/WebRequest.cpp:566
0x4023a27c: String::concat(char const*) at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/cores/esp8266/WString.cpp:279
0x402313ba: AsyncWebServerRequest::_onData(void*, unsigned int) at ~/Projects/github/espurna/code/.pio/libdeps/wemos-d1mini/ESP Async WebServer/src/WebRequest.cpp:122
0x4010772c: vPortFree at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/cores/esp8266/heap.c:18
0x3fff492c: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/cores/esp8266/umm_malloc/umm_malloc.c:624
0x40101f30: wDev_ProcessFiq at ??:?
0x402313e9: std::_Function_handler<void (void*, AsyncClient*, void*, unsigned int), AsyncWebServerRequest::AsyncWebServerRequest(AsyncWebServer*, AsyncClient*)::{lambda(void*, AsyncClient*, void*, unsigned int)#7}>::_M_invoke(std::_Any_data const&, void*, AsyncClient*, void*, unsigned int) at ~/.platformio/packages/toolchain-xtensa@1.40802.0/xtensa-lx106-elf/include/c++/4.8.2/functional:2073
0x40244143: pp_tx_idle_timeout at ??:?
0x3fff4f0c: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/tcp_in.c:73
0x4022a35c: AsyncClient::_recv(tcp_pcb*, pbuf*, signed char) at ~/Projects/github/espurna/code/.pio/libdeps/wemos-d1mini/ESPAsyncTCP@src-26361d8fbca2048a9d8f0cfbb644ba49/src/ESPAsyncTCP.cpp:420
0x3fff5554: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/ipv4/ip.c:107
0x3fff5560: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/ipv4/ip.c:109
0x3fff4f08: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/tcp_in.c:74
0x3fff4f10: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/tcp_in.c:70
0x3fff4f0c: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/tcp_in.c:73
0x4022a3a7: AsyncClient::_s_recv(void*, tcp_pcb*, pbuf*, signed char) at ~/Projects/github/espurna/code/.pio/libdeps/wemos-d1mini/ESPAsyncTCP@src-26361d8fbca2048a9d8f0cfbb644ba49/src/ESPAsyncTCP.cpp:498
0x3fff4f10: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/tcp_in.c:70
0x3fff4f0c: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/tcp_in.c:73
0x40257030: tcp_input at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/tcp_in.c:394 (discriminator 1)
0x40251a31: pbuf_alloc at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/pbuf.c:388
0x40250000: dhcp_select at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/dhcp.c:335 (discriminator 1)
0x3fff5558: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/ipv4/ip.c:100
0x3fff5560: ?? at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/ipv4/ip.c:109
0x40255539: ip_input at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/core/ipv4/ip.c:559
0x40257dea: ieee80211_hostap_attach at ??:?
0x402582de: hostap_input at ??:?
0x40254931: ethernet_input at ~/.platformio/packages/framework-arduinoespressif8266@1.20300.1/tools/sdk/lwip/src/netif/etharp.c:1379
0x40244146: pp_tx_idle_timeout at ??:?
0x402440d6: pp_tx_idle_timeout at ??:?
0x4024c5e3: system_get_sdk_version at ??:?

@ruimarinho
Copy link
Contributor Author

ruimarinho commented Jul 22, 2020

Also see https://community.bosch-sensortec.com/t5/MEMS-sensors-forum/BSEC-library-installation-instructions-for-ESP8266/td-p/14862

I was about to submit the ticket myself :-) the community manager marked as resolved, so we might get some traction there!

Update: sent them an email directly as well just to reinforce the message.

@ruimarinho ruimarinho force-pushed the enhancement/bme680 branch 3 times, most recently from 5438a64 to 07932de Compare July 22, 2020 01:13
@mcspr
Copy link
Collaborator

mcspr commented Jul 27, 2020

If defined(ARDUINO_ESP8266_RELEASE_2_3_0), instead of setting BME680_SUPPORT to 0 we should simply throw an error with some help message. WDYT?

You would fail Travis test then, it also builds bme680_support with both. So it's either disabling it or ignoring, I'd rather disable

@ruimarinho
Copy link
Contributor Author

I think that's a bit of a heavy handed approach -- what about disabling WEB_SUPPORT instead? Seems like a good compromise. Pushed for your review.

@ruimarinho ruimarinho force-pushed the enhancement/bme680 branch 2 times, most recently from a6c5393 to 807a388 Compare July 28, 2020 00:32
@mcspr
Copy link
Collaborator

mcspr commented Jul 28, 2020

I'd still rather not do that. You sure other sensor boards work at all in your configuration? We deal with strange network traffic and bam - no memory. It is not necessarily web's fault, just an OOM sittuation

What I meant by Travis issue: https://travis-ci.org/github/xoseperez/espurna/jobs/712391274#L311-L312
Sensors are built in bulk, not separately, so you introduce not obvious configuration change this way. Splitting into a separate header could work... but now we complicate test builds :/

@ruimarinho
Copy link
Contributor Author

Updated with your recommendation.

@mcspr
Copy link
Collaborator

mcspr commented Jul 29, 2020

Thx!
I noticed another issue though (lol, guess with what)

When archive is packed back, we cannot use *.c.o expression in the shell command when running Windows. cmd.exe does not expand it, unlike other systems. Shortest solution I found is to use env.Append(BSEC_FILES_LIST=new_file_names) in the renamer function and use $BSEC_FILES_LIST as ar argument instead of glob. Meaning, we don't necessarily need shell cd too, because paths become absolute. I guess there is a right scons-way to do it, have not seen yet (env.StaticLibrary()?)

@ruimarinho
Copy link
Contributor Author

I like the renamer approach, but StaticLibrary() definitely seems like the native icons way to do this. Do you have experimental code you can push or do you want me to give it a go?

@mcspr
Copy link
Collaborator

mcspr commented Jul 30, 2020

There is also a subtle issue with caching - .patched gets saved, but no the lib. I remade the script to actually produce an archive file in the build directory, however now the issue is we always rebuild b/c something in action list changes. Not sure what though :/ BSEC_RENAMED_O_FILES?

edit: I have added wip8 as a POC that this is indeed the case
edit2: And wip9 gets rid of that altogether, replacing 'backtick' / shell calls with subprocess directly.

@ruimarinho
Copy link
Contributor Author

@mcspr thanks for all your help on the patcher! It's looking good - is it working well on Windows? We should probably add it to the OS matrix on Travis.

@mcspr
Copy link
Collaborator

mcspr commented Jul 31, 2020

It does, I was switching between WSL and cmd.exe shells trying this out.
Is this done then?

Travis would be tricky here. What they propose by default is to select language: bash and use Git bash.exe. For example, injecting mv:

+            Touch("$BUILD_DIR/1.txt"),
+            "mv $BUILD_DIR/1.txt $BUILD_DIR/2.txt",

Shows --debug=explain:

rebuilding `.pio\build\esp8266-4m-latest-base\libalgobsec_patched\libalgobsec.a' because `C:\Program Files\Git\usr\bin\mv.EXE' is a new dependency

However, with normal PIO Windows installation:

mv .pio\build\esp8266-4m-latest-base/1.txt .pio\build\esp8266-4m-latest-base/2.txt
'mv' is not recognized as an internal or external command,
operable program or batch file.

🤦

bash.exe kind of defeats the purpose of testing this at all

@ruimarinho
Copy link
Contributor Author

Maybe GitHub Actions could be an alternative to try.

I pushed two new commits, one fixing a bug (incorrect order of values) and some minor comment changes.

If it it looks good to you, I can squash into a single commit to prepare for merge. Sounds good?

@ruimarinho
Copy link
Contributor Author

What did you have in mind with ESPURNA_LIBALGOBSEC_PATCHER_TMPDIR?

@mcspr
Copy link
Collaborator

mcspr commented Aug 1, 2020

If it it looks good to you, I can squash into a single commit to prepare for merge. Sounds good?

Ok.

re. environment config, I think I also intended to add ESPURNA_LIBALGOBSEC_PATCHER_LIBPATH and experiment again with out-of-build-dir targets, but did not get around to it. atm it is not really useful

Maybe GitHub Actions could be an alternative to try.

Same with travis, depends on the implementation. Assuming shell: powershell / cmd really calls it as the top process, it will work. With Travis, alternative would be to use powershell -command ... / cmd /C, but it still has some quirks:

max@git-bash-exe-shell MINGW64 ~
$ powershell -command 'cmd.exe /c where mv'
C:\Program Files\Git\usr\bin\mv.exe

max@git-bash-exe-shell MINGW64 ~
$ cmd //C where mv
C:\Program Files\Git\usr\bin\mv.exe

@ruimarinho
Copy link
Contributor Author

Commits squashed.

Are Windows users supposed to have WSL2 or MinGW installed?

@ruimarinho
Copy link
Contributor Author

Btw, there is a good chance GitHub Actions acts differently from Travis. See https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#using-a-specific-shell.

@mcspr
Copy link
Collaborator

mcspr commented Aug 2, 2020

Are Windows users supposed to have WSL2 or MinGW installed?

Nope. I just meant to show that running Windows apps via Git bash.exe shell inherits the PATH, which in turn inherits Git's UNIX tools, which Windows does not have by default. PIO on Windows only uses native tools and system's python version - so, only thing required is to fetch python.org's setup and install it. Then, either run VSCode and install PIO extension (pip install is done automatically into the ~/.platformio/penv virtualenv), or do pip install platformio in the shell

@mcspr mcspr merged commit 6266930 into xoseperez:dev Aug 2, 2020
@ruimarinho ruimarinho deleted the enhancement/bme680 branch August 2, 2020 10:47
@ruimarinho
Copy link
Contributor Author

Ok, fair enough. If pio works out of the box on Windows then that's the sort of support we should be aiming for.

Thanks for all your effort on this PR! Learned a lot 👍

@henfri
Copy link

henfri commented Aug 2, 2020

Hello,
thanks a Lot for this Work!
What I do not understand yet ist why No precompiled binaries are possible. Does BSEC prohibit that?
Otherwise one could Just Show the licence at First webgui Login...

@mcspr
Copy link
Collaborator

mcspr commented Aug 11, 2020

btw re to the memory issue
API_SUPPORT makes a custom function per magnitude by binding magnitude's reference to it, so we end up wasting a lot of memory on useless std::function objects. I remember noticing it in the #2247, but have not changed the original API module to parse ID instead of storing as part of the std::function object.
Same with relays, which is slightly more annoying as functions are larger. Change is pretty small though, so at least that's an easy fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3rd party libraries usage for sensors
3 participants