Mixed Sensors BH1750 + BMP180 : Calibration

[kris969]

Hi,

Thanks for your great job, I'm using Espurna for many years ;)
I'm not sure it's an issue, maybe I was wrong somewhere...

I have generated a LOLIN 8266 node, based on espurna 1.14.2-dev, using relative compile option in platformio.ini for the two sensors as mentioned in the title, it works fine. I receive from it : light, pressure and temperature values as expected.

My surprise is about the Web GUI for this node :
In the sensor configuration tab, I was expected to find three entries (Temperature, pressure and Lux) for calibration, only the one for temperature is available here.
So here is my question:
Is there additional compile option to set, or is it an issue?

Let me know if you need additional information.

Regards
Kris

[mcspr]

Do you mean the missing ..._CORRECTION flag setting or something related to the sensor calibration?
edit: I missed the flag description, value can be negative

[kris969]

Cause pictures are more clear than lot of words, I propose you this:
Here is the status tab, where sensors appears:

here is the sensors configuration tab:

Only information relative to temperature is available.

[mcspr]

Referencing existing code, we have similar absoluteone-point 'correction' for lux (through luxCorrection settings key). It is not in the WebUI, though, as seen above.
Nothing for pressure atm and no other methods like 2-point calibration, but I guess that can be easily implement.

[davebuk]

Which file/files would need to be modified? Is it just a matter of using the code that allows temperature calibration in the webui and adding the necessary lines of code for LUX and PRESSURE?

[mcspr]

I think I misunderstood the original issue. Re-reading, I think the summary is:

Missing luxCorrection input in WebUI, like this one:

espurna/code/html/index.html

Line 1677 in 18dea89

<input name="tmpCorrection" class="pure-u-1 pure-u-lg-1-4" type="number" action="reboot" min="-100" step="0.1" max="100" tabindex="18" />

(also see surrounding form)

espurna/code/espurna/sensor.cpp

Line 1083 in 18dea89

root["luxCorrection"] = getSetting("luxCorrection", SENSOR_LUX_CORRECTION);

(we do send it, so only form input is needed)

espurna/code/espurna/sensor.cpp

Lines 2113 to 2114 in 18dea89

	case MAGNITUDE_LUX:
	magnitude.correction = getSetting({"luxCorrection", magnitude.index_global}, luxCorrection);

(since it is set when user adds this key manually)

Missing pressure correction input and setting (see code above).

This is something new, because we should probably have an ability of setting correction per-magnitude. Some options are:

1. pressCorrection# setting, where index means this is n'th pressure magnitude (we do have these exposed in MQTT and HTTP, for example)
2. Using global magnitude index as displayed in magnitude command output and add snsCorrection#. Discoverability is questionable though.
3. sns#Correction# where we use MAGNITUDE_... integer value (config/types.h) as the first id and the rest is from option 1. Otherwise, we would need to create abbreviations for each type.

2-point calibration / correction (since I assumed we are missing some calibration method)

A slightly different way of correction that we use right now. For example:
https://learn.adafruit.com/calibrating-sensors/two-point-calibration is the general algorithm

[davebuk]

Yes, I presume @kris969 is looking for something similar to what I was looking for in #2165. The pressure value and lux are shown in the webUI but not in the sensors section allowing us to have the ability to add or subtract a correction offset. I guess the code for temperature or humidity like the lines linked below can be copied and modified for pressure and lux? I suggest luxCorrection entries and
pressureCorrection entries.

Missing luxCorrection input in WebUI, like this one:

espurna/code/html/index.html

Line 1677 in 18dea89

<input name="tmpCorrection" class="pure-u-1 pure-u-lg-1-4" type="number" action="reboot" min="-100" step="0.1" max="100" tabindex="18" />

(since it is set when user adds this key manually)

Missing pressure correction input and setting (see code above).

This is something new, because we should probably have an ability of setting correction per-magnitude.

With the above point I believe you are suggesting a calibration routine similar to what espurna uses to calibrate the energy values for Watts/Amps/Volts values for Energy monitor. I don't know if a calibration is necessary. I presume over their standard range, these digital sensors are fairly linear in their outputs?

[mcspr]

I guess the code for temperature or humidity like the lines linked below can be copied and modified for pressure and lux? I suggest luxCorrection entries and
pressureCorrection entries.

Exactly. 2 things about web ui:

• module-something html class is hidden by default and needs somethingVisible to be sent beforehand. Like here:
  

  espurna/code/espurna/sensor.cpp

  Line 943 in 18dea89

  if (magnitude.type == MAGNITUDE_TEMPERATURE) root["temperatureVisible"] = 1;

• key prefix will be checked when uploading config from the web:
  

  espurna/code/espurna/sensor.cpp

  Line 839 in 18dea89

  if (strncmp(key, "tmp", 3) == 0) return true;

  
  sensor code needs to 'know' that it can handle it, if something new like pressure... is added

With the above point I believe you are suggesting a calibration routine similar to what espurna uses to calibrate the energy values for Watts/Amps/Volts values for Energy monitor

Existing tmpCorrection now can be tmpCorrection0, tmpCorrection1, etc. per-magnitude. tmpCorrection is a global default, when indexed key is not available. Right now it is only extended for the existing ...Correction settings. What I was suggesting is go about converting each magnitude ID to do the same, we just need to have nice short name for each measurement / magnitude (the shorter the better).

magnitude.correction is a simple offset for the value, ratios are working on the level before that and are settings for the sensor itself (e.g. in case of HLW8012 values without ratios are nonsense, unlike temperature sensors output)

[kris969]

Hi davebuk,
I have a look on #2165, I can confirm you are totally right in your analysis.
Your suggestions for theses new entries are matching with my expectation.

[davebuk]

If it was as simple as adding a few lines of code to sensor.cpp and index.html I'd probably have a go myself. I don't know anywhere near enough about the code to go any deeper I'm afraid.

[mcspr]

The idea that I got seems to be working, I'll need to work out some small issues / typos / js+html sanity

It also raises another question - while I added corrections code for Temperature, Pressure, Humidity & Lux, does it make sense to introduce it for each magnitude? Including flags like SENSOR_TEMPERATURE_CORRECTION and settings keys (tmpCorrection). Essentially, it only checks those 4 types via tmp, hum, press and lux prefixes and ignores the rest

[davebuk]

I'm not sure what you mean 'per magnitude'. Do you mean to add correction adjustment options for any type of sensor, e.g. CO, CO2, NO2 etc, rather than just the four listed above?

I don't know how "calibrated" other sensors are. I guess temp, humidity and pressure are linear and you would just need to adjust plus or minus from the reading to calibrate from a known value. Would air quality sensors be linear as well?

With Lux, although my sensor gives 0lux in the dark, users may want to "calibrate" the 0 and maximum for a known light strength. This would allow a brightness that would normally give 10 Lux measured to output 0 Lux in espurna and 100 Lux measured to output 255 Lux (or whatever maximum brightness value is). Espurna would scale it's output from 0 to max using the Lux range 10 to 100 in this example.

[mcspr]

The terminology comes from the code, 'magnitude' (measurement?) is something that sensor defines. So, in case of BMP180 we have MAGNITUDE_TEMPERATURE and MAGNITUDE_PRESSURE.

Correction is kind-of abstract. It does not care which unit is used, it just adds a number when we read it from the sensor. Sorry to reference the code again:) I hope it is descriptive enough just by looking at names of things

espurna/code/espurna/sensor.cpp

Line 2388 in e26ba9f

value_show = _magnitudeProcess(magnitude, value_raw);

espurna/code/espurna/sensor.cpp

Line 828 in e26ba9f

value = value + magnitude.correction;

That's current dev. Part in the previous comment is where magnitude.correction = ... actually applies, otherwise it is just 0.

Scaling value is another question though? If there is any use of that, sure, but that probably would not be named Correction.

[davebuk]

That makes sense re magnitudes. I didn't know if you meant the correction was adjustable for different ranges of values. Say a sensor value of 0-100, correction could be +/- 10 but values 101-200 could be +/- 5. I realise now that's not what you meant.

Unless it's a quick copy/paste for the other sensor type magnitude values and doesn't create too much additional work for yourself, I'd say just add the correction entries for Pressure and Lux. If other users find they need correction for other sensors then maybe add them at a later date.

The option to "scale" the sensor data and re-index that sensors raw data over a range sounds like it's a whole separate issue. In my example above, if I did want to use a reading of 10 Lux to actually represent 0 Lux, with your correction option I could just minus 10 from the sensor output.

[mcspr]

See #2265
I do see settings for each magnitude via DummySensor. Errors seem to be working, settings are saved.