Flow Reading Accuracy Troubleshooting

[aneisch]

First of all, thanks for all your work on this!

I got things set up with an ESP8266 (D1 Mini) and things initially looked perfect. However, during periods of higher flow (eg: irrigation) I found that my readings were considerably lower than expected and compared to my flume 2.

After some reading in issues in this repository I decided to rebuild my meter with an ESP32 (Lolin S2-mini). It seems to return slightly more accurate values at higher flow, but interestingly seems to also have a longer loop time (and reports "qmc5883l.sensor took a long time for an operation" ocasionally).

Has anyone experienced this, or do you have any suggestions for improving accuracy? I am using the same set up for each device with the screenshots below. I built a D1 mini shield with my QMC5883L that I simply moved between the two microcontrollers (leaving the sensor in the same position on the meter). For this test device I'm using ~1ft Cat5.

D1 Mini Flow

D1 Mini Loop Time

D1 Mini Config:

Compared to:

S2 Mini Flow

S2 Mini Loop Time

S2 Mini Configuration

[tronikos]

My recommendation is to try again without logging, with the magnetic field sensors disabled, and maybe without the loop time sensor. If you have these enabled, at high flow the MCU has to push tons of data to Home Assistant which slow it down.

[aneisch]

I should have specified: I had the field sensors disabled in HA. Do I need to disable them in the yaml config as well or does disabling the entity in HA accomplish what we need?

I ordered a different esp32, hoping that I see an improvement. I will report back.

[tronikos]

I was referring to disabling them in HA. But since you mentioned, can you try converting them to internal in the config? Add this at the end of your config:

sensor:
  - id: !extend ${prefix_id}qmc5883l_id
    field_strength_x:
      internal: true
    field_strength_y:
      internal: true
    field_strength_z:
      internal: true

[aneisch]

After switching from Lolin S2 Mini (single core) to Lolin S3 Mini (dual core) I am seeing sub 5ms loop times and improved accuracy, matching my Flume 2 during up to 25gal/m flows.

I also noted that regardless of whether the debug entities were enabled, I still saw a consistently higher loop time than if I have internal: true set for the axis sensors (typically 2x). I would suggest that others set this as well.

Full config:

substitutions:
  name: water_meter_s3
  friendly_name: "Water Meter S3"
  volume_unit: 'gal'
  i2c_scl: GPIO36  # D1
  i2c_sda: GPIO35  # D2
  prefix_name: s3
  flow_update_interval_seconds: "5"

esphome:
  name: ${name}
  platformio_options:
    board_build.arduino.memory_type: qio_qspi
    board_upload.ram_size: 327680
    board_upload_maximum_size: 4193404
    board_upload_speed: 460800
    build_flags: [
      "-DBOARD_HAS_PSRAM",
      "-DARDUINO_LOLIN_S3_MINI",
      "-DARDUINO_USB_MODE=1"
      ]

esp32:
  board: lolin_s3_mini
  variant: esp32s3
  flash_size: 4MB
  framework:
    type: arduino
    version: 2.0.8
    platform_version: 6.3.2

wifi:
  ssid: "SSID"
  password: !secret wifi_password

logger:
  level: DEBUG

api:
  #reboot_timeout: 0s # default
  encryption:
    key: !secret encryption_key

ota:
  platform: esphome
  password: esphome_recovery

i2c:
  id: i2c_bus
  sda: ${i2c_sda}
  scl: ${i2c_scl}
  scan: True
  frequency: 400kHz

packages:
  meter:
    url: https://github.com/tronikos/esphome-magnetometer-water-gas-meter
    ref: main
    file: esphome-water-meter.yaml
    # Or for gas meter:
    # file: esphome-gas-meter.yaml
    # Or if you are using HMC5883L instead of QMC5883L:
    # files: [esphome-water-meter.yaml, hmc5883l.yaml]
    refresh: 0s 

binary_sensor:
  - platform: template
    name: ${prefix_name}Flow Status
    lambda: |-
      if (id(${prefix_id}sensor_flow).state > 0) {
        return true;
      } else {
        return false;
      }
    
debug:
  update_interval: 5s

sensor:
  - platform: debug
    loop_time:
      name: "Loop Time"

  # loop time cut in half with internal: true
  - id: !extend ${prefix_id}qmc5883l_id
    field_strength_x:
      internal: true
    field_strength_y:
      internal: true
    field_strength_z:
      internal: true

[tronikos]

Great to hear you got it working! I'll update the README recommending a dual core ESP32.

It makes sense that internal: true is faster since ESPHome doesn't have to update Home Assistant. It seems when you disable an entity in Home Assistant, the integration keeps receiving updates from ESPHome, it just ignores them. I'll add a substitution to use internal: true by default and only expose them to HA if the user manually wants to calibrate. e.g. the calibrate button doesn't work well for them.

Try also adding this:

sensor:
  - id: !extend ${prefix_id}sensor_half_rotations_total
    internal: true

This will save cycles continuously updating HA when water is flowing. Let me know if it's helpful so that I add a substitution for this too.

Separately of the above try adding the following:

sensor:
  - id: !extend ${prefix_id}qmc5883l_id
    field_strength_x: !remove
    field_strength_y: !remove
    field_strength_z:
      internal: true

Make sure you adjust it first. The above assumes calibration picked the z axis.
This will save some more cycles updating the internal state of unused magnetic field sensors. If you see an improvement, I can add a recommendation in the README.

[tronikos]

I just committed:

1. Set logger level to info to speed up device
2. Mark some sensors internal to speed up device

On my ESP8266 there was a big improvement, especially with the 1st one. @aneisch make sure you remove the logger lines from your config. The loop time now remains constant around 10-12ms while water is running. Before it would jump to 50ms.

Removing field_strength_x and field_strength_y didn't have any noticeable impact for me.

@aneisch @jbronikowski could you please let me know what's the maximum flow you can measure with ESP8266, single core ESP32, and dual core ESP32 to update the README? All my faucets and shower heads are low flow.

[aneisch]

With latest changes (but hide half-rotations set to false as I didn't notice significant change):

S3 (dual core ESP32)): 25g/m 2-3ms avg loop time
S2 (single core ESP32): 25g/m 7-9ms avg loop time
ESP8266: 25g/m 4-5ms avg loop time

[tronikos]

What's your ESP8266? Getting 4-5ms is perfect. The fastest we can get data from the magnetometer is 5ms so any loop time less than that is perfect. Mine is a D1 mini and I get 10-12ms.

[aneisch]

Mine is a D1 mini pro. It looks like the clock speed is the same as a standard D1 mini though. Not sure how it's performing so well?

https://github.com/aneisch/home-assistant-config/blob/master/extras/esphome/water_meter8266.yaml

[tronikos]

I removed the api encryption and the loop time decreased by 2ms. Now it's 8-10ms. You might want to do the same as long as you understand the risks, i.e. any device in the same password protected WiFi network can access the ESP device. For me that's not a concern.