Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mixing of two data points #79

Closed
Sineos opened this issue Dec 4, 2022 · 9 comments · Fixed by #80
Closed

Mixing of two data points #79

Sineos opened this issue Dec 4, 2022 · 9 comments · Fixed by #80
Labels
problem Something isn't working

Comments

@Sineos
Copy link

Sineos commented Dec 4, 2022

What happened?

  • I'm using a motion sensor, which is also reporting illuminance.
  • It is reporting illuminance_lux and illuminance as MQTT messages
  • In the zigbee2mqtt adapter these two values are represented as illuminance and illuminance_raw
  • In the node-red adapter, I'm processing the illuminance value
  • Whenever a new message arrives, I get two values in node-red: The value for illuminance and illuminance_raw

image

First one is the raw value and second one is the lux value

Also in the adapter itself, it seems that the raw value is not updated unless the adapter is restarted:
image
image

What did you expect to happen?

Both values are updated and separated accordingly.
Is there same regex matching that is too broad?

How to reproduce it (minimal and precise)

No response

Adapter version

abc738c (current git)

Log

zigbee2mqtt.0 2022-12-04 16:36:13.794 info MQTT publish: topic 'zigbee2mqtt/MotionFrontDoor/illuminance_lux', payload '2'
zigbee2mqtt.0 2022-12-04 16:36:13.794 debug Device: MotionFrontDoor/illuminance_lux not found, queue state in incStatsQueue!
zigbee2mqtt.0 2022-12-04 16:36:13.791 info MQTT publish: topic 'zigbee2mqtt/MotionFrontDoor/illuminance', payload '3064'
zigbee2mqtt.0 2022-12-04 16:36:13.790 debug Device: MotionFrontDoor/illuminance not found, queue state in incStatsQueue!
zigbee2mqtt.0 2022-12-04 16:36:13.789 info MQTT publish: topic 'zigbee2mqtt/MotionFrontDoor', payload '{"battery":100,"illuminance":3064,"illuminance_lux":2,"last_seen":"2022-12-04T16:36:13+01:00","linkquality":69,"motion_sensitivity":"high","occupancy":false,"occupancy_timeout":0,"temperature":3.07,"update":{"state":"idle"}}'
@Sineos Sineos added the problem Something isn't working label Dec 4, 2022
@o0shojo0o
Copy link
Contributor

The data points are determined by the exposes, so I don't really have to do much there and it works flawlessly with my motion sensor RTCGQ11LM.

Which one do you use?

Did you also set your z2m according to the configuration?
FAQ

@Sineos
Copy link
Author

Sineos commented Dec 5, 2022

Did you also set your z2m according to the configuration?
FAQ

In fact, I had output: attribute_and_json instead output: json. It does not make a difference that I could see and in the following only output: json is used.

Test setup

  • Simple node-red flow with 4 different motion sensors
  • Just takes the Iobroker data-point for illuminance and illuminance_raw and writes it back to the Iobroker log

image

Results

image

  • illuminance_raw is never processed
  • The Aqara sensors only return once. I think this is due to the fact that illuminance and illuminance_raw have the same value and same timestamp in JSON so it is already filtered by Iobroker
  • The Philips sensors have different values for illuminance and illuminance_raw. The value for illuminance_raw is never returned but both values are returned under the illuminance data-point

@o0shojo0o
Copy link
Contributor

Can you share the exposes that the device reports?
Instruction

@o0shojo0o o0shojo0o added device problem Data points or functionalities are missing and removed problem Something isn't working labels Dec 5, 2022
@Sineos
Copy link
Author

Sineos commented Dec 5, 2022

0x001788010bd25a8c (Philips 9290030674) exposes:

{
	"date_code": "20210830",
	"definition": {
		"description": "Hue motion outdoor sensor",
		"exposes": [
			{
				"access": 1,
				"description": "Measured temperature value",
				"name": "temperature",
				"property": "temperature",
				"type": "numeric",
				"unit": "°C"
			},
			{
				"access": 1,
				"description": "Indicates whether the device detected occupancy",
				"name": "occupancy",
				"property": "occupancy",
				"type": "binary",
				"value_off": false,
				"value_on": true
			},
			{
				"access": 1,
				"description": "Remaining battery in %",
				"name": "battery",
				"property": "battery",
				"type": "numeric",
				"unit": "%",
				"value_max": 100,
				"value_min": 0
			},
			{
				"access": 1,
				"description": "Measured illuminance in lux",
				"name": "illuminance_lux",
				"property": "illuminance_lux",
				"type": "numeric",
				"unit": "lx"
			},
			{
				"access": 1,
				"description": "Raw measured illuminance",
				"name": "illuminance",
				"property": "illuminance",
				"type": "numeric"
			},
			{
				"access": 7,
				"name": "motion_sensitivity",
				"property": "motion_sensitivity",
				"type": "enum",
				"values": [
					"low",
					"medium",
					"high",
					"very_high",
					"max"
				]
			},
			{
				"access": 7,
				"description": "Blink green LED on motion detection",
				"name": "led_indication",
				"property": "led_indication",
				"type": "binary",
				"value_off": false,
				"value_on": true
			},
			{
				"access": 7,
				"name": "occupancy_timeout",
				"property": "occupancy_timeout",
				"type": "numeric",
				"unit": "second",
				"value_max": 65535,
				"value_min": 0
			},
			{
				"access": 1,
				"description": "Link quality (signal strength)",
				"name": "linkquality",
				"property": "linkquality",
				"type": "numeric",
				"unit": "lqi",
				"value_max": 255,
				"value_min": 0
			}
		],
		"model": "9290030674",
		"options": [
			{
				"access": 2,
				"description": "Sends a message after the last time no occupancy (occupancy: false) was detected. When setting this for example to [10, 60] a `{\"no_occupancy_since\": 10}` will be send after 10 seconds and a `{\"no_occupancy_since\": 60}` after 60 seconds.",
				"item_type": "number",
				"name": "no_occupancy_since",
				"property": "no_occupancy_since",
				"type": "list"
			},
			{
				"access": 2,
				"description": "Number of digits after decimal point for temperature, takes into effect on next report of device.",
				"name": "temperature_precision",
				"property": "temperature_precision",
				"type": "numeric",
				"value_max": 3,
				"value_min": 0
			},
			{
				"access": 2,
				"description": "Calibrates the temperature value (absolute offset), takes into effect on next report of device.",
				"name": "temperature_calibration",
				"property": "temperature_calibration",
				"type": "numeric"
			},
			{
				"access": 2,
				"description": "Calibrates the illuminance value (percentual offset), takes into effect on next report of device.",
				"name": "illuminance_calibration",
				"property": "illuminance_calibration",
				"type": "numeric"
			},
			{
				"access": 2,
				"description": "Calibrates the illuminance_lux value (percentual offset), takes into effect on next report of device.",
				"name": "illuminance_lux_calibration",
				"property": "illuminance_lux_calibration",
				"type": "numeric"
			}
		],
		"supports_ota": true,
		"vendor": "Philips"
	},
	"endpoints": {
		"2": {
			"bindings": [
				{
					"cluster": "genPowerCfg",
					"target": {
						"endpoint": 1,
						"ieee_address": "0x00124b00237dfd02",
						"type": "endpoint"
					}
				},
				{
					"cluster": "msIlluminanceMeasurement",
					"target": {
						"endpoint": 1,
						"ieee_address": "0x00124b00237dfd02",
						"type": "endpoint"
					}
				},
				{
					"cluster": "msTemperatureMeasurement",
					"target": {
						"endpoint": 1,
						"ieee_address": "0x00124b00237dfd02",
						"type": "endpoint"
					}
				},
				{
					"cluster": "msOccupancySensing",
					"target": {
						"endpoint": 1,
						"ieee_address": "0x00124b00237dfd02",
						"type": "endpoint"
					}
				}
			],
			"clusters": {
				"input": [
					"genBasic",
					"genPowerCfg",
					"genIdentify",
					"msOccupancySensing",
					"msIlluminanceMeasurement",
					"msTemperatureMeasurement"
				],
				"output": [
					"genOta",
					"genBasic",
					"genIdentify",
					"genOnOff"
				]
			},
			"configured_reportings": [
				{
					"attribute": "batteryPercentageRemaining",
					"cluster": "genPowerCfg",
					"maximum_report_interval": 62000,
					"minimum_report_interval": 3600,
					"reportable_change": 0
				},
				{
					"attribute": "occupancy",
					"cluster": "msOccupancySensing",
					"maximum_report_interval": 3600,
					"minimum_report_interval": 0,
					"reportable_change": 0
				},
				{
					"attribute": "measuredValue",
					"cluster": "msTemperatureMeasurement",
					"maximum_report_interval": 3600,
					"minimum_report_interval": 10,
					"reportable_change": 100
				},
				{
					"attribute": "measuredValue",
					"cluster": "msIlluminanceMeasurement",
					"maximum_report_interval": 3600,
					"minimum_report_interval": 10,
					"reportable_change": 5
				}
			],
			"scenes": []
		}
	},
	"friendly_name": "MotionFrontDoor",
	"ieee_address": "0x001788010bd25a8c",
	"interview_completed": true,
	"interviewing": false,
	"manufacturer": "Signify Netherlands B.V.",
	"model_id": "SML004",
	"network_address": 14200,
	"power_source": "Battery",
	"software_build_id": "2.53.6",
	"supported": true,
	"type": "EndDevice"
}

IMO the issue is not limited to the Philips Motions Sensor but also the Aqara ones apparently do not correctly send the raw value (practically it does not matter, since both values are the same anyway)
For sake of completeness, here is the exposure of the Aqara P1 sensor as well:

0x54ef44100053c862 exposes:

{
	"date_code": "Jun 9 2022",
	"definition": {
		"description": "Aqara P1 human body movement and illuminance sensor",
		"exposes": [
			{
				"access": 1,
				"description": "Indicates whether the device detected occupancy",
				"name": "occupancy",
				"property": "occupancy",
				"type": "binary",
				"value_off": false,
				"value_on": true
			},
			{
				"access": 1,
				"description": "Measured illuminance in lux",
				"name": "illuminance_lux",
				"property": "illuminance",
				"type": "numeric",
				"unit": "lx"
			},
			{
				"access": 1,
				"description": "Measured illuminance in lux",
				"name": "illuminance",
				"property": "illuminance",
				"type": "numeric",
				"unit": "lx"
			},
			{
				"access": 7,
				"description": ". Press pairing button right before changing this otherwise it will fail.",
				"name": "motion_sensitivity",
				"property": "motion_sensitivity",
				"type": "enum",
				"values": [
					"low",
					"medium",
					"high"
				]
			},
			{
				"access": 7,
				"description": "Time interval for detecting actions. Press pairing button right before changing this otherwise it will fail.",
				"name": "detection_interval",
				"property": "detection_interval",
				"type": "numeric",
				"unit": "s",
				"value_max": 65535,
				"value_min": 2
			},
			{
				"access": 7,
				"description": "When this option is enabled then blue LED will blink once when motion is detected. Press pairing button right before changing this otherwise it will fail.",
				"name": "trigger_indicator",
				"property": "trigger_indicator",
				"type": "binary",
				"value_off": false,
				"value_on": true
			},
			{
				"access": 1,
				"description": "Temperature of the device",
				"name": "device_temperature",
				"property": "device_temperature",
				"type": "numeric",
				"unit": "°C"
			},
			{
				"access": 1,
				"description": "Remaining battery in %",
				"name": "battery",
				"property": "battery",
				"type": "numeric",
				"unit": "%",
				"value_max": 100,
				"value_min": 0
			},
			{
				"access": 1,
				"description": "Voltage of the battery in millivolts",
				"name": "voltage",
				"property": "voltage",
				"type": "numeric",
				"unit": "mV"
			},
			{
				"access": 1,
				"description": "Link quality (signal strength)",
				"name": "linkquality",
				"property": "linkquality",
				"type": "numeric",
				"unit": "lqi",
				"value_max": 255,
				"value_min": 0
			}
		],
		"model": "RTCGQ14LM",
		"options": [
			{
				"access": 2,
				"description": "Time in seconds after which occupancy is cleared after detecting it (default is \"detection_interval\" + 2 seconds). The value must be equal to or greater than \"detection_interval\", and it can also be a fraction.",
				"name": "occupancy_timeout",
				"property": "occupancy_timeout",
				"type": "numeric",
				"unit": "s",
				"value_min": 0,
				"value_step": 0.1
			},
			{
				"access": 2,
				"description": "Sends a message the last time occupancy (occupancy: true) was detected. When setting this for example to [10, 60] a `{\"no_occupancy_since\": 10}` will be send after 10 seconds and a `{\"no_occupancy_since\": 60}` after 60 seconds.",
				"item_type": "number",
				"name": "no_occupancy_since",
				"property": "no_occupancy_since",
				"type": "list"
			},
			{
				"access": 2,
				"description": "Calibrates the illuminance value (percentual offset), takes into effect on next report of device.",
				"name": "illuminance_calibration",
				"property": "illuminance_calibration",
				"type": "numeric"
			},
			{
				"access": 2,
				"description": "Calibrates the device_temperature value (absolute offset), takes into effect on next report of device.",
				"name": "device_temperature_calibration",
				"property": "device_temperature_calibration",
				"type": "numeric"
			}
		],
		"supports_ota": true,
		"vendor": "Xiaomi"
	},
	"endpoints": {
		"1": {
			"bindings": [],
			"clusters": {
				"input": [
					"genBasic",
					"genPowerCfg",
					"genIdentify",
					"aqaraOpple"
				],
				"output": [
					"genIdentify",
					"genOta",
					"aqaraOpple"
				]
			},
			"configured_reportings": [],
			"scenes": []
		}
	},
	"friendly_name": "MotionHallEG",
	"ieee_address": "0x54ef44100053c862",
	"interview_completed": true,
	"interviewing": false,
	"manufacturer": "LUMI",
	"model_id": "lumi.motion.ac02",
	"network_address": 43230,
	"power_source": "Battery",
	"software_build_id": "0.0.0_0006",
	"supported": true,
	"type": "EndDevice"
}

o0shojo0o added a commit that referenced this issue Dec 5, 2022
@o0shojo0o
Copy link
Contributor

o0shojo0o commented Dec 5, 2022

Thanks for your data, I was able to discover a possible problem in the state identification.

Please test this dev branch: https://github.com/o0shojo0o/ioBroker.zigbee2mqtt/tree/dev_%2379

@o0shojo0o o0shojo0o added problem Something isn't working and removed device problem Data points or functionalities are missing labels Dec 5, 2022
@Sineos
Copy link
Author

Sineos commented Dec 5, 2022

Many thanks, it looks like a definitive improvement 👍

image

Some questions remain:

Red marked log area:

  • Do the unknown keys have a deeper meaning?

Green marked log area:

  • MQTT illuminance seems mapped to Iobroker illuminance_raw

Yellow marked log area:

  • Seems correct but since both values are the same it is hard to tell

Blue marked log area:

  • Not sure where this is coming from. Seems to have no source

Need to do some further testing but the outdoor sensors report all 0 at the moment. Will check tomorrow

@o0shojo0o
Copy link
Contributor

First of all, the debug and info logs shown on the screenshot of zigbee2mqtt.0 come from your Zigbee2MQTT instance and are only forwarded by my adapter because the option Proxy Zigbee2MQTT Protokolle zu ioBroker Protokollen is set in the adapter configuration.

Red marked log area:

  • Do the unknown keys have a deeper meaning?

This is RAW data coming from the Zigbee network and then Zigbee2MQTT processes it to the nice JSON.
Why some keys are not processed I can't answer you, you have to ask Zigbee2MQTT.

Green marked log area:

  • MQTT illuminance seems mapped to Iobroker illuminance_raw

illuminance is always the raw form of the measurement from the device, which differs from manufacturer to manufacturer or from firmware to firmware, so I map this as illuminance_raw.
illuminance_lux is then if necessary the converted (from Zigbee2MQTT) value in lux .

Yellow marked log area:

  • Seems correct but since both values are the same it is hard to tell

Yes I assume so, since this is calculated by Zigbee2MQTT.

Blue marked log area:

  • Not sure where this is coming from. Seems to have no source

I can't tell you where your flow was triggered.
But two log entries before at least the raw data from the devices came in.

@Sineos
Copy link
Author

Sineos commented Dec 6, 2022

I have watched my logs now and I'm positive that your change actually solved the issues. This is an example for the Philips Sensors:

image

illuminance is always the raw form of the measurement from the device, which differs from manufacturer to manufacturer or from firmware to firmware, so I map this as illuminance_raw.
illuminance_lux is then if necessary the converted (from Zigbee2MQTT) value in lux .

While I fully agree with your logic, it "feels" a bit weird for me:
MQTT illuminance is mapped to Iobroker illuminance_raw
MQTT illuminance_lux mapped to Iobroker illuminance

May I propose to:
MQTT illuminance to be mapped to Iobroker illuminance
MQTT illuminance_lux to be mapped to Iobroker illuminance_lux

Or:

MQTT illuminance to be mapped to Iobroker illuminance_raw
MQTT illuminance_lux to be mapped to Iobroker illuminance_lux

Whereas I like the first proposal best because it keeps the closest "naming relationship" between MQTT and Iobroker.

In any case:
Many thanks for your support and for this incredibly useful adapter. 👍

@o0shojo0o
Copy link
Contributor

While I fully agree with your logic, it "feels" a bit weird for me:
........

This was deliberately chosen as it is also present in other adapters.

o0shojo0o added a commit that referenced this issue Dec 6, 2022
o0shojo0o added a commit that referenced this issue Dec 6, 2022
-   (o0shojo0o) better state identification ([#79](#79))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
problem Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants