Testing: device availability

[Koenkk]

This allows zigbee2mqtt to mark individual devices as offline. E.g. when you switch a bulb off using the regular switch.

The device availability is published on zigbee2mqtt/[FRIENLDY_NAME]/availability with possible payloads being offline and online.

Home Assistant integration has been updated, devices will be shown as unavailable when they are offline.

Device availability is only checked for AC powered routers.

Todos

• Check device availability for new paired devices (without having to restart zigbee2mqtt)
• Mark device online when message is received
• Retrieve state after online
• Update documentation
• Allow to blacklist devices from being checked
• Availability timeout for non AC-powered devices (Availability Timeout not working #1020)
• Test, test and test

How to test
Update zigbee2mqtt to the latest dev branch (edge for hassio users). Stop zigbee2mqtt, add to zigbee2mqtt configuration.yaml

advanced:
  availability_timeout: 60 # Availability check interval

start zigbee2mqtt and restart Home Assistant.

Most of the credits of this feature go to @ugrug

[wimpie007]

doesnt work with hassio/zigbee2mqtt-edge.
Can't see anything different. (when is zigbee2mqtt-edge updated to most recent source?)

[Koenkk]

Should be, can you post your startup log?

[wimpie007]

sure.
https://pastebin.com/VaamLD4M
and in configuration.yaml:
experimental:
availablility_timeout: 60

regards,

[Koenkk]

@wimpie007 I see that your are running #2a197a4, this was introduced in a later commit #afeed4f372bced737206bd91f8e62ce65b127eb9 (this can be seen in the startup log).

Note that also a typo has just been fixed availablility -> availability, I've updated the configuration in the OP. Wait at least one hour to give the hassio addon some time to regenerate.

[wimpie007]

I stopped/started the zigbee2mqtt-edge addon in hassio this morningn (auto-update is on), but still on commit #2a197a4.
In hassio, i can't do more to 'force' the update?

[Koenkk]

I dont understand why, the image has been updated 14 hours ago which makes sense (https://hub.docker.com/r/dwelch2101/zigbee2mqtt-edge-armhf/tags). @danielwelch could you help?

[danielwelch]

I don’t think starting and stopping alone should work, as this doesn’t pull the new image. You need to uninstall and re-install.

[wimpie007]

Thanks! uninstall/install did the trick, bit annoying, but ok!:) will report later on....

[wimpie007]

doesnt seem to do anything...
here is the start-log.
https://pastebin.com/u5kANWCm

[Koenkk]

Can you post a bit more of you startup log? I'm especially interested in the homeassistant/ topic it publishes to.

[wimpie007]

sure!
https://pastebin.com/WmXvQ2Qz
PS Happy 2019!!:)

[Koenkk]

Happy 2019! Somehow the functionallity is not enabled yet (because it still publishes zigbee2mqtt/bridge/state as availabillity topic). Can you post the experimental part of your configuration.yaml again?

[wimpie007]

config in hass.io:
{
"data_path": "/share/zigbee2mqtt",
"homeassistant": true,
"permit_join": false,
"mqtt_base_topic": "zigbee2mqtt",
"mqtt_server": "mqtt://homeassistant:1883",
"serial_port": "/dev/ttyACM0",
"disable_led": true,
"log_level": "debug",
"zigbee_shepherd_debug": false
}

and config yaml:
https://pastebin.com/mtYCa2G4

EDIT!
DOH!
i did a paste from the instruction site, and copied also the typo:
availablility_timeout: 60

will test again with
availability_timeout

[wimpie007]

YES! we are getting somewhere!
for the lights it works! show up as "unavailable" when powered-off.
BUT... also for al the sensors.
Sensors show all up as unavailable as they do not respond to ping...
"Device availability is only checked for AC powered routers.": not true!:)

[Koenkk]

Can you post your log_level: debug log?

[wimpie007]

https://pastebin.com/bbu0MEnD
see for example device:
0x00158d00029c076d (0x00158d00029c076d): RTCGQ11LM - Xiaomi Aqara human body movement and illuminance sensor (EndDevice)

[Koenkk]

Found the bug, should be fixed

[wimpie007]

Koen, thanks!
it works now!
state of the lamps doesnt reflect reality after power off/on, but that is another issue...
a big step for this project! nice!

[Koenkk]

I think that could be fixed relatively easy, we just need to poll the state when yhe bulb comes back online. (unfortunately I dont have access to my laptop for a few days)

[martinrosenauer]

@Koenkk I can also confirm that it works for me on a bunch of different Trådfri bulbs as well as some Philips Hues. I don't see any updates for my battery driven sensors/buttons as expected..

It would be great if you also manage the state of the lights send to Home Assistant as mentioned by @wimpie007.

Under all circumstances - thanks for doing/maintaining this brilliant project !

[wimpie007]

Another improvement would be:

• faster convergence to state online by using "Received zigbee message"..."with data"... as 'online-trigger'
  Now, even when the "received zigbee message" is received, the device remains "unavailable" until the next ping cycle...

[mrwiwi]

Great feature !

[Iv4nS]

Question.. I have the E11-G13 - Sengled Element Classic (A19) (EndDevice) bulbs, these do not report when they are turned ON by the wall switch nor their status periodically.
The thing is, the type of this device is "EndDevice" so this code will not actually try pinging these bulbs. Should EndDevice be added or should I be using another method to detect if they are available or not?

[Koenkk]

@Iv4nS you should modify this function: https://github.com/Koenkk/zigbee2mqtt/blob/dev/lib/extension/deviceAvailability.js#L29 to also return true when the modelId is the one of your bulb.

[sehraf]

My OSRAM Smart+ plugs are detected (falsely) as offline from time to time. Does anybody else has observed this behavior?

EDIT: This is how it looks like (grey is offline)

there is something going on ...

[emontnemery]

@Koenkk Starting with Home Assistant 0.112.5, it's possible to configure multiple availability topics, for example:

availability:
 - topic: zigbee2mqtt/bridge/state
 - topic: zigbee2mqtt/0x00158d0001d0f272/availability

An updated availability on any configured topic will flick availabaility, so for this to work:

• zigbee2mqtt/bridge/state should only be used to send offline for MQTT will and before disconnecting zigbee2mqtt from MQTT
• zigbee2mqtt/0x00158d0001d0f272/availability should be used to send online and offline when connection is established or lost with the xigbee device

HA PR: home-assistant/core#37418

[Koenkk]

@emontnemery thanks, so if I understand correctly, I should change the following in the device discovery payload:

availability_topic: zigbee2mqtt/my_device/availability

to

availability: 
  - topic: zigbee2mqtt/my_device/availability
  - topic: zigbee2mqtt/bridge/state

If yes, for backwards compatibility purposes is it allowed to have both (will this work on both < 0.112.5 and 0.112.5 >=)?

availability_topic: zigbee2mqtt/my_device/availability
availability: 
  - topic: zigbee2mqtt/my_device/availability
  - topic: zigbee2mqtt/bridge/state

[emontnemery]

availability: 
  - topic: zigbee2mqtt/my_device/availability
  - topic: zigbee2mqtt/bridge/state

Exactly! Just take care not to send online to topic: zigbee2mqtt/bridge/state or all devices will be marked as available

If yes, for backwards compatibility purposes is it allowed to have both (will this work on both < 0.112.5 and 0.112.5 >=)?

No, that's unfortunetaly not allowed :(

[Koenkk]

@emontnemery thanks, I will set a reminder to update this after the 0.114 release (so everybody got time to update).

[DrWilder]

Thanks to @sjorge the availability feature is now also available for non-pingable devices. These will automatically be marked as unavailable when no messages has been received for 25 hours. Currently available in the latest dev.

For non-pingable devices I would suggest moving from a global "availability_timeout" to a device specific. That is, move this option under the individual device sections of the configuration file.

Justification is that some battery sensors update more regularly than others. For example Xiaomi temperature sensors are typically updating every 1000 seconds, while their contact sensors may only update every 5000 seconds.

[fdaemon]

Would it be possible to enable the "ping-poll" for some of the battery powered devices? Use case: contact or motion sensors, where it would be good to learn within a few minutes that the sensor is offline. For eg: loss of a contact/motion sensor can be a sign of a possible security breach/tampering.

[steve41200]

I have try to change type of Bulb which is often shutdown from Router to EndDevice to avoid some other device choose this router, but now i cannot ping them with availibility time out ... Is ther a way to ping specifically End Device ? (in my cas bulb wich is not on battery) @Koenkk ?

[Koenkk]

I have try to change type of Bulb which is often shutdown from Router to EndDevice to avoid some other device choose this router,

Don't do this, this will have zero effect

[steve41200]

Strange, in Zigbee2mqttassitant, it show my bulb in cercle since i have change device type in the database file and i have no arrow who go to in. It seems work. It's just an error of zigbeetomqttassistant ?

[emontnemery]

@Koenkk The possibility to define a list of availability topics was included in HA 0.112, and 0.117 will be released in a couple of days. I think it's OK to change zigbee2mqtt to use the list now.

[Koenkk]

@emontnemery yes you are right, implemented this. Thanks for pinging!

[Morphy99]

Would it be possible to show a device as unavailable if Z2M is unable to reach it? IE if a device is switched on and doesn't respond and throws up this message:

MQTT publish: topic 'zb2mqtt/bridge/log', payload '{"message":"Publish 'set' 'state' to 'Light Back Garden' failed: 'Error: Command 0x00158d0003441430/1 genOnOff.off({}, {\"timeout\":10000,\"disableResponse\":false,\"disableRecovery\":false,\"disableDefaultResponse\":false,\"direction\":0,\"srcEndpoint\":null,\"reservedBits\":0,\"manufacturerCode\":null,\"transactionSequenceNumber\":null}) failed (Timeout - 42692 - 1 - 182 - 6 - 11 after 10000ms)'","meta":{"friendly_name":"Light Back Garden"},"type":"zigbee_publish_error"}'

[Koenkk]

@Morphy99 there is already a feautre request for this: #5579

[ashar38]

Hello @Koenkk ,
Did not want to start a new issue for this.
Would it be possible to add another Sengled bulb (RGB) to the list of end devices.
zigbeeHerdsmanConverters.devices.find((d) => d.model === 'E11-N1EA')

[Koenkk]

@Indybeak done

[Drealine]

Hi,

You say this :
Non-pingable devices # Note that not all devices can be pinged for availability (e.g. battery powered devices). This category of devices will be marked as unavailable when no message has been received from them for 25 hours (persistent across restarts) This is only enabled when the availability feature is enabled.

Possible to make an option to change the timeout for non pingable devices ? I think is a best option if user use it with an alarm.

[ashar38]

@Indybeak done

@Koenkk
Thank you very much. Really appreciate you taking time out to make this change so soon.
Just two observations:

1. Looks like it is not published to the master branch.
2. On Dev, I guess you accidentally paste it twice. This is what I see. Although it should not make a material difference since it is an array.
   const pingableEndDevices = [
   zigbeeHerdsmanConverters.devices.find((d) => d.model === 'E11-N1EA'),
   zigbeeHerdsmanConverters.devices.find((d) => d.model === 'E11-G13'),
   zigbeeHerdsmanConverters.devices.find((d) => d.model === 'E11-N1EA'),
   zigbeeHerdsmanConverters.devices.find((d) => d.model === '53170161'),
   ];

[Koenkk]

@Indybeak thanks fixed.

Fix will be in master with next z2m release (1 feb)

[JanvdW304]

Should the availability_timeout parameter work with a Domoticz environment as well? I have added this parameter, but
Domoticz devices are not shown as unavailable when they are offline. See #5978.

[andrey-skat]

Non-pingable devices # Note that not all devices can be pinged for availability (e.g. battery powered devices). This category of devices will be marked as unavailable when no message has been received from them for 25 hours (persistent across restarts) This is only enabled when the availability feature is enabled.

Possible to make an option to change the timeout for non pingable devices ? I think is a best option if user use it with an alarm.

Same question. For water leak sensors 25 hours may be too late :)

[JanvdW304]

I don't worry about the battery powered devices, because I have a script that alerts me if one is not available for some time (not a water leak sensor by the way ;-) ). I want to check availability of the IKEA bulbs to detect when it's switched off. In case power is switched off manually Domoticz doesn't update the bulb status. The availability feature doesn't make a difference.

[steve41200]

I don't worry about the battery powered devices, because I have a script that alerts me if one is not available for some time (not a water leak sensor by the way ;-) ). I want to check availability of the IKEA bulbs to detect when it's switched off. In case power is switched off manually Domoticz doesn't update the bulb status. The availability feature doesn't make a difference.

Can you share the script ? how did you do that ? I try to create a think like this in Node red
Thank you

[JanvdW304]

I have used this post to set up my script. Works as expected