[RFC] Exposing device capabilities

[Koenkk]

Currently there is no way for clients to determine the capabilities of a device. With this change I would like to introduce this. Please let me know what you think!

Benefits

• Avoid having to maintain a manual mapping for integrations e.g.:
  • homebridge-z2m @itavero
  • domoticz-zigbee2mqtt-plugin @stas-demydiuk
  • homeassistant.js
  • ioBroker.zigbee @kirovilya
• Provide more info on what values to expect for a certain device on the device specific pages
• Improve controlling of devices through the frontend (as currently it does not know what it can send) (@nurikk)
• Get rid of non-uniform supports attribute

Proposal

EDIT: All devices have exposes now, docs can be found here: https://github.com/Koenkk/zigbee2mqtt.io/blob/new_api/docs/information/exposes.md

[nurikk]

what about reading attributes from the device (something like power on state). Some devices can expose the same attribute for multiple endpoint (per endpoint configuration, like ZCL_CLUSTER_ID_GEN_ON_OFF_SWITCH_CONFIGfoe every button )

[nurikk]

For boolean(or other) attribute isn't clear whether the user can modify this state or no

[sjorge]

For boolean(or other) attribute isn't clear whether the user can modify this state or no

Yes! Having a ‘writable’ indicator is a good idea.
Maybe with options range for int, or enum for int and str.

E.g.

color_mode = read, int, enum [0=Hue/Sat, 1=CIExy,2=ColorTemp]

Lots of this is in the zcl, so some way to encode these in herdsman and then pull that info into the device info would he great.

The enum hits is great for things like z2m-frontend and for writing say a native node-red binding.

[kirovilya]

I have a positive attitude.
This partially overlaps with the classification of devices (as it is done here https://zigbee.blakadder.com/).
I also suggest putting this in devices.js as a list, since there are devices with different functionality. For example, distinguish light bulbs from sockets, switches or relays.

This all resembles the standardization of device capabilities, as is done in Apple HomeKit, Google Home, Amazon Alexa, Yandex Alice.
We declare what capabilities the device has, and the end systems themselves interpret how to display the work with this capability. If the device is specified as a "temperature sensor" or "socket with consumption measurement", then it is clear which attributes to display and which commands can be sent to the device.

But will we be able to describe all the capabilities of the device in this way? Basic - yes, but there are additional ones that are very different.
For example, it is possible to call selftest of a smoke detector or change its sensitivity, or switch the binding mode on the Opple remote control.
This should be described in more detail. And this possibility should have a name ("Sensor sensitivity", "Binding mode"), not just a type.

[Koenkk]

what about reading attributes from the device (something like power on state). Some devices can expose the same attribute for multiple endpoint (per endpoint configuration, like ZCL_CLUSTER_ID_GEN_ON_OFF_SWITCH_CONFIGfoe every button )

I don't understand what you mean here.

@kirovilya @sjorge @nurikk regarding the settable part, if we e.g. take the sensitivity of a sensor we can easily add it with:

{
    "type": "enum",
    "property": "sensitivity",
    "values": ["low", "medium", "high"],
    "direction": 2, // 0 = get only, 1 = set only, 2 = get and set
}

Selftest:

{
    "type": "enum",
    "property": "self_test",
    "values": [""],
    "direction": 1, // 0 = get only, 1 = set only, 2 = get and set
}

[nurikk]

Can we have typescript definitions for new exposes properties?

[kirovilya]

OK. I think I need to have an optional description of the property.

{
    "type": "enum",
    "property": "self_test",
    "description": "Invoke self-test mode (send any value)",
    "values": [""],
    "direction": 1, // 0 = get only, 1 = set only, 2 = get and set
}

[Koenkk]

@nurikk typescript defs shouldn't be a problem

@kirovilya description field is no problem!

Since I believe we agree on the general idea and structure, I will start implementing this for lights (since those represent the biggest share of supported devices).

[sjorge]

What is the plan for groups? For uniform groups we can probably publish the info too, not sure for mixed groups... only the common ones?

E.g. 4 bulbs that are ws and one that is cws, we could still publish the ws part as all 5 support it?

But probably something for after we have devices exposed

[Koenkk]

@sjorge lets do that after this indeed, I think we can also add Home Assistant discovery then.

[Koenkk]

exposes is now available for all lights (+- 50% of the supported devices). Docs: https://github.com/Koenkk/zigbee2mqtt.io/blob/new_api/docs/information/mqtt_topics_and_message_structure.md#exposes

[nurikk]

I would like to see implementation of sensors and "set converters" (ex: power on state, how to read/set values)

[nurikk]

First try, already available in dev

[nurikk]

Currently implemented color_xy, color_temp, state, brightness
Kindly share device states with other color_XX features (I don't have anything but color_xy)

[Koenkk]

Also added for switch devices!

[sjorge]

@Koenkk perhaps we should split out the 'color' convert into one for xy and one for hue/enhanced hue...

Hitting refresh on the xy one now seems to read both, also it looks like the redfix does not get applied when going through the frontend? It does when set via mqtt though.

Thats with an RB 250 C or RB 285 C

[Koenkk]

@sjorge there should be no difference with settings vs MQTT vs frontend. I'm not sure if I understood you correctly but if you want a separate refresh option this is something that has to be implemented in the frontend so it should be requested here: https://github.com/nurikk/z2m-frontend/issues

[hidaris]

@Koenkk Hi, I tried the new expose api but didn't find an easy way to determine the device type, is there a way to provide it, otherwise I seem to need to do some judging based on the nested json rules.

[itavero]

@hidaris What exactly do you mean by "device type"?

[hidaris]

@itavero Like is this some kind of sensor ? For now, if the expose contains features I will know that it is a specific type, but if it is a sensor, do I need to know if it is a TemperatureSensor or ContactSensor by property name?

[itavero]

@hidaris I'm currently using the name for that.

The thing is that just saying it's a Temperature sensor or Contact sensor, still doesn't give you any guarantee as to what kind of features you can expect. For instance, some temperature sensors also provide Relative Humidity and/or Air Pressure.

To make it even more complex: I also have a Motion Sensor, that also provides temperature and illuminance information. So, how would you list that?

As far as I can tell from looking at the specifications and the data provided for all the devices I have, the safest way to determine the capabilities is looking at the name field. The property might change if you have multiple endpoints, for instance.

[sjorge]

To make it even more complex: I also have a Motion Sensor, that also provides temperature and illuminance information. So, how would you list that?

I expose my hue motion like this:

MotionSensor (parent service) linked to it are TemperatureSensor (linked service) IlluminanceSensor (linked service) and a BatteryService (linked service)

It shows up correctly for me in homekit and I can split or group them depending on the preference pre iOS 14, so I think his can indeed be done based on name, my manual rule for now is Motion/Occupancy > Temperature > Humidity > Illuminace for which becomes the parents

[hidaris]

@hidaris I'm currently using the name for that.

The thing is that just saying it's a Temperature sensor or Contact sensor, still doesn't give you any guarantee as to what kind of features you can expect. For instance, some temperature sensors also provide Relative Humidity and/or Air Pressure.

To make it even more complex: I also have a Motion Sensor, that also provides temperature and illuminance information. So, how would you list that?

As far as I can tell from looking at the specifications and the data provided for all the devices I have, the safest way to determine the capabilities is looking at the name field. The property might change if you have multiple endpoints, for instance.

Yes, I see what you mean, but in a practical adaptation (e.g. web of things), perhaps I could choose to have multiple types of such sensors.

[itavero]

@sjorge @hidaris I know you can have multiple services in things like HomeKit (I also have that in the homebridge-z2m plugin). I'm just trying to say that having yet another field to deduct information from won't really bring you anything more than the name field currently does.

[Anjerlaan]

@Koenkk, Is this already fully operational in the official release 1.16.2 or is the dev branch still required?
I ask this because the domoticz-zigbee2mqtt-plugin is generating errors, this happens because (according to a user):

After update Zigbee2Mqtt to 1.16.2, the answer of the request to "bridge/config/devices" is not longer in the "bridge/log"

Did this changed in version 1.16.2 (when using the new_api or in general)?

[Koenkk]

@Anjerlaan new api will be officialy released on 1 January. Regarding the old api this still works however it seems that earlier Zigbee2MQTT versions had a bug allowing to request a zigbee2mqtt/bridge/config/devices by sending a message to zigbee2mqtt/bridge/config/devices, this was never intended to work and should be zigbee2mqtt/bridge/config/devices/get (https://www.zigbee2mqtt.io/information/mqtt_topics_and_message_structure.html#zigbee2mqttbridgeconfigdevicesget)

[Anjerlaan]

@Koenkk, so if I understand you correct, in version 1.16.2, the bug of the "zigbee2mqtt/bridge/config/devices" is already changed to "zigbee2mqtt/bridge/config/devices/get"?

Does this bug also counted for groups?

[Koenkk]

@Anjerlaan

so if I understand you correct, in version 1.16.2, the bug of the "zigbee2mqtt/bridge/config/devices" is already changed to "zigbee2mqtt/bridge/config/devices/get"?

yes

Does this bug also counted for groups?

For groups it is zigbee2mqtt/bridge/config/groups (I expect this should still work). Note that it's not consistent with requesting devices.

[Anjerlaan]

This is correct, because when I did a fresh install of the domoticz-Z2M plug-in, the group devices were recognized but not the devices.

[TaGGoU91]

Hello,

I have an issue. i have the parameter new_api : true, but my devices are still with access: r / w / rw, some friends have the same configuration, and they have access :1 / 2 / 7 for example.
I have tried to follow the links you shared about the new api to find the match between R/1 W/7 for exemple, but the links are all dead.

Do you know where i can find some information on this subject ?
I am trying to make a plugin for some home automation controler and need this ;).

Thanks for your help
Regards
TaG

[TaGGoU91]

Well the solution for the access : number instead of r or w or rw is ok. i needed a docker pull, but i still need access to the api to matchs :
r/1
w/?
rw/???

Thanks ;)

[nurikk]

Well the solution for the access : number instead of r or w or rw is ok. i needed a docker pull, but i still need access to the api to matchs :
r/1
w/?
rw/???

Thanks ;)

Here it is
https://github.com/Koenkk/zigbee2mqtt.io/blob/develop/docs/information/exposes.md#exposes

[TaGGoU91]

Hello,

Many thanks, i am going to read that :).

Thanks :)

[lionel590]

Hi Koenkk,

in order to be able to "create" a device automatically in a "home automation" solution we need to be able to distguish exposed capabilities from informative ones to primary ones. Let me take an exemple. Philipps hue motion detector exposes:

{"access":1,"description":"Measured temperature value","name":"temperature","property":"temperature","type":"numeric","unit":"°C"},
{"access":1,"description":"Indicates whether the device detected occupancy","name":"occupancy","property":"occupancy","type":"binary","value_off":false,"value_on":true},
{"access":1,"description":"Remaining battery in %","name":"battery","property":"battery","type":"numeric","unit":"%","value_max":100,"value_min":0},
{"access":1,"description":"Measured illuminance in lux","name":"illuminance_lux","property":"illuminance_lux","type":"numeric","unit":"lx"},
{"access":1,"description":"Raw measured illuminance","name":"illuminance","property":"illuminance","type":"numeric"},
{"access":7,"name":"motion_sensitivity","property":"motion_sensitivity","type":"enum","values":["low","medium","high"]},
{"access":7,"name":"occupancy_timeout","property":"occupancy_timeout","type":"numeric","unit":"second","value_max":65535,"value_min":0},
{"access":1,"description":"Link quality (signal strength)","name":"linkquality","property":"linkquality","type":"numeric","unit":"lqi","value_max":255,"value_min":0}

I want to create automatically the GUI showing Temp, LUX, Motion as primary sensor and for exemple Version, Vendor, Model as device information. So we would need to add a parameter such as "feature:primary", "feature:info"
This will allow to split between pure information and device "sensors" see bellow

What do you think ?

[Koenkk]

@lionel590 I don't think it's up to Zigbee2MQTT to define what to show in a certain GUI. I suggest you keep a mapping in your application of "core" features (e.g. temperature, lux, humidity, motion, switch)

[lionel590]

Hi Koenkk thx for your reply I agree that's hwat I do today, you made a valid point

[Koenkk]

Good, since the API has now officially been released and is enabled for everyone (with the 1.17.0 release) I will close this issue. Even though being closed, feel free to ask any questions here.

[JKRhb]

Hi! Thanks for adding in this cool feature! I am currently dealing with the recently published Web of Things Thing Description (WoT TD) and I was wondering if it would be possible to use/extend the Exposes API to also provide TDs for the connected devices. Using the (upcoming) Discovery mechanisms and the already existing Scripting API it would very easy to interact with the devices. I therefore wanted to ask if you see the possibility to include this either directly or as an add-on as I see a lot of potential in the new standard.