Z2M does not send device statuses on HA restart

[corporategoth]

What happened?

If Home Assistant restarts (running Z2M), all device states are "unknown" and not intractable. If Z2M restarts (running HA), all states are updated.

What did you expect to happen?

On EITHER Z2M or HA restart, the state of all devices should be sent to Home Assistant (via. MQTT). Home Assistant sends a birth message to aid in this (though they may need an 'afterbirth' message, for when they have completed doing their device detection / auto-configure). This used to happen a long time ago, not sure if a Z2M or HA change broke it.

Alternatively, it would be nice to have a separate button to send all device states to MQTT, without having to restart Z2M or go through re-discovery/autoconfiguration.

How to reproduce it (minimal and precise)

Restart Home Assistant without restarting Z2M.

Zigbee2MQTT version

1.37.1

Adapter firmware version

7.4.1.0

Adapter

ember

Setup

HA, Z2M and MQTT are in individual docker contasiners.

Debug log

No response

[francisp2]

This

Should match this

[corporategoth]

@francisp2 Thanks - they didn't match, but DO match now - I verified it in HA and Z2M.

However, while I see this in the Z2M log:
[2024-05-31 03:52:14] debug: z2m: Received MQTT message on 'homeassistant/status' with data 'online'
[2024-05-31 03:52:14] debug: z2m: Received MQTT message on 'homeassistant/status' with data 'online'

I still don't see the status of zigbee devices in HA after restarting HA.

I am assuming I don't need legacy attributes or triggers turned on.
And I'm assuming it doesn't matter if the status comes on hass/status or homeassistant/status.

So this issue is still valid, at least for me.

[LaurentChardin]

And I'm assuming it doesn't matter if the status comes on hass/status or homeassistant/status.

Correct. hass/status would come from your settings, and homeassistant/status will be tested anyway as well.

When MQTT is receiving the message on startup, Z2M will listen to it, and it triggers a republishing of all the devices states.

if ((data.topic === this.statusTopic || data.topic === defaultStatusTopic) &&
            data.message.toLowerCase() === 'online') {
            const timer = setTimeout(async () => {
                // Publish all device states.
                for (const entity of [...this.zigbee.devices(false), ...this.zigbee.groups()]) {
                    if (this.state.exists(entity)) {
                        this.publishEntityState(entity, this.state.get(entity), 'publishCached');
                    }
                }

                clearTimeout(timer);
            }, 30000);
        }

The publishing state is supposed to start 30s after receiving the message.

[corporategoth]

A fixed time might not work.

First because autoconf can take longer than that (even with recent improvements it takes 50 seconds on the Z2M side).

Also, that's just the time for Z2M to do it's autoconf publishing. Who knows how long it will take HA to do it's side of auto configuration and subscribe to the topics.

That's what I think is happening, even if statuses are send 30s after the birth, HA has not yet had time to process autoconf and subscribe to device topics.

On one hand, future HA is introducing device autoconf (instead of entity) which will help as I have 180 devices whis is more than 12000 entities. They are also adding a bunch of other performance improvements.

On the other hand, we should ask HA to add an afterbirth message sent when HA has finished autoconf and subscribed to all relevant topics. Z2M could then trigger off that for sending statuses without guessing the required delay.

[francisp2]

You could create an automation after start

  alias: Status bij opstarten HA
  trigger:
  - event: start
    platform: homeassistant
  condition: []
  action:
  - data:
      payload: 'online'
      topic: hass/status
    service: mqtt.publish

[LaurentChardin]

That's what I think is happening, even if statuses are send 30s after the birth, HA has not yet had time to process autoconf and subscribe to device topics.

I am not familiar with how and when HA is sending the birth message : i would assume that the message is sent when HA has completed all necessary setup. Maybe to check in the HA code when this is triggered.

Also this to follow : #21500 (comment)

[corporategoth]

@francisp2 Hrm, looks like you're right https://github.com/home-assistant/core/blob/ca89d22a34884b789a4154a3e426f93243041042/homeassistant/components/mqtt/client.py#L997 - the birth should be sent after discovery is done and subscriptions to device topics are done.

So I'm not 100% sure what's going on in my case.

[corporategoth]

I just double checked, ant yeah, I get the homeassistant/status online message,

[2024-06-01 08:07:49] debug:    z2m: Received MQTT message on 'homeassistant/status' with data 'online'

And 30s later, I see all my devices publish to their topics. So I guess I'll have to investigate this on HA's side, as Z2M does appear to do what it's supposed to.

[LaurentChardin]

@corporategoth out of curiosity, when you see the hass/status message being posted in MQTT, and that you also see that Z2M acknowledge this message in the logs, do you actually see the devices updates in MQTT being sent by Z2M ? (you could check this with MQTT explorer for example, and in the Z2M logs). Sure to be sure that your issue is on the HA side rather than Z2M side.

Nevermind :) you answered my question :)

[corporategoth]

Closing this. After upgrading Z2M to 1.38.0 (which reduced the auto detection cycle on startup to 5 seconds now instead of 50), and upgrading HA to 2024.6.0 (which includes a bunch of efficiency improvements to the MQTT interface), this seems to be working again properly.