After around 1 day z2m is still running, but devices do not respond to commands

[nefh]

What happened?

Hi!
I've been using z2m for several years now. After recent update mine entire network stops responding to commands after several hours. The z2m itself seems working, there is no crash, UI is alright etc. Here is example, I've just tried toggle my light in bedroom, in the log it is being registered, but the actual bulb is not changing it's state:

[2024-11-17 20:35:19] �[34mdebug�[39m: 	z2m: Publishing 'set' 'state' to 'SypialniaSufitLampa'
[2024-11-17 20:35:19] �[34mdebug�[39m: 	zh:controller:endpoint: ZCL command 0x04cd15fffebf430f/1 genOnOff.on({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"reservedBits":0,"writeUndiv":false})
[2024-11-17 20:35:21] �[34mdebug�[39m: 	z2m:mqtt: Received MQTT message on 'zigbee2mqtt/0x04cd15fffebf430f/set' with data '{"state":"OFF"}'
[2024-11-17 20:35:21] �[34mdebug�[39m: 	z2m: Publishing 'set' 'state' to 'SypialniaSufitLampa'
[2024-11-17 20:35:21] �[34mdebug�[39m: 	zh:controller:endpoint: ZCL command 0x04cd15fffebf430f/1 genOnOff.off({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"reservedBits":0,"writeUndiv":false})
[2024-11-17 20:35:22] �[34mdebug�[39m: 	z2m:mqtt: Received MQTT message on 'zigbee2mqtt/0x04cd15fffebf430f/set' with data '{"state":"ON"}'
[2024-11-17 20:35:22] �[34mdebug�[39m: 	z2m: Publishing 'set' 'state' to 'SypialniaSufitLampa'
[2024-11-17 20:35:22] �[34mdebug�[39m: 	zh:controller:endpoint: ZCL command 0x04cd15fffebf430f/1 genOnOff.on({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"reservedBits":0,"writeUndiv":false})
[2024-11-17 20:35:23] �[34mdebug�[39m: 	z2m:mqtt: Received MQTT message on 'zigbee2mqtt/0x04cd15fffebf430f/set' with data '{"state":"OFF"}'
[2024-11-17 20:35:23] �[34mdebug�[39m: 	z2m: Publishing 'set' 'state' to 'SypialniaSufitLampa'
[2024-11-17 20:35:23] �[34mdebug�[39m: 	zh:controller:endpoint: ZCL command 0x04cd15fffebf430f/1 genOnOff.off({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"reservedBits":0,"writeUndiv":false})
[2024-11-17 20:35:26] �[34mdebug�[39m: 	z2m:mqtt: Received MQTT message on 'zigbee2mqtt/0x04cd15fffebf430f/set' with data '{"state":"ON"}'
[2024-11-17 20:35:26] �[34mdebug�[39m: 	z2m: Publishing 'set' 'state' to 'SypialniaSufitLampa'
[2024-11-17 20:35:26] �[34mdebug�[39m: 	zh:controller:endpoint: ZCL command 0x04cd15fffebf430f/1 genOnOff.on({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"reservedBits":0,"writeUndiv":false})

I can make the commands via Web UI, via homebridge plugin, via MQTT, but it doesn't matter, it won't. After service restart it will start respond again, but after several hours it will break again.

What did you expect to happen?

Work properly, as it was for many years :)

How to reproduce it (minimal and precise)

It happens by itself, just wait. Mine network has ~30 devices, mainly Ikea and Aqara. The last device I've added was Aqara TVOC sensor. Not sure if that matters.

Zigbee2MQTT version

1.41.0; I've tried also dev, the same issue

Adapter firmware version

20221226

Adapter

Launchpad 1352P2

Setup

FreeBSD jail

Debug log

zigbee2mqtt_daemon.zip

[purche42]

Maybe this is the same issue which we discuss here:
lights problem #24293

Z2M works stable since years - with 1.40.2+ I see the same problem with lights after some hours.

I also use Ikea and Aqara but also Sonoff Tuya innr devices and more.
I switched back to 1.40.1 or 1.40.0 lights responds to On/Off again.

Maybe you try to downgrade and check if this helps.

[nefh]

Maybe you try to downgrade and check if this helps.

I was able to downgrade to 1.38.0 (via snapshot) and confirm the problem was not existing then

How properly upgrade to 1.40.1? I've tried:

git tag
git checkout tags/1.40.1 -b v1.40.1
npm ci

However afterwards z2m could not find my coordinator:

z2m: Logging to console, file (filename: log.log)
z2m: Starting Zigbee2MQTT version 1.40.1 (commit #403d3c09)
z2m: Starting zigbee-herdsman (0.57.3)
zh:zstack:znp: Failed to determine if path is valid: 'Error: ENOENT: no such file or directory, lstat '/dev/ttyACM0''
zh:zstack:znp: Opening SerialPort with {"path":"/dev/ttyACM0","baudRate":115200,"rtscts":false,"autoOpen":false}
z2m: Error while starting zigbee-herdsman
z2m: Failed to start zigbee
z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
z2m: Exiting...
z2m: Error: Error: No such file or directory, cannot open /dev/ttyACM0

[purche42]

Hi nefh

It seems that your stick is not available at /dev/ttyACM0. Are you using CC2531 and have you checked if you see the device ?

I'm using docker container which allows easy up & downgrades. Everytime I pray that Z2M re-connects to my ZB Stick ;) -
I will not re-pair my ~75 devices. This is why I stay with a fixed Z2M version as long I do not add any new unsupported devices.
BTW mine is configured as /dev/ttyUSB0

I have switched to docker some years ago and did not install Z2M any more. Maybe saving the data directory and using docker is the easiest way or stay also at version 1.38 as long the problem is not solved.

Hope this helps you to get

[nefh]

Hi,
I've recently upgraded from 1.38.0 to 1.42.0 and this issue is again present. How can I investigate this issue? The UI is working, but no commands are being send to devices after some tim.

I am using FreeBSD, therefore there is no docker. /dev/ttyACM0 is the address mine Launchpad 1352P2 of course.

[nefh]

I can confirm that upgrade to Z2M 2.0(dev) solved this issue. Altough the info how to switch the branch and update is outdated (at least it did not work for FreeBSD).

[Koenkk]

I guess this was fixed by Koenkk/zigbee-herdsman#1256, thanks for confirming!

[FakeTruth]

I'm having the same problem with 1.40.1 and 1.40.2

At first everything works well, but after a few hours all devices stop responding to commands. I do get updates from them (temperatures etc.), but simply cannot interact with them anymore.

I believe, but not sure, I first encountered this problem after updating to 1.41.0 but thought it may be related to the Zigbee2MQTT 2.0 changes. I restored a backup and updated to 1.39.1 (yes I'm far behind). Now after some time I attempted to update again step by step. Now I'm on 1.40.2 and am having this problem again. Restarting Zigbee2MQTT also seems impossible through the web UI as it simply gets stuck and the UI becomes unreachable. The log only shows zh:zstack:* lines.

I will attempt to update to 2.0.0 now

UPDATE: Everything seems to work now :)