Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Popp/Danfoss TRVs become unresponsive with CC2652P #13478

Closed
moster11 opened this issue Aug 9, 2022 · 258 comments · Fixed by Koenkk/zigbee-herdsman#680
Closed

Popp/Danfoss TRVs become unresponsive with CC2652P #13478

moster11 opened this issue Aug 9, 2022 · 258 comments · Fixed by Koenkk/zigbee-herdsman#680
Labels
problem Something isn't working

Comments

@moster11
Copy link

moster11 commented Aug 9, 2022

What happened?

I have a setup with several Popp-TRVs (identical to Danfoss Ally TRVs), lots of Aqara sensors and two routers, all connected to a Sonoff USB Dongle 3.0 Plus. I run Home Assistant and have automations that send temperatures measured by the Aqara sensors to the external_measured_room_sensor endpoint of the TRVs.

Everything runs smoothly for some time - until the TRVs stop reacting to any command sent to them. They still report their states without any issue, it's just that I cannot send any command to them any more. Most notably this ONLY happens to TRVs which are connected directly to the coordinator, never if connected to a router.

Looking at the devices tab shows recent "Last seen" values. I highlighted "Thermostat Badezimmer Fenster" as that's the one that's unresponsive (did an initial re-pairing of all devices this morning):

183062004-cf64e4a1-6a94-4db8-b79c-1f92ad40371f

Looking at the map gives me a clear sign that this TRV will not react to my commands anymore. LQI is shown as 0 here, while the previous picture showed LQI 72:

183062402-7ac67ea2-0fec-446c-ad33-8cff9e321cb2

I tried Z2M versions 1.27.0 and 1.26.0. The Sonoff dongle was on ZStack 20220219, now upgraded to dev branch 20220726. I resetted and repaired all devices numerous times, even changed to position of the coordinator to the other side of the house. Nothing helps, it's always the devices connected to the coordinator that stop reacting after some hours, at max one or two days. Only way to make them react again for some time is to take the batteries in and out.

I completely reinstalled Z2M 1.27.0 and connected all devices to the coordinator directly. The result is the same - after some hours the first TRVs become unresponsive but happily send their states.

In the debug log can see some "data confirm errors" which might give a clue.

What did you expect to happen?

I expect the devices always react to the commands I send to them and not become unresponsive.

How to reproduce it (minimal and precise)

Just setup the TRVs, make sure they are connected to the coordinator, and wait some hours.

Zigbee2MQTT version

1.27.0

Adapter firmware version

20220219 and dev 20220726

Adapter

Sonoff USB Dongle 3.0 plus

Debug log

2022-08-09T22:17:35.975Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,1,100,1,0,100]
2022-08-09T22:17:35.975Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 1 - 3 - 4 - 1 - [0] - 100
2022-08-09T22:17:35.975Z zigbee-herdsman:adapter:zStack:znp:SRSP <-- AF - dataRequest - {"status":0}
2022-08-09T22:17:35.975Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:37.682Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,3,68,128,205,1,13,6]
2022-08-09T22:17:37.682Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,3,68,128,205,1,13,6]
2022-08-09T22:17:37.684Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 3 - 2 - 4 - 128 - [205,1,13] - 6
2022-08-09T22:17:37.684Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - dataConfirm - {"status":205,"endpoint":1,"transid":13}
2022-08-09T22:17:37.684Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:37.685Z zigbee-herdsman:adapter:zStack:adapter Data confirm error (0xb4e3f9fffedf0571:10453,205,2)
2022-08-09T22:17:37.685Z zigbee-herdsman:adapter:zStack:adapter Request network address of '0xb4e3f9fffedf0571'
2022-08-09T22:17:37.686Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - nwkAddrReq - {"ieeeaddr":"0xb4e3f9fffedf0571","reqtype":0,"startindex":0}
2022-08-09T22:17:37.686Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,10,37,0,113,5,223,254,255,249,227,180,0,0,43]
2022-08-09T22:17:37.701Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,1,101,0,0,100]
2022-08-09T22:17:37.701Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,1,101,0,0,100]
2022-08-09T22:17:37.701Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 1 - 3 - 5 - 0 - [0] - 100
2022-08-09T22:17:37.701Z zigbee-herdsman:adapter:zStack:znp:SRSP <-- ZDO - nwkAddrReq - {"status":0}
2022-08-09T22:17:37.701Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:40.813Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,23,68,129,0,0,32,0,113,235,1,242,0,87,0,126,88,7,0,0,3,9,96,0,113,235,29,0]
2022-08-09T22:17:40.814Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,23,68,129,0,0,32,0,113,235,1,242,0,87,0,126,88,7,0,0,3,9,96,0,113,235,29,0]
2022-08-09T22:17:40.814Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 23 - 2 - 4 - 129 - [0,0,32,0,113,235,1,242,0,87,0,126,88,7,0,0,3,9,96,0,113,235,29] - 0
2022-08-09T22:17:40.815Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - incomingMsg - {"groupid":0,"clusterid":32,"srcaddr":60273,"srcendpoint":1,"dstendpoint":242,"wasbroadcast":0,"linkquality":87,"securityuse":0,"timestamp":481406,"transseqnumber":0,"len":3,"data":{"type":"Buffer","data":[9,96,0]}}
2022-08-09T22:17:40.816Z zigbee-herdsman:controller:log Received 'zcl' data '{"frame":{"Header":{"frameControl":{"frameType":1,"manufacturerSpecific":false,"direction":1,"disableDefaultResponse":false,"reservedBits":0},"transactionSequenceNumber":96,"manufacturerCode":null,"commandIdentifier":0},"Payload":{},"Command":{"ID":0,"parameters":[],"name":"checkin"}},"address":60273,"endpoint":1,"linkquality":87,"groupID":0,"wasBroadcast":false,"destinationEndpoint":242}'
2022-08-09T22:17:40.818Z zigbee-herdsman:controller:device:log check-in from 0x0c4314fffe45317c: declining fast-poll
2022-08-09T22:17:40.818Z zigbee-herdsman:controller:endpoint Command 0x0c4314fffe45317c/1 genPollCtrl.checkinRsp({"startFastPolling":false,"fastPollTimeout":0}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false})
2022-08-09T22:17:40.818Z zigbee-herdsman:adapter:zStack:adapter sendZclFrameToEndpointInternal 0x0c4314fffe45317c:60273/1 (0,0,2)
2022-08-09T22:17:40.819Z zigbee-herdsman:adapter:zStack:znp:SREQ --> AF - dataRequest - {"dstaddr":60273,"destendpoint":1,"srcendpoint":1,"clusterid":32,"transid":14,"options":0,"radius":30,"len":6,"data":{"type":"Buffer","data":[1,6,0,0,0,0]}}
2022-08-09T22:17:40.819Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,16,36,1,113,235,1,1,32,0,14,0,30,6,1,6,0,0,0,0,158]
2022-08-09T22:17:40.819Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:40.835Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,1,100,1,0,100]
2022-08-09T22:17:40.835Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,1,100,1,0,100]
2022-08-09T22:17:40.835Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 1 - 3 - 4 - 1 - [0] - 100
2022-08-09T22:17:40.836Z zigbee-herdsman:adapter:zStack:znp:SRSP <-- AF - dataRequest - {"status":0}
2022-08-09T22:17:40.836Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:41.170Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,3,68,128,233,1,14,33]
2022-08-09T22:17:41.170Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,3,68,128,233,1,14,33]
2022-08-09T22:17:41.170Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 3 - 2 - 4 - 128 - [233,1,14] - 33
2022-08-09T22:17:41.170Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - dataConfirm - {"status":233,"endpoint":1,"transid":14}
2022-08-09T22:17:41.170Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:41.171Z zigbee-herdsman:adapter:zStack:adapter Data confirm error (0x0c4314fffe45317c:60273,233,0)
2022-08-09T22:17:41.171Z zigbee-herdsman:adapter:zStack:adapter Wait 2000ms
2022-08-09T22:17:43.173Z zigbee-herdsman:adapter:zStack:adapter sendZclFrameToEndpointInternal 0x0c4314fffe45317c:60273/1 (0,1,2)
2022-08-09T22:17:43.173Z zigbee-herdsman:adapter:zStack:znp:SREQ --> AF - dataRequest - {"dstaddr":60273,"destendpoint":1,"srcendpoint":1,"clusterid":32,"transid":15,"options":0,"radius":30,"len":6,"data":{"type":"Buffer","data":[1,6,0,0,0,0]}}
2022-08-09T22:17:43.174Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,16,36,1,113,235,1,1,32,0,15,0,30,6,1,6,0,0,0,0,159]
2022-08-09T22:17:43.190Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,1,100,1,0,100]
2022-08-09T22:17:43.190Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,1,100,1,0,100]
2022-08-09T22:17:43.190Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 1 - 3 - 4 - 1 - [0] - 100
2022-08-09T22:17:43.190Z zigbee-herdsman:adapter:zStack:znp:SRSP <-- AF - dataRequest - {"status":0}
2022-08-09T22:17:43.190Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:43.414Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,3,68,128,233,1,15,32]
2022-08-09T22:17:43.414Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,3,68,128,233,1,15,32]
2022-08-09T22:17:43.414Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 3 - 2 - 4 - 128 - [233,1,15] - 32
2022-08-09T22:17:43.414Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - dataConfirm - {"status":233,"endpoint":1,"transid":15}
2022-08-09T22:17:43.414Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:43.415Z zigbee-herdsman:adapter:zStack:adapter Data confirm error (0x0c4314fffe45317c:60273,233,1)
2022-08-09T22:17:43.415Z zigbee-herdsman:adapter:zStack:adapter Discovering route to 60273
2022-08-09T22:17:43.416Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - extRouteDisc - {"dstAddr":60273,"options":0,"radius":30}
2022-08-09T22:17:43.416Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,4,37,69,113,235,0,30,224]
2022-08-09T22:17:43.430Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,1,101,69,0,33]
2022-08-09T22:17:43.431Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,1,101,69,0,33]
2022-08-09T22:17:43.431Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 1 - 3 - 5 - 69 - [0] - 33
2022-08-09T22:17:43.431Z zigbee-herdsman:adapter:zStack:znp:SRSP <-- ZDO - extRouteDisc - {"status":0}
2022-08-09T22:17:43.431Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:46.433Z zigbee-herdsman:adapter:zStack:adapter sendZclFrameToEndpointInternal 0x0c4314fffe45317c:60273/1 (0,2,2)
2022-08-09T22:17:46.433Z zigbee-herdsman:adapter:zStack:znp:SREQ --> AF - dataRequest - {"dstaddr":60273,"destendpoint":1,"srcendpoint":1,"clusterid":32,"transid":16,"options":0,"radius":30,"len":6,"data":{"type":"Buffer","data":[1,6,0,0,0,0]}}
2022-08-09T22:17:46.433Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,16,36,1,113,235,1,1,32,0,16,0,30,6,1,6,0,0,0,0,128]
2022-08-09T22:17:46.450Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,1,100,1,0,100]
2022-08-09T22:17:46.450Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,1,100,1,0,100]
2022-08-09T22:17:46.450Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 1 - 3 - 4 - 1 - [0] - 100
2022-08-09T22:17:46.451Z zigbee-herdsman:adapter:zStack:znp:SRSP <-- AF - dataRequest - {"status":0}
2022-08-09T22:17:46.451Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:46.674Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,3,68,128,233,1,16,63]
2022-08-09T22:17:46.675Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,3,68,128,233,1,16,63]
2022-08-09T22:17:46.675Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 3 - 2 - 4 - 128 - [233,1,16] - 63
2022-08-09T22:17:46.675Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - dataConfirm - {"status":233,"endpoint":1,"transid":16}
2022-08-09T22:17:46.675Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:46.675Z zigbee-herdsman:adapter:zStack:adapter Data confirm error (0x0c4314fffe45317c:60273,233,2)
2022-08-09T22:17:46.675Z zigbee-herdsman:adapter:zStack:adapter Request network address of '0x0c4314fffe45317c'
2022-08-09T22:17:46.675Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - nwkAddrReq - {"ieeeaddr":"0x0c4314fffe45317c","reqtype":0,"startindex":0}
2022-08-09T22:17:46.676Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,10,37,0,124,49,69,254,255,20,67,12,0,0,125]
2022-08-09T22:17:46.682Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,1,101,0,0,100]
2022-08-09T22:17:46.682Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,1,101,0,0,100]
2022-08-09T22:17:46.682Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 1 - 3 - 5 - 0 - [0] - 100
2022-08-09T22:17:46.682Z zigbee-herdsman:adapter:zStack:znp:SRSP <-- ZDO - nwkAddrReq - {"status":0}
2022-08-09T22:17:46.683Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:47.703Z zigbee-herdsman:adapter:zStack:adapter sendZclFrameToEndpointInternal 0xb4e3f9fffedf0571:10453/1 (0,3,2)
2022-08-09T22:17:47.704Z zigbee-herdsman:adapter:zStack:znp:SREQ --> AF - dataRequest - {"dstaddr":10453,"destendpoint":1,"srcendpoint":1,"clusterid":32,"transid":17,"options":0,"radius":30,"len":6,"data":{"type":"Buffer","data":[1,5,0,0,0,0]}}
2022-08-09T22:17:47.704Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,16,36,1,213,40,1,1,32,0,17,0,30,6,1,5,0,0,0,0,229]
2022-08-09T22:17:47.720Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,1,100,1,0,100]
2022-08-09T22:17:47.721Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,1,100,1,0,100]
2022-08-09T22:17:47.721Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 1 - 3 - 4 - 1 - [0] - 100
2022-08-09T22:17:47.721Z zigbee-herdsman:adapter:zStack:znp:SRSP <-- AF - dataRequest - {"status":0}
2022-08-09T22:17:47.721Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:49.165Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,3,68,128,205,1,17,26]
2022-08-09T22:17:49.165Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,3,68,128,205,1,17,26]
2022-08-09T22:17:49.165Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 3 - 2 - 4 - 128 - [205,1,17] - 26
2022-08-09T22:17:49.165Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - dataConfirm - {"status":205,"endpoint":1,"transid":17}
2022-08-09T22:17:49.165Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-08-09T22:17:49.165Z zigbee-herdsman:adapter:zStack:adapter Data confirm error (0xb4e3f9fffedf0571:10453,205,3)
2022-08-09T22:17:49.166Z zigbee-herdsman:adapter:zStack:adapter Wait 2000ms

@moster11 moster11 added the problem Something isn't working label Aug 9, 2022
@TheLongAndOnly
Copy link

I can second this. Similar issue, Danfoss Ally, Popp and Eurotronic TRV stop communicating after a while of flawless operation. I'm using a ConBeeII. Only way top fix I found so far is to rejoin the Zigbee network again. At least in my case the connections also run through routers.

@wormiedk
Copy link

I have the same problem. I was hoping it was a cc2652 problem so that I could solve it by switching adapter. I might have to go back to zha instead then

@moster11
Copy link
Author

moster11 commented Oct 3, 2022

I found a solution for my use case. I had Zigbee2MQTT installed in Home Assistant OS which runs on a VM on my Synology NAS (DSM 7).

No matter if I used a Conbee II or the Sonoff Dongle I encountered the bespoken dropouts. As soon as I installed Zigbee2MQTT as a Docker image and set up my TRVs from there everything suddenly started to run without any flaws. Rock solid setup for more than 4 weeks already.

Seems to me that the USB passthrough to the VM was the root cause.

@TheLongAndOnly
Copy link

Interesting, but I'm running z2m directly installed on a Raspi4. Maybe I should try docker or check, if there are options to improve USB behavior. Figured out that removing batteries from the devices also fixed the problem.

@TheLongAndOnly
Copy link

I have a ZigBee sniffer available, but I'm not very familiar with the protocol. Is there anything I should watch for?

@DonDonatello
Copy link

DonDonatello commented Oct 5, 2022

Experienced this as well with both the Sonoff and Conbee II stick. Newest (1.28) stable Z2M running in Docker on RPi4 and all devices connected directly to the coordinator.

Happy to produce any relevant logs if needed.

@lukas-mc
Copy link

lukas-mc commented Oct 6, 2022

I see the same here. SonOff Coordinator with lates firmware. I have Popp and Danfoss Ally TRV which should be the same inside.
Some of the TRV are connected over Routers (ledadvance and lights) others are directly connected to the coordinator.

I see the ,,pubish" event in logs but npo communication to or back from the device. Poer off the device for some time (10 to 15 min) and it will be responive again.

Regards Lukas

@xeri-sk
Copy link

xeri-sk commented Oct 17, 2022

I have same problem with Danfoss Ally TVR. After some time (hours, days) TVR still reporting states, but not responding for any command sent to TVR (through openhab or Z2M frontend).

When this happened to TVR i have to remove it from Z2M, factory reset TVR and re-join.
I hope that somebody will fix it soon couse i bought 3 TVR devices and are not usable with Z2M now.

My enviroment: OS Linux (Debian), Sonoff USB Dongle 3.0 Plus (with USB extension) FW 20220219, Z2M 1.28.0, openhab 3.3.0.

@nikolai5slo
Copy link

Also happening to me. I have two Ally TRV and both stop responding after few hours. Restarting Z2M doesn't help. Only factory reset of TRV + re-join helps. I am not having any connection or signal issues with any other devices (around 20 devices on the network).

@wormiedk
Copy link

I switched to ZHA and I don't have the problem there

@stachjankowski
Copy link

stachjankowski commented Oct 19, 2022

In my case, it helped removing the external antenna (reduce the range of the coordinator).
Now all TRVs connect through routers (IKEA bulbs, TUYA Smart plug, etc).

My enviroment: QNAP (docker), Sonoff ZBDongle CC2652P FW 20220219, Z2M 1.28.0.

@TheLongAndOnly
Copy link

TheLongAndOnly commented Oct 20, 2022

I did some more research. I sniffed the network while the Danfoss Ally (0x4d8e) did not work anymore. I see data requests from the device reaching out to 0xd1ac, which is a GLEDOPTO GL-C-009 (Low Voltage LED Driver)

1118 251.639075 0x4d8e 0xd1ac IEEE 802.15.4 12 Data Request
1138 258.690621 0x4d8e 0xd1ac IEEE 802.15.4 12 Data Request
1167 265.741399 0x4d8e 0xd1ac IEEE 802.15.4 12 Data Request
1185 272.792932 0x4d8e 0xd1ac IEEE 802.15.4 12 Data Request

When I try to change a value through HA, I see a lot of route requests, but there is no success to reach the device

1195 275.161548 0xe6ae Broadcast ZigBee 51 Route Request, Dst: 0x4d8e, Src: 0xe6ae
1196 275.177605 0xe6ae Broadcast ZigBee 51 Route Request, Dst: 0x4d8e, Src: 0xe6ae
1197 275.184444 0xe6ae Broadcast ZigBee 51 Route Request, Dst: 0x4d8e, Src: 0xe6ae

I removed the battery and restarted the Danfoss (leaving everything else untouched) and it started to communicate again (just a short snippet:
1928 195.633365 0x4d8e 0xda68 IEEE 802.15.4 12 Data Request
1930 195.652961 0x0000 0x4d8e ZigBee 49 APS: Ack, Dst Endpt: 1, Src Endpt: 1
1932 195.656802 0x4d8e 0xda68 IEEE 802.15.4 12 Data Request
1934 195.669037 0x0000 0x4d8e ZigBee 49 APS: Ack, Dst Endpt: 1, Src Endpt: 1
1936 195.673227 0x4d8e 0xda68 IEEE 802.15.4 12 Data Request
1938 195.684501 0x0000 0x4d8e ZigBee HA 56 ZCL: Default Response, Seq: 14

The biggest obvious difference seems to be the router which is used. It is now device 0xda68, which is an OSRAM Zigbee socket.

My suspect is that the GL-C-009 claims to be a router as it is permanently power with 24V, but does not do the job. Sometimes the TRV switches the router to the GL-C-009 and then it is disconnected as the router does not work. I will verify next time the TRV disconnects.

Question: IS there a way to exclude nodes from routing?

Update:
One additional observation: When looking at the network map in zigbee2mqttassistant, I can see only outgoing arrows from the GLEDOPTO device. No inbound connections as other routers show.

@xeri-sk
Copy link

xeri-sk commented Oct 21, 2022

I tried to add two routers (Sonoff S26R2ZB) to my Zigbee network and now all of my 3 Danfoss TVRs still working for 5 days.
Before i had only endpoint devices connected directly to coordination and Danfoss TVR works for 1 - 3 days.
Now 2 TVRs are connected through routers and 1 still directly to coordinator.
I will continue with monitoring and share results here.

image

@xeri-sk
Copy link

xeri-sk commented Oct 25, 2022

After 7days 2 from 3 TVR not responding. This time i solved by rebooting my linux server.
I like my heating solution with Danfoss Ally but this issue is killing it totally.

Now testing 5m USB extension cable to avoid any interference from WiFi AP.

@lukas-mc
Copy link

I changed the setup for my HA installation. Have VM now running HAOS and debian with ser2net. This setup is running fine so far -- except some TRV from Danfoss/Popp.

Now I have 6 TRV running without problems and exact one I can not communicate to.

I receive messages from the trv:

2022-10-25 17:12:10Received Zigbee message from 'Heizung_Wohnzimmer_Anna', type 'attributeReport', cluster 'hvacThermostat', data '{"localTemp":2187}' from endpoint 1 with groupID 0 Info

The same as soon I changed the ,,occupied temp" at the TRV:

2022-10-25 17:11:43Received Zigbee message from 'Heizung_Wohnzimmer_Anna', type 'attributeReport', cluster 'hvacThermostat', data '{"occupiedHeatingSetpoint":1750,"setpointChangeSource":0}' from endpoint 1 with groupID 0 Info

But when I try to change any settings using the GUI or M2Z this is not published to or received by the trv:

2022-10-25 17:21:39Received MQTT message on 'zigbee2mqtt/Heizung_Wohnzimmer_Anna/set' with data '{"occupied_heating_setpoint":19.5}' Debug 2022-10-25 17:21:39Publishing 'set' 'occupied_heating_setpoint' to 'Heizung_Wohnzimmer_Anna' Debug 2022-10-25 17:21:46Received MQTT message on 'zigbee2mqtt/Heizung_Wohnzimmer_Anna/set/external_measured_room_sensor' with data '2100' Debug 2022-10-25 17:21:46Publishing 'set' 'external_measured_room_sensor' to 'Heizung_Wohnzimmer_Anna'

The value on the TRV is not changed and resetted to the original value within the GUI,

I tried to restart the TRV , HAOS and ser2net also as repairing the TRV nothing helped to ,,awak" the beast.

Regards Lukas

@lukas-mc
Copy link

After 7days 2 from 3 TVR not responding. This time i solved by rebooting my linux server. I like my heating solution with Danfoss Ally but this issue is killing it totally.

Well , which of the TRV's are not respondig? Both connected to the routers?

@DonDonatello
Copy link

DonDonatello commented Oct 25, 2022

I did some fiddling in the Dev console of Z2M and experienced something which might be strange - I'm not sure.

I tried reading some attributes and sending commands to one of my TRVs which currently works. However, after i sent a command that did not work/was not recognized by the TRV, i did not get any error from Z2M. I then tried changing the heating setpoint from Z2M and nothing happened - the TRV did not react and the slider reset itself back to the previous setpoint (just as when the TRV is not functioning as expected).

After a minute or so Z2M suddenly gave multiple error messages from the wrong commands/get requests, and then changed the setpoint to the value I had been trying previously. And now the TRV is working as expected again.

I am not sure if this is expected behaviour or not, however including log if relevant (0x847127fffef2bc71 is the relevant TRV).

I wonder if something like this might be happening "under the hood" without us knowing, i.e. a queue of commands of some sort is created but never sent to the TRV because something else is blocking it? To me, this fits nicely with the fact that Z2M continues to receive state updates from the TRV when the "disconnection" happens.

TRV_log.txt

@xeri-sk
Copy link

xeri-sk commented Oct 25, 2022

After 7days 2 from 3 TVR not responding. This time i solved by rebooting my linux server. I like my heating solution with Danfoss Ally but this issue is killing it totally.

Well , which of the TRV's are not respondig? Both connected to the routers?

Exactly. But i am not sure if routers make problem, before i had Zigbee network without router and same problem with TVR occoured.

@TheLongAndOnly
Copy link

I think I can now confirm the the GLEDOPTO LED drivers cause my problems. Another TRV went out of service and it is again trying to send data request to a GELDOPTO device:
861 174.081297 0x0549 0xe6ae IEEE 802.15.4 12 Data Request
898 181.159780 0x0549 0xe6ae IEEE 802.15.4 12 Data Request
966 188.238124 0x0549 0xe6ae IEEE 802.15.4 12 Data Request
993 195.316823 0x0549 0xe6ae IEEE 802.15.4 12 Data Request

0x0549 is the Popp TRV, 0xe6ae is the GLEDOPTO device.

I either need to find a way to tell the system not to route through the GLEDOPTO device or remove those devices. I read another article that those devices do not work properly as routers.

@lukas-mc
Copy link

What I figured out now fits to yours DonDonatello,

I am not sure if this is expected behaviour or not, however including log if relevant (0x847127fffef2bc71 is the relevant TRV).

I wonder if something like this might be happening "under the hood" without us knowing, i.e. a queue of commands of some sort is created but never sent to the TRV because something else is blocking it? To me, this fits nicely with the fact that Z2M continues to receive state updates from the TRV when the "disconnection" happens.

When I delete the not responsive TRV completely from Z2M, restart the System and add it again, it will become responsive again. But You have to restart the System before pairing the TRV again. Other ways the TRV will remain unresponsive.

This looks like something like a ,,queue" waiting to be send to the TRV, witch ill never be cleaned up on ,,error".

Regards Lukas

@wormiedk
Copy link

I switched to ZHA a month ago and my thermostats have kept responding since. Maybe something is handled differently in ZHA?

@MattWestb
Copy link

First OSRAM plug is made for the "black box of bad Zigbee device" then its not working as routers in the mesh so if having problem with them, taking them out of the network.
GLEDOPTO is using the same Zigbee chip so its using the same Zigbee stack with the same bugs = problems.

If some brave user have one not TI or de(F)CONZ coordinator and can testing im 110% sure its working having those TRV connected to the coordinator.

One coordinator is one "normal" router with little extras like the Trust Center that is manage the security in the mesh network.

Latest finding is that TI coordinators is having problems in the router part and real Zigbee 3 end device is having problem then the router is not setting up "end device timeout request" and is leaving or falling out of the network if using the coordinator as there parent.

Also possible adding some good Zigbee 3 router in the network and trying getting the TRVs using them shall working OK.

@TheLongAndOnly
Copy link

@MattWestb how can I prevent devices from changing the router? My problem is that devices seem to change the route after a while, then picking the GLEDOPTO.

@Koenkk
Copy link
Owner

Koenkk commented Oct 26, 2022

Can someone provide me a sniff starting from the point where controlling the thermostat still works until it fails?

https://www.zigbee2mqtt.io/advanced/zigbee/04_sniff_zigbee_traffic.html#with-cc2531

@TheLongAndOnly
Copy link

I think we have different issue here, I have sniffs from a TRV being unresponsive and from the restart after battery disconnect, but none which is tracking the moment of failure. It takes days to weeks to happen. But I'm pretty sure my issue is caused by the GLEDOPTP devices.

@MattWestb
Copy link

@TheLongAndOnly You cant the its the self healing of one mesh network.
Best practice using good Zigbee 3 router and not "OSRAM Plug" types.

@Koenkk Can you pleas making the TI coordinator replaying to Command Frame: End Device Timeout Request from joining end devices with one End Device Timeout Response, Success ?

If not its not working with most Zigbee 3 end devices.

Its one part of the Zigbee PRO = Zigbee 3 moderate.

@Koenkk
Copy link
Owner

Koenkk commented Oct 27, 2022

@MattWestb

Can you pleas making the TI coordinator replaying to Command Frame: End Device Timeout Request from joining end devices with one End Device Timeout Response, Success ?

With a Zigbee 3 coordinator it should already do this.

@MattWestb
Copy link

MattWestb commented Oct 27, 2022

One sniff make with ZZH last week it was not doing but the device was working OK if forcing it using one working Zigbee 3 router.

Frame 33: 56 bytes on wire (448 bits), 56 bytes captured (448 bits) on interface -, id 0
IEEE 802.15.4 Data, Dst: 0x0000, Src: 0x1672
    Frame Control Field: 0x8861, Frame Type: Data, Acknowledge Request, PAN ID Compression, Destination Addressing Mode: Short/16-bit, Frame Version: IEEE Std 802.15.4-2003, Source Addressing Mode: Short/16-bit
    Sequence Number: 110
    Destination PAN: 0x12af
    Destination: 0x0000
    Source: 0x1672
    [Extended Source: IEEERegi_ff:fe:d1:2d:e6 (3c:6a:2c:ff:fe:d1:2d:e6)]
    [Origin: 31]
    [Ack In: 34]
    FCS: 0xbbf3 (Correct)
ZigBee Network Layer Command, Dst: 0x0000, Src: 0x1672
    Frame Control Field: 0x1a09, Frame Type: Command, Discover Route: Suppress, Security, Destination, Extended Source Command
    Destination: 0x0000
    Source: 0x1672
    Radius: 1
    Sequence Number: 86
    Destination: TexasIns_00:24:c1:69:02 (00:12:4b:00:24:c1:69:02)
    Extended Source: IEEERegi_ff:fe:d1:2d:e6 (3c:6a:2c:ff:fe:d1:2d:e6)
    ZigBee Security Header
        Security Control Field: 0x28, Key Id: Network Key, Extended Nonce
        Frame Counter: 4097
        Extended Source: IEEERegi_ff:fe:d1:2d:e6 (3c:6a:2c:ff:fe:d1:2d:e6)
        Key Sequence Number: 0
        Message Integrity Code: 155825d6
        [Key: KA]
        [Key Origin: 29]
    Command Frame: End Device Timeout Request
        Command Identifier: End Device Timeout Request (0x0b)
        Requested Timeout Enumeration: 256 min (8)
        End Device Configuration: 0x00

And one 154 ack in next frame:

Frame 34: 5 bytes on wire (40 bits), 5 bytes captured (40 bits) on interface -, id 0
IEEE 802.15.4 Ack, Sequence Number: 110
    Frame Control Field: 0x0002, Frame Type: Ack, Destination Addressing Mode: None, Frame Version: IEEE Std 802.15.4-2003, Source Addressing Mode: None
    Sequence Number: 110
    [Destination PAN: 0x12af]
    [Destination: 0x1672]
    [Source: 0x0000]
    [Ack To: 33]
    [Ack Time: 0.002176000 seconds]
    FCS: 0x3fc0 (Correct)

And the end device i puling its parent many times but is not getting one response from the request sequence 110 = the parent router is not Zigbee 3 = ZZH coordinator.

And one wrong formed answer: its not the coordinator its the router part of the coordinator that shall replaying with one end device timeout response.

@BartRuSec
Copy link

@MattWestb in my opinion Koenkk/Z-Stack-firmware#439 (comment) was a reason of strange behavior of my TS0044 device, so far.

@MattWestb
Copy link

@BartRuSec Sounds great !!
tuya is not using all Zigbee 3 end device advanced options like pull control check ins like IKEA is doing in the production versions of there devices but they is using not so old stacks but from different vendors / chips so its no so easy knowing how they have making all OK but its good that you devices is working well with the fixed firmware.

@xdubx
Copy link

xdubx commented May 16, 2023

@MMeinhardt1 i have 4 Bosch Thermostats. Same behaviour after flashing. 3 Are connected to the coordinator. 1 is over a repeater.

@Gerry33
Copy link

Gerry33 commented May 17, 2023

In the release notes of 'Z-Stack_3.x.0 coordinator 20230507' its said, "... but not for older Xiaomi devices as they do not ..."

Is there a criteria what is an 'Old Xiaomi ' device ?
And how can I determine that ?

Sorry, but I was not able to follow the technical details of this thread, but I also do suffer from this problem.

Thanks you all for your work and spending time on this !
Gerry

@Morphy99
Copy link

Morphy99 commented Jun 8, 2023

To add to what @Gerry33 said, will this new firmware adversely affect the functioning of the older Xiaomi devices?

@Koenkk
Copy link
Owner

Koenkk commented Jun 8, 2023

will this new firmware adversely affect the functioning of the older Xiaomi devices?

No, Xiaomi devices will work like before.

@lux73
Copy link

lux73 commented Jun 8, 2023

To add to what @Gerry33 said, will this new firmware adversely affect the functioning of the older Xiaomi devices?

can second this - all of my Xiaomi Devices works as expected -> running 20230507 @ CC2652RB at least for 4 Weeks now

even my 4 Danfoss TRVs works without any Problems

@slugzero
Copy link
Contributor

slugzero commented Jun 8, 2023

@Koenkk happy to see that you made 20230507 the recommended firmware today :-)
This firmware also contains my fixes for child aging, which, according to the feedback in this thread, have solved this issue. So I think it is finally time to close this issue as fixed?

Latest firmware for reference: https://github.com/Koenkk/Z-Stack-firmware/releases/tag/Z-Stack_3.x.0_coordinator_20230507

@Koenkk
Copy link
Owner

Koenkk commented Jun 8, 2023

@slugzero yes, thanks, I will close this 😄

@Koenkk Koenkk closed this as completed Jun 8, 2023
@ipeacocks
Copy link

Updated firmware to latest one on CC2652P but still same issue with Danfoss Ally TRVs.

https://imgur.com/a/WvjKSTa

How I've tested that:

  1. My hass/z2m is installed on Pi4 with UPS
  2. Fully turned off electricity in my apartment
  3. All devices went to coordinator
  4. Danfoss Ally became unresponsive: in network but can't set temperature.

@Gerry33
Copy link

Gerry33 commented Jul 7, 2023

I can confirm ipeacocks finding.

@Wickedy123
Copy link

I'm not into the deep technical details of this stuff, but I'll share my experience and some suggestions:

I have 20 POPP TRVs which all now appear to be working fine (but it's summertime). This includes one right next to the coordinator (Sonoff 3.0 stick) which was constantly disconnected beforehand (and had me waiting on this thread all winter).

It's worth checking your TRVs are on the latest firmware (via OTA feature in z2m).

It might also be worth deleting and repairing - I had a few devices unresponsive after upgrading the Sonoff stick: a couple of Moes light Switches (not dimmers), Aqara vibration sensor DJT11LM and a Salus dry contact relay. They have all been fine now for a few weeks.

However, none of my POPP/Danfoss TRVs needed repairing.

Also things that might be unique in my setup compared to yours:

  • Some time ago, I set each TRV to "defaultSendRequestWhen":"immediate" in the Z2m database, which is discussed somewhere in this thread. I never changed them back. (This is hackery of the database.db file, at your own risk)
  • Each of my TRVs receive a temperature update from an external sensor every five minutes, via a Node Red flow. Perhaps this helps to preserve connections.

oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 16, 2024
If pDev is not NULL, everything in the buffer from linkInfo.InFrmCntr
onwards was also garbage:
The *pBuf+=4 is definitely a bug, it increments the value, not the pointer
address. So linkInfo.InFrmCntr is written to the buffer, then
linkInfo.InFrmCntr in the buffer is incremented by 4, and then the first
two bytes (because the pointer was not incremented) are overwritten with
linkInfo.TxFailure. I replaced it with the correct pBuf +=4.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 16, 2024
If pDev is NULL (i.e. not found in associated devices list), only the first
2 bytes of the buffer (shortAddr) were filled (with INVALID_NODE_ADDR).
All the remaining buffer contained uninitialized memory garbage.
Hence the memset, to zero out the entire buffer beforehand.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 16, 2024
The values from associated_devices_t struct after linkInfo.txFailure were
not written to the buffer at all (without the proper pointer increment, they
would have been written to the wrong spot anyway). So this part of the buffer
also contained uninitialized memory only. I completed the function to copy
the entire struct, because the endDev.deviceTimeout and timeoutCounter were
particularly interesting for me to inspect child aging.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 16, 2024
The values from associated_devices_t struct after linkInfo.txFailure were
not written to the buffer at all (without the proper pointer increment, they
would have been written to the wrong spot anyway). So this part of the buffer
also contained uninitialized memory only. I completed the function to copy
the entire struct, because the endDev.deviceTimeout and timeoutCounter were
particularly interesting for me to inspect child aging.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 17, 2024
If pDev is not NULL, everything in the buffer from linkInfo.InFrmCntr
onwards was also garbage:
The *pBuf+=4 is definitely a bug, it increments the value, not the pointer
address. So linkInfo.InFrmCntr is written to the buffer, then
linkInfo.InFrmCntr in the buffer is incremented by 4, and then the first
two bytes (because the pointer was not incremented) are overwritten with
linkInfo.TxFailure. I replaced it with the correct pBuf +=4.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 17, 2024
If pDev is NULL (i.e. not found in associated devices list), only the first
2 bytes of the buffer (shortAddr) were filled (with INVALID_NODE_ADDR).
All the remaining buffer contained uninitialized memory garbage.
Hence the memset, to zero out the entire buffer beforehand.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 17, 2024
The values from associated_devices_t struct after linkInfo.txFailure were
not written to the buffer at all (without the proper pointer increment, they
would have been written to the wrong spot anyway). So this part of the buffer
also contained uninitialized memory only. I completed the function to copy
the entire struct, because the endDev.deviceTimeout and timeoutCounter were
particularly interesting for me to inspect child aging.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 17, 2024
The values from associated_devices_t struct after linkInfo.txFailure were
not written to the buffer at all (without the proper pointer increment, they
would have been written to the wrong spot anyway). So this part of the buffer
also contained uninitialized memory only. I completed the function to copy
the entire struct, because the endDev.deviceTimeout and timeoutCounter were
particularly interesting for me to inspect child aging.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 26, 2024
If pDev is not NULL, everything in the buffer from linkInfo.InFrmCntr
onwards was also garbage:
The *pBuf+=4 is definitely a bug, it increments the value, not the pointer
address. So linkInfo.InFrmCntr is written to the buffer, then
linkInfo.InFrmCntr in the buffer is incremented by 4, and then the first
two bytes (because the pointer was not incremented) are overwritten with
linkInfo.TxFailure. I replaced it with the correct pBuf +=4.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 26, 2024
If pDev is NULL (i.e. not found in associated devices list), only the first
2 bytes of the buffer (shortAddr) were filled (with INVALID_NODE_ADDR).
All the remaining buffer contained uninitialized memory garbage.
Hence the memset, to zero out the entire buffer beforehand.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 26, 2024
The values from associated_devices_t struct after linkInfo.txFailure were
not written to the buffer at all (without the proper pointer increment, they
would have been written to the wrong spot anyway). So this part of the buffer
also contained uninitialized memory only. I completed the function to copy
the entire struct, because the endDev.deviceTimeout and timeoutCounter were
particularly interesting for me to inspect child aging.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 28, 2024
If pDev is not NULL, everything in the buffer from linkInfo.InFrmCntr
onwards was also garbage:
The *pBuf+=4 is definitely a bug, it increments the value, not the pointer
address. So linkInfo.InFrmCntr is written to the buffer, then
linkInfo.InFrmCntr in the buffer is incremented by 4, and then the first
two bytes (because the pointer was not incremented) are overwritten with
linkInfo.TxFailure. I replaced it with the correct pBuf +=4.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 28, 2024
If pDev is NULL (i.e. not found in associated devices list), only the first
2 bytes of the buffer (shortAddr) were filled (with INVALID_NODE_ADDR).
All the remaining buffer contained uninitialized memory garbage.
Hence the memset, to zero out the entire buffer beforehand.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
oliv3r added a commit to oliv3r/simplelink-lowpower-f2-sdk that referenced this issue Aug 28, 2024
The values from associated_devices_t struct after linkInfo.txFailure were
not written to the buffer at all (without the proper pointer increment, they
would have been written to the wrong spot anyway). So this part of the buffer
also contained uninitialized memory only. I completed the function to copy
the entire struct, because the endDev.deviceTimeout and timeoutCounter were
particularly interesting for me to inspect child aging.

See Koenkk/zigbee2mqtt#13478 (comment)

@slugzero

Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
problem Something isn't working
Projects
None yet