ZNP Adapter Commissioning/Restore Decision Process #286
Comments
I'm wondering if this is true. At least for the CC2652 and CC1352 it's not (that's why https://www.zigbee2mqtt.io/information/FAQ.html#how-do-i-migrate-from-a-cc2531-to-a-more-powerful-coordinator-eg-zzh is needed). For the CC2530/CC2531 (1.2) the firmware is hacked always start on the specified pan_id, regardless if any other networks are running on it. Does this only apply to the CC2538 and CC2530/CC2531 on zstack 3.0.x? Also involving @hacker-cb in the discussion since we've discussed this earlier. |
|
Good, I base my CC2538 firmware on @hacker-cb's patches. Cannot tell about CC2652/CC1352 since I haven't had my hands of one of those yet. I guess these are based on SimpleLink and you may know better if they use the same NV structure or not. Just tested with CC2531 running pre-built firmware from https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator/Z-Stack_3.0.x. In the commissioning process the PAN ID is incremented. I have configured PAN ID 6754 and ended up with 6755:
Actually I guess it's true as well - the re-commissioning is the reason you need to increment the PAN ID in config, otherwise it would be done automatically making inconsistencies between config / NV RAM and inside NV RAM between configured PAN_ID and NIB. |
|
My last suggestion was to generate instance_id and save it to the NVMEM during initializing. |
|
The first problem we should tackle it the wrong |
@Koenkk you wrote to me? |
|
@hacker-cb I am going to test it right away. |
|
@Koenkk I have found that function before but just assumed it would crash the radio firmware since I expected the underlying proprietary code to call it in loop until suitable PAN ID was found. But it actually worked, the network has been commissioned with colliding PAN ID and basically hijacked my other network. This might be considered a solution, however it's a pretty rough one - since you can basically create rogue networks with no warning at all. I would still consider finding other solution to mitigate the issue while minimising the possible security impact. However it can still be done if someone really wants to. So the second issue in question: commissioning instead of coordinator restore. |
|
Good, since the pan id will now match the one in the config I think we can go for solution 3 from the OP? |
|
Ok, so I will "rape" my radio firmware to do such in a configurable manner. For the rest of the issue I think we could go with your solution (3). However, wouldn't you think it would be a good idea to crash the application with error if right after commissioning (initialisation) the response from Shall I provide a PR or you want to do this yourself? |
|
Agree about: the response from ZDO:extNwkInfo does not match the configured PAN ID?, but note that is is a breaking change for older firmwares. Firmware here have to be updated first since those are the once linked from z2m docs: https://github.com/jethome-ru/zigbee-firmware/tree/master/ti/coordinator/cc2538_cc2592 |
|
@hacker-cb What do you think about updating the firmware mentioned above to force PAN ID even if conflict is detected? @Koenkk I don't think it's a breaking change since this will only happen while initialising the adapter. Any adapter that already is configured according to |
|
Okay I have managed to make Force PAN ID Commissioning optional in Z-Stack firmware: ZComDef.h (add custom NV item): ZDApp.c (ignore PAN ID conflicts if configured to do so): |
|
@castorw |
I think this is a good solution. |
|
@0x3EC This can be determined from NIB, however I have found out that NIBs may differ in size (so far 110B and 116B). This is probably due to internal includes based on firmware compile-time configuration. Also 8/16/32-bit architecture of radio MCU may play role due to memory alignment. I will be looking deeper into this later today and try to figure out a suitable way to prevent/detect PAN ID collision and coordinator NV restore issues. I also had issue restoring coordinator backup - didn't notice it then - but I have restored 110 byte NIB over firmware which produced 116 byte NIBs. This caused the radio to listen on channel 12 instead of 11 as expected and don't know what else. Hopefully I will be able to identify the differences between these NIB formats. |
|
So it looks like NIBs generated in CC2530/CC2531 are not aligned since they run 8051 (8-bit) MCU. Therefore size of CC2530/CC2531 NIB is 110 bytes. CC2538 however creates 116 byte NIB due to word-alignment (16-bit aligned memory access). Therefore size of That this means: restoring CC2531 backup on CC2538 or other 16/32-bit MCUs shouldn't be done or the NIB should be fixed beforehand. @Koenkk @hacker-cb @0x3EC Would any of you be able to provide Here's a little detail on the matter: EDIT: By the looks of |
|
I've been messing around with the low-level NIB stuff for a while in zigpy-znp so if you want to compare results, here's what I am using right now: https://github.com/zigpy/zigpy-znp/blob/dev/zigpy_znp/znp/nib.py#L50-L123 A few other data type sizes change between the different architectures so it's not enough to tweak padding. I've documented all of the differences between the two structs in the comment of the I also have complete NVRAM dumps for a CC2652R, though the format is a little different than what Z2M uses: https://github.com/zigpy/zigpy-znp/tree/dev/tests/nvram. I would be very interested in some complete CC2538 NVRAM dumps if you have any :) Regarding your main issue with PAN ID conflicts in newer Z-Stacks (especially with existing routers on the network): I have been trying to form a network with a PAN ID of In my tests it has worked and causes my CC2652R to happily form a conflicting network, even with other routers sharing that same PAN ID. Not entirely related to this discussion, but if you want to collaborate on a unified network backup format between Z2M and zigpy/ZHA and possibly get deCONZ on board, it would be awesome: zigpy/zigpy#557 (comment) |
|
@castorw this is from 2652: coordinator_backup.json{
"adapterType": "zStack",
"time": "Tue, 12 Jan 2021 17:11:13 GMT",
"meta": {
"product": 1
},
"data": {
"ZCD_NV_EXTADDR": {
"id": 1,
"offset": 0,
"osal": true,
"product": -1,
"value": [
38,
31,
129,
34,
0,
75,
18,
0
],
"len": 8
},
"ZCD_NV_NIB": {
"id": 33,
"offset": 0,
"osal": true,
"product": -1,
"value": [
25,
5,
2,
51,
20,
51,
0,
30,
0,
0,
0,
1,
5,
1,
143,
0,
7,
0,
2,
5,
30,
0,
0,
0,
18,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
18,
58,
8,
0,
0,
0,
4,
0,
15,
15,
4,
0,
1,
0,
0,
0,
1,
0,
0,
0,
0,
221,
221,
221,
221,
221,
221,
221,
221,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
15,
3,
0,
1,
120,
10,
1,
0,
0,
0,
185,
103,
0,
0
],
"len": 116
},
"ZCD_NV_PANID": {
"id": 131,
"offset": 0,
"osal": true,
"product": -1,
"value": [
18,
58
],
"len": 2
},
"ZCD_NV_EXTENDED_PAN_ID": {
"id": 45,
"offset": 0,
"osal": true,
"product": -1,
"value": [
221,
221,
221,
221,
221,
221,
221,
221
],
"len": 8
},
"ZCD_NV_NWK_ACTIVE_KEY_INFO": {
"id": 58,
"offset": 0,
"osal": true,
"product": -1,
"value": [
0,
167,
13,
78,
236,
154,
129,
229,
186,
153,
253,
107,
247,
192,
237,
0,
154
],
"len": 17
},
"ZCD_NV_NWK_ALTERN_KEY_INFO": {
"id": 59,
"offset": 0,
"osal": true,
"product": -1,
"value": [
0,
167,
13,
78,
236,
154,
129,
229,
186,
153,
253,
107,
247,
192,
237,
0,
154
],
"len": 17
},
"ZCD_NV_APS_USE_EXT_PANID": {
"id": 71,
"offset": 0,
"osal": true,
"product": -1,
"value": [
0,
0,
0,
0,
0,
0,
0,
0
],
"len": 8
},
"ZCD_NV_PRECFGKEY": {
"id": 98,
"offset": 0,
"osal": true,
"product": -1,
"value": [
167,
13,
78,
236,
154,
129,
229,
186,
153,
253,
107,
247,
192,
237,
0,
154
],
"len": 16
},
"ZCD_NV_PRECFGKEY_ENABLE": {
"id": 99,
"offset": 0,
"osal": true,
"product": -1,
"value": [
0
],
"len": 1
},
"ZCD_NV_CHANLIST": {
"id": 132,
"offset": 0,
"osal": true,
"product": -1,
"value": [
0,
0,
4,
0
],
"len": 4
},
"ZCD_NV_EX_TCLK_TABLE": {
"sysid": 1,
"itemid": 4,
"subid": 0,
"product": 1,
"osal": false,
"offset": 0,
"value": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
255,
0,
0,
0
],
"len": 20
},
"ZCD_NV_EX_NWK_SEC_MATERIAL_TABLE": {
"sysid": 1,
"itemid": 7,
"subid": 0,
"product": 1,
"osal": false,
"offset": 0,
"value": [
247,
102,
100,
0,
221,
221,
221,
221,
221,
221,
221,
221
],
"len": 12
}
}
} |
|
About Frame Counter problem @0x3EC wrote some time ago: Koenkk/Z-Stack-firmware#225 |
In NVMEM, memory is allocated for PANID (#define ZCD_NV_PANID 0x0083). It is necessary to add in the firmware so that the current PANID (on which the network started) is recorded in this memory. |
coordinator_backup.json
|
I believe this is only used when the network is being formed, at least on newer devices. Once the network has formed and the NIB's network state attribute has the correct value, the NIB will be the authoritative source after a reset for all of the runtime network state (including the pan id, extended pan id, channel mask, current channel, etc.) |
From what I have tested for several tens of times you are right! That parameter is only used in BDB commissioning network formation process. And I suppose the other parameters as well (Extended PAN ID, Channel List, Keys, ...). All of these can be found in NIB afterwards. |
|
I have updated the original comment with data from CC2652's provided by you all: #286 (comment) @puddly: Here are CC2538 NVRAM dumps for you (created with Regarding the The idea of unified backup format seems nice. And the finding that NIB can be safely altered is great. I will dedicate some time for testing this with CC2531/CC2538 and conversions between NIBs as well. @Koenkk:
Agreed? |
|
One recommendation for Z2M to implanting erasing / resetting of the NVR for TI ZNP so user dont need reflashing it then its being corrupted (oft reported then having strange problems with the mesh and solved then reflashing the ZNP). And if going little further doing the same for the other ZNP that Z2M is supporting for getting it working for all ZNP (radio) platforms. |
|
@castorw agree |
|
@Koenkk I have decided to give it a spin and already am working on solution for the issues described here. When ready I will update this issue and hopefully produce a PR. If I hit a wall or have any issues I will split this one to smaller parts. |
|
The primary goals are:
Here's a rough and ugly flow chart of updated startup process I am trying to achieve. I am already WIP, any thoughts are welcome. |
|
@castorw Following suggestion: "Backup NIB is restored properly according to target platform (memory alignment)" has a low priority here. For the CC2531 1.2 we don't create backups (and I think we should keep it as such, seems to work fine). CC2538 <-> CC2652/CC1352 already works. That would only leave the CC2531 and 3.0 which is a firmware I never recommend (and I expect it is rarely used). |
|
@Koenkk It actually is a case I encountered several times and basically it breaks your network up without any warning. I already have base for this ready - NIB structure + conversion methods. Also I am updating the restore process in a way it will basically not use the backup NIB, rather only parts of it. If you change adapter - NIB from new adapter will be used and only required values will be updated so other parts of target adapter are respected. I have one more question however - is there any way currently I could log a warning from |
This is currently not possible, what do you want to use it for? |
|
@Koenkk To issue warnings regarding inconsistencies between:
|
|
@castorw I'm fine with passing an optional logger for such cases. Change the constructor in |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days |
|
Thanks for removing the stale tag. Currently short on free time, still fighting tests. Hopefully I will be ready in a few days ;-) |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days |
|
#303 ready for review. I guess we can close this upon review and merge. |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days |
Current State
Currently the following startup process is implemented (as per startZnp.ts):
hasConfiguredNV item:startupFromAppstartupFromApp,startupFromAppBDB Commissioning
Part of BDB commissioning which forms a new ZigBee network is to make sure, no other network with the same PAN ID lives on the same channel in the vicinity of the adapter. This is done by Z-Stack adapter firmware by sending a
Beacon Requestwhen forming a new network. Adapter then waits forBeaconframes and determines if the configured PAN ID is in use by other ZigBee network nearby.If a network with the same PAN ID is present, the commissioning process will keep incrementing the PAN ID by one until suitable (free) PAN ID is found. If the configured PAN ID is NOT found it is used. This information is then stored within NIB in adapters NV memory.
This means that contents of ZCD_NV_PANID and ZCD_NV_NIB may differ after network commissioning. This information is discovered by Z2M by
ZDO:extNwkInfoSREQ/SRSP but not taken into account.Problem
During an adapter upgrade or replacement, where the new adapter has been previously used by Z2M without having NVRAM cleared (eg. used in other network or used before with other parameters), Z2M would start BDB commissioning instead of restoring NV backup to recover the previous network. This is due to the decision process implementation -
hasConfiguredparameter is set from previous Z2M instance but parameters mismatch -> BDB Commissioning. The expected action, however, would be coordinator restore from backup.Proposed Solutions
ZDO:extNwkInfoSRSP mismatches ZCD_NV_PANID (therefore configured PAN ID) update the PAN ID in NIB and write it back to device - then SREQSYS:resetReq,hasConfiguredNV item in adapter introduce a value, which uniquely represents every Z2M instance (eg. UUIDv4) - if the instance ID mismatches (use in different Z2M instance) - coordinator NV is rather restored than network re-commissioned.This might be 2 issues as well (the decision process and commissioned PAN ID validation). I am posting this issue to open a discussion in this matter since it has caused me several problems and by searching on forums and other issues I see other people struggle with this as well (sometimes without a gist where the problem lies).
If we can find a suitable solution, I am willing to do the implementation and submit a PR.
The text was updated successfully, but these errors were encountered: