ZNP Adapter Manager/Backup Refactor #303

castorw · 2021-01-31T11:35:14Z

This PR addresses issues discussed in #286. Although quite large - it is a complex refactor of ZNP adapter startup, backup management and commissioning process.

Status:

@puddly - you are linking frame counter with network key in your proposal (zigpy/zigpy#557 (comment)), however from what I see the frame counters are bound to extended PAN IDs in NWK_SEC_MATERIAL table. Or am I doing this wrong?

@Koenkk If you'd like I would be happy to discuss this further, preferably in some more interactive manner - Discord, Teams, ... you name it. Thanks.

…-herdsman into znp-provisioning-refactor

Koenkk · 2021-01-31T15:13:45Z

You can contact me on telegram (@Koenkk)

puddly · 2021-01-31T16:21:12Z

@puddly - you are linking frame counter with network key in your proposal (zigpy/zigpy#557 (comment)), however from what I see the frame counters are bound to extended PAN IDs in NWK_SEC_MATERIAL table. Or am I doing this wrong?

It's stored in different places for each Z-Stack version but I believe the only way to reset the counter is to rotate the network key. You can't rotate the EPID. This also matched the structure of the SiLabs EZSP backup format.

Determine if NWK_TCLK_TABLE backup/restore is necessary

You'll need those for APS encryption, which uses unique keys for each device. They're derived from the "TCLK_SEED", the device's IEEE, and a shift, to save space, so you'll have to actually "export" them before use. I haven't had time to investigate if Z-Stack can re-import them in their expanded form, since each stack handles hashed keys differently.

It you want to discuss the backup stuff further (it's not set in stone but diverging from the unified backup format this soon won't make it very unified :) ) or find anything interesting within Z-Stack, you can contact me over Discord: puddly#8728.

nurikk · 2021-05-26T04:06:07Z

@nurikk would it be ok for you if the data is provided as json to the frontend?

yes, should be fine

Koenkk · 2021-05-27T15:56:09Z

@nurikk thinking more about this, there are 3 things to backup: database.db, coordinator_backup.json and the configuration.yaml. When the user would click the backup button in the frontend, I think it's the most handy that a zip file is downloaded with those 3 files in it. Does it make more sense that the data is returned as a binary string instead?

nurikk · 2021-05-28T00:37:18Z

@nurikk thinking more about this, there are 3 things to backup: database.db, coordinator_backup.json and the configuration.yaml. When the user would click the backup button in the frontend, I think it's the most handy that a zip file is downloaded with those 3 files in it. Does it make more sense that the data is returned as a binary string instead?

we can keep backup file as simple json with key-value pairs, key - path, value - data uri like:

{
    "meta": {
        "date": "2021-05-28T00:36:24.604Z",
        "some_random_stats": "here",
         "z2m_version": "0.15.5"
    },
    "files": {
        "./database.db": "data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==",
        "./configuration.yaml": "data:text/plain;base64,......"
    }
}

Hedda · 2021-05-28T08:12:30Z

@nurikk thinking more about this, there are 3 things to backup: database.db, coordinator_backup.json and the configuration.yaml. When the user would click the backup button in the frontend, I think it's the most handy that a zip file is downloaded with those 3 files in it.

The ability for an end-user to get a such full backup package as zip file from the frontend would be awesome and make it very user-friendly to perform manual backups and restores when for example migrating between different computers/installations/setups.

Koenkk · 2021-05-28T18:28:35Z

@nurikk but how would the user restore it? (since it cannot restore a json file, the files need to be replaced)

nurikk · 2021-05-29T02:42:08Z

@nurikk but how would the user restore it? (since it cannot restore a json file, the files need to be replaced)

I thought about two ways of restoring this backups.

User uploads this backup file from ui and boom, magic happens
User drops this backup file into z2m directory and restart z2m, and then magic happens.

But I agree that simple zip file works nice as well KISS :)

Koenkk · 2021-05-29T13:29:10Z

If I provide the info like #303 (comment) (as json), could you contstruct a zip from the frontend?

nurikk · 2021-05-29T13:31:21Z

If I provide the info like #303 (comment) (as json), could you contstruct a zip from the frontend?

Yes

kirovilya · 2021-05-30T12:04:50Z

A small note / question ... Some iobroker.zigbee users had their own ExtPanID different from the default ("DDDDDDDDDDDDDDDD").
These users now have an error at startup:


Configuration is not consistent with adapter state/backup!
 - PAN ID: configured=5203, adapter=5204
 - Extended PAN ID: configured=341bd2fe158bdcb5, adapter=00124b00192e822f
 - Network Key: configured=0a03050709cb0d0f66020406080a0c0e, adapter=0a03050709cb0d0f66020406080a0c0e
 - Channel List: configured=25, adapter=25
 Please update configuration to prevent further issues.
 If you wish to re-commission your network, please remove coordinator backup at /opt/iobroker/iobroker-data/zigbee_0/nvbackup.json.
 Re-commissioning your network will require re-pairing of all devices!
 Starting zigbee-herdsman problem : "startup failed - configuration-adapter mismatch - see logs above for more information"
 Failed to start Zigbee
 Error herdsman start

It can be seen that the "Extended PAN ID" in the settings and in the adapter do not match.
If they go back to the previous version, then there is no error and everything works.

341bd2fe158bdcb5 - correct extpanid from settings.
00124b00192e822f - extpanid read from chip / adapter, matches ieee chip.

Hence the questions:

why is the extpanid not correct in the adapter?
previously configured networks did not correctly register extpanid in the adapter?
what should such users do now so that they do not have to reconfigure the network again? try to set extpanid DDDDDDDDDDDDDDDD in settings?

kirovilya · 2021-05-30T14:28:30Z

Another case - wrong values in adapter side:

Configuration is not consistent with adapter state/backup!
 - PAN ID: configured=6754, adapter=65535
 - Extended PAN ID: configured=00124b00192e81a5, adapter=0000000000000000
 - Network Key: configured=01030507090b0d0f00020406080a0c0d, adapter=01030507090b0d0f00020406080a0c0d
 - Channel List: configured=11, adapter=
 Please update configuration to prevent further issues.
 If you wish to re-commission your network, please remove coordinator backup at /opt/iobroker/iobroker-data/zigbee_0/nvbackup.json.
 Re-commissioning your network will require re-pairing of all devices!
 Starting zigbee-herdsman problem : "startup failed - configuration-adapter mismatch - see logs above for more information"
 Failed to start Zigbee
 Error herdsman start

Koenkk · 2021-05-30T15:26:08Z

@castorw can you check this?

castorw · 2021-05-31T08:12:01Z

why is the extpanid not correct in the adapter?

EPID configuration wasn't implemented properly before in ZH. This PR fixed that, but it is possible that adapters initialized by pervious version of ZH may not have EPID properly configured. The occurrence of this issue is related to different firmwares in radios.

previously configured networks did not correctly register extpanid in the adapter?

See above.

what should such users do now so that they do not have to reconfigure the network again? try to set extpanid DDDDDDDDDDDDDDDD in settings?

I would suggest they set the EPID to the value reported in adapter. This is the easiest way to fix the issue for now.

castorw · 2021-05-31T08:14:48Z

Another case - wrong values in adapter side:

Configuration is not consistent with adapter state/backup!
 - PAN ID: configured=6754, adapter=65535
 - Extended PAN ID: configured=00124b00192e81a5, adapter=0000000000000000
 - Network Key: configured=01030507090b0d0f00020406080a0c0d, adapter=01030507090b0d0f00020406080a0c0d
 - Channel List: configured=11, adapter=
 Please update configuration to prevent further issues.
 If you wish to re-commission your network, please remove coordinator backup at /opt/iobroker/iobroker-data/zigbee_0/nvbackup.json.
 Re-commissioning your network will require re-pairing of all devices!
 Starting zigbee-herdsman problem : "startup failed - configuration-adapter mismatch - see logs above for more information"
 Failed to start Zigbee
 Error herdsman start

This looks like improperly configured adapter - possibly even in Router/EndDevice mode (wonder if this adapter could have worked before at all). This worked before migrating to updated ZH codebase? Was it run with ZH before?

kirovilya · 2021-05-31T08:27:18Z

EPID configuration wasn't implemented properly before in ZH. This PR fixed that, but it is possible that adapters initialized by pervious version of ZH may not have EPID properly configured. The occurrence of this issue is related to different firmwares in radios.

Of course the adapter and network were initialized in previous versions of ZH (most users will have this case).
It is necessary to somehow give recommendations on what to do in this case, otherwise there will be many questions about this.

This is the easiest way to fix the issue for now.

it seems to me the simplest way is to indicate DDDDDDDDDDDDDDDD in extpanid.

kirovilya · 2021-05-31T08:33:05Z

This looks like improperly configured adapter - possibly even in Router/EndDevice mode (wonder if this adapter could have worked before at all). This worked before migrating to updated ZH codebase? Was it run with ZH before?

this adapter used to work as a coordinator (in the previous version of ZH). this is probably a problem with early firmwares for cc2538, but they worked for users. I'm afraid this problem cannot be solved by re-initializing the network. most likely the settings are saved at different addresses than in other firmware.

maybe add the ability to ignore some checks? otherwise some adapters will not be able to work

castorw · 2021-05-31T09:21:51Z

@kirovilya

Yeah, updating the EPID to 0xDDDDDDDDDDDDDDDD is as good solution as putting the value reported by adapter in the startup.
Regarding the other issue, I am not really sure this can be caused by different firmware version. Only if NIB was relocated to another NV item ID - which I think is unlikely. Can you provide me with details on the firmware used in the CC2538 radio?

kirovilya · 2021-05-31T10:17:11Z

Regarding the other issue, I am not really sure this can be caused by different firmware version. Only if NIB was relocated to another NV item ID - which I think is unlikely. Can you provide me with details on the firmware used in the CC2538 radio?

first version of firmware was https://github.com/antst/CC2538-ZNP-Coordinator-firmware (without sources)
then moved here https://github.com/reverieline/CC2538-CC2592-ZNP with sources and history.
there was some moves of NV_MEM reverieline/CC2538-CC2592-ZNP@6329103#diff-b0b2076efea010195f6a749aa25eddc1c1ef974c3d96ad5a2dc1eae71527d542

castorw · 2021-06-03T07:30:43Z

@kirovilya the diff does not imply NV relocation, it's only NV stretch so larger tables can be included. Anyway its unrelated to the NV item indexes which hasn't changed for a long time. I am positive that NIB location (item this problem is related to) is the same for Z-Stack 1.2 and Z-Stack 3.0.x and Z-Stack 3.x.0+ as well. The reason for the NV being in the state described above is unknown to me and so far the only occurrence of such state.

kirovilya · 2021-06-03T07:34:57Z

I see a lot of problems with the latest update. therefore, I will repeat my proposal to make it possible to disable the check at startup in order to work with the current settings.

castorw · 2021-06-03T07:37:33Z

@kirovilya Could you provide me with NV dump from the adapter which caused this issue (with PANID being 65535)? Possibly using https://github.com/zigpy/zigpy-znp/blob/dev/zigpy_znp/tools/nvram_read.py.

kirovilya · 2021-06-03T07:42:29Z

@castorw I can't do it quickly. this is not my stick, and the user is not an expert. will have to figure out a way for him to execute the code

castorw · 2021-06-03T07:45:03Z

@kirovilya Thank you, this would be essential for further diagnostics. I am trying to think about the ability to disable the checks, but most of the updated code-base loses sense in such way. The goal was to stabilize the way ZH handles network provisioning and not create another cracks allowing improperly configured adapters to run.

@Koenkk any thoughts on this?

Koenkk · 2021-06-03T07:47:41Z

@kirovilya @castorw let's continue in #376

castorw added 6 commits January 30, 2021 18:20

added mz/fs for promise-based fs access

ee98388

wip: znp adapter manager

c985f74

added mz/fs for promise-based fs access

8f60cc9

wip: znp adapter manager

0022f03

Merge branch 'znp-provisioning-refactor' of github.com:castorw/zigbee…

41bc771

…-herdsman into znp-provisioning-refactor

fixed nib coordinator address byte order

97ee02c

castorw added 3 commits January 31, 2021 16:42

added toJSON method to structs

82788d5

fixed unified backup nwkUpdateId and securityLevel handling

c124abf

various enhancements to restore and commissioning

8ef17cd

castorw added 5 commits January 31, 2021 18:27

wip: added aps_use_ext_panid

cd226f5

added epid to state comparisons

86be10f

removed struct inline factory function

f4c9988

unified backup enhancements, struct buffer fixes

f18d78e

added APP_CNF::setNwkFrameCounter definition

198c640

This was referenced Feb 2, 2021

[RFC] Unified radio command line interface and standard network backup format zigpy/zigpy#557

Closed

[RQUEST] Feature for automatic NVRAM Backup before upgrading and restore after JelmerT/cc2538-bsl#109

Open

castorw added 12 commits February 3, 2021 18:53

fixed nv multi-chunk reads and writes

15620d5

wip: more structs and complex backup

aae1644

added request retries to nv driver

744dfe5

wip: working backup/restore with aps security

87f1596

added mz/fs for promise-based fs access

f9da95b

wip: znp adapter manager

2f34326

fixed nib coordinator address byte order

5e54756

added toJSON method to structs

af04d7e

fixed unified backup nwkUpdateId and securityLevel handling

97858ca

various enhancements to restore and commissioning

c0eaad2

wip: added aps_use_ext_panid

d828cd5

added epid to state comparisons

62bd263

DubhAd mentioned this pull request Jun 2, 2021

HomeAssistant addon 1.19.0 Configuration is not consistent Koenkk/zigbee2mqtt#7664

Closed

Hedda mentioned this pull request Jun 2, 2021

Zigbee-Herdsman error handling and error reporting #376

Closed

Koenkk mentioned this pull request Jun 18, 2021

Can only pair devices when close to the coordinator (ZZH) Koenkk/zigbee2mqtt#7762

Closed

castorw mentioned this pull request Jul 26, 2021

Error: All table entries need to be the same length Koenkk/zigbee2mqtt#8135

Closed

Koenkk mentioned this pull request Jul 30, 2021

Z-Stack 1.2 resets when updating from Zigbee2MQTT 1.18.1 to 1.20.0 with custom epid #397

Closed

Hedda mentioned this pull request Feb 21, 2022

[REQUEST] Standard "Open ZigBee Coordinator Backup Format" support for Zigbee network backups and restoring from and to ConBee/RaspBee (deCONZ based adapters)? dresden-elektronik/deconz-rest-plugin-v2#12

Open

Hedda mentioned this pull request Mar 20, 2023

[REQUEST] Standard "Open ZigBee Coordinator Backup Format" support for Zigbee network backups and restoring from and to other ecosystems/platforms? zsmartsystems/com.zsmartsystems.zigbee#1379

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZNP Adapter Manager/Backup Refactor #303

ZNP Adapter Manager/Backup Refactor #303

castorw commented Jan 31, 2021 •

edited

Loading

Koenkk commented Jan 31, 2021

puddly commented Jan 31, 2021

nurikk commented May 26, 2021

Koenkk commented May 27, 2021

nurikk commented May 28, 2021 •

edited

Loading

Hedda commented May 28, 2021 •

edited

Loading

Koenkk commented May 28, 2021

nurikk commented May 29, 2021

Koenkk commented May 29, 2021

nurikk commented May 29, 2021

kirovilya commented May 30, 2021

kirovilya commented May 30, 2021

Koenkk commented May 30, 2021

castorw commented May 31, 2021 •

edited

Loading

castorw commented May 31, 2021

kirovilya commented May 31, 2021

kirovilya commented May 31, 2021

castorw commented May 31, 2021

kirovilya commented May 31, 2021

castorw commented Jun 3, 2021

kirovilya commented Jun 3, 2021

castorw commented Jun 3, 2021

kirovilya commented Jun 3, 2021 •

edited

Loading

castorw commented Jun 3, 2021

Koenkk commented Jun 3, 2021

ZNP Adapter Manager/Backup Refactor #303

ZNP Adapter Manager/Backup Refactor #303

Conversation

castorw commented Jan 31, 2021 • edited Loading

Koenkk commented Jan 31, 2021

puddly commented Jan 31, 2021

nurikk commented May 26, 2021

Koenkk commented May 27, 2021

nurikk commented May 28, 2021 • edited Loading

Hedda commented May 28, 2021 • edited Loading

Koenkk commented May 28, 2021

nurikk commented May 29, 2021

Koenkk commented May 29, 2021

nurikk commented May 29, 2021

kirovilya commented May 30, 2021

kirovilya commented May 30, 2021

Koenkk commented May 30, 2021

castorw commented May 31, 2021 • edited Loading

castorw commented May 31, 2021

kirovilya commented May 31, 2021

kirovilya commented May 31, 2021

castorw commented May 31, 2021

kirovilya commented May 31, 2021

castorw commented Jun 3, 2021

kirovilya commented Jun 3, 2021

castorw commented Jun 3, 2021

kirovilya commented Jun 3, 2021 • edited Loading

castorw commented Jun 3, 2021

Koenkk commented Jun 3, 2021

castorw commented Jan 31, 2021 •

edited

Loading

nurikk commented May 28, 2021 •

edited

Loading

Hedda commented May 28, 2021 •

edited

Loading

castorw commented May 31, 2021 •

edited

Loading

kirovilya commented Jun 3, 2021 •

edited

Loading