Testing ember v7+v8 support in dev branch/edge or 1.39.1+ release

[Nerivec]

Edit: This is now available in 1.39.1 release and later.

Dev branch (edge add-on) now supports both EmberZNet v7 and v8 (zigbee-herdsman 0.51+). It introduces a lot of changes (which have been applied to v7 as well).
We need some feedback from dev testers! 😉

Please do not report known issues, unless they behave differently, or have been fixed with the new dev branch.

Make sure to specify which version you are testing the dev branch with (v7 or v8).

For v8, you can try one of these firmware. Please note these are preliminary, and remain vastly untested on most adapters at the moment. And also, flashers that don't support v8 yet, will likely not allow you to revert back to v7 after flashing v8 unless you enter the bootloader manually. You can try https://github.com/Nerivec/ember-zli which uses the latest zigbee-herdsman under the hood (please report any issue with this new tool in the repository directly, not here).

Things you will undoubtedly notice:

• [v7-v8] Driver-level error/status codes have changed (slightly or completely depending on the place) due to a complete revamp of the EmberZNet firmware.
• [v7-v8] Startup sequence no longer assigns some firmware values that were causing troubles due to changes in recent EmberZNet versions. As a result, the startup sequence will look quite a bit shorter in the logs.
• [v7-v8] Some logging has been removed/altered to remove sensitive information from log files.
• [v8] Support for Silabs chips series 0 and 1 has been removed in new firmware. If you have an older adapter that uses one of these two series, you will not see a v8 firmware for it, and will have to remain to v7, or get a newer model adapter.
  • Note: MG24-based adapters are still not widely available, but from all appearances, they do offer a nice upgrade from previous models, so you may want to look into these if you find yourself in that situation.

Known issues with v8 firmware:

• [Fixed in 8.0.2] LQI values are broken in devices list/page (ok in network map). The firmware appears to not be reporting the values properly. - Issue reported to Silabs.
• [Appears fixed in 8.0.1] Stability issues in networks under stress (networks with lots of spamming, poorly implemented devices, etc.)

Silabs v8 release notes: https://www.silabs.com/documents/public/release-notes/emberznet-release-notes-8.0.2.0.pdf

[JuanDYB]

MG21 chips are not recommended? I have seen the SLZB-06M which uses the EFR32MG21 chip model. There is a model from SMLight with EFR32MG24 ( SLZB-06Mg24) but at least in spain I have not seen in any place.

[Nerivec]

Unfortunately MG24 adapters are still rare indeed. As previously stated, they will offer a nice specs upgrade (2-4 times!), for anyone looking to upgrade in the near future. Until then, MG21 remains the most widely available option.
https://www.zigbee2mqtt.io/guide/adapters/emberznet.html#recommended

[JuanDYB]

So, in your opinnion maybe it's better to wait for mg24? I don't know if mg21 will be well supported when mg24 are more extended in the market ..
With regards,

[Hedda]

There is no need to wait as a later migration from EFR32MG21 to EFR32MG24 or EFR32MG26 should not be any problems in theory.

The firmware is the same and EFR32MG21 can handle up to around 200 Zigbee 3.0 devices, so really need EFR32MG24 or EFR32MG26 once you start adding more Zigbee 3.0 devices than 200, (you can add as many older Zigbee devices as you like regardless as they do not have the same security overhead and thus do not require the additional RAM and CPU resources that are provided in the EFR32MG24 MCU).

Newer EFR32MG24 or EFR32MG26 might technically or at least on paper provide other radio benefits and can be a little less sensitive to noise, but you should never rely on the Zigbee Coordinator alone and instead make sure to add loads of Zigbee Router devices regardless.

[Hedda]

For developer reference, EZSP protocol version is incremented to EZSP v14 in EmberZNet 8.0.0.0 firmware.

EmberZNet 8.0.0.0 brings support for EFR32MG26 (and MGM260) also known as xG26 and MG26 as well.

The firmware is not also built from the new Simplicity SDK, so no longer built from the "old" Gecko SDK.

• More information and links here -> Silabs Zigbee EmberZNet 8.0.0.0 (and Silicon Labs Gecko SDK 5.0.0.0) announced and to be released in Q3 2024 zigbee-herdsman#1093

[chris-1243]

@Nerivec

I finally decided to flash your latest firmware on a Dongle-E for testing (v8.0.1.0).

I paired easily an Ikea led and I had a try with a PTM216Z laying around (which brings me crazy as well for NFC config....).

Here is what I tried with my ZGP device:

• Direct pairing to the coordinator as ZGP7 (seen as PTM216Z): working;
• Direct pairing to the coordinator as ZGP2 (I did modify the config via NFC and it is seen as a PTM215ZE): working, at least like in zstack regarding the commands sent;
• Direct pairing to the Ikea bulb as ZGP7 (seen as PTM216Z): not working. This was expected as this light is not able to translate ZGP7 correctly like in zstack;
• Direct pairing to the Ikea bulb as ZGP2 (seen as PTM215ZE): working.

I hope it will help you. I had to revert to 7.4.3.0 as the software I used on Windows to sniff traffic is not updated yet for EZSP v14. The flasher from @darkxst https://github.com/darkxst/silabs-firmware-builder made a great job in updating and downgrading without any issues.

A log file of my different tries in debug mode log.txt

[Nerivec]

Thanks for the feedback. Looks like GP is on the right track!

PS: Check out https://github.com/Nerivec/ember-zli/wiki/Sniff if you need a 8.0.x-compatible sniffer (it uses zigbee-herdsman under the hood, same as Z2M).

[chris-1243]

PS: Check out https://github.com/Nerivec/ember-zli/wiki/Sniff if you need a 8.0.x-compatible sniffer (it uses zigbee-herdsman under the hood, same as Z2M).

I had a quick check and your tool looks great. May I use it on a windows 11 based computer ? (Edit) Sorry, I was too lazy to search the web and the Zigbee2mqtt documentation

Your job is just incredible...thanks

Definitely, it is too late to read correctly

https://github.com/Nerivec/ember-zli/wiki

[Nerivec]

Tarballs offer portability, and "no-setup" for one-offs, but if you are planning to use it regularly, install nodejs (tested with win 11 + node v20), it's easier to update packages with npm, and ember-zli will be in the PATH, so, can execute from anywhere with ease 😉.

[Nerivec]

@Hellou1337 @lawrencedudley @0BanZai0
I built some 8.0.1 firmware with higher config values (untested), if you want to test them, pick "max" version here.

Make sure to read the first post here, as there are still some caveats with 8.0.1 (like broken LQI).
If you encounter any issue with these firmware, please report them in the repo, not here. Same for feedback 😉

[Hellou1337]

Anyway,
tried to flash the 8.0.1.0 max firmware and the flasher ends up with error.
Can flash 8.0.0.0, but not any 8.0.1.0 or 8.0.1.0

Logs

s6-rc: info: service banner successfully started
s6-rc: info: service universal-silabs-flasher: starting
[10:28:56] INFO: Starting universal-silabs-flasher with /dev/ttyACM0 (bootloader baudrate 115200)
Traceback (most recent call last):
  File "/usr/local/bin/universal-silabs-flasher", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/universal_silabs_flasher/flash.py", line 40, in inner
    return asyncio.run(f(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/universal_silabs_flasher/flash.py", line 297, in flash
    metadata = fw_image.get_nabucasa_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/universal_silabs_flasher/firmware.py", line 178, in get_nabucasa_metadata
    return NabuCasaMetadata.from_json(json.loads(metadata))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/universal_silabs_flasher/firmware.py", line 111, in from_json
    fw_type = FirmwareImageType(fw_type)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/enum.py", line 714, in __call__
    return cls.__new__(cls, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/enum.py", line 1137, in __new__
    raise ve_exc
ValueError: 'ncp-uart-sw' is not a valid FirmwareImageType

[Nerivec]

Try using ember-zli instead (link in first post).

[Hellou1337]

So i tried the max firmware with rebuilding the z2m groups as it was.
And - i have zero problems and errors since then.

[mickdekkers]

@Nerivec I'm having the same issue @Hellou1337 described -- [ZCL GROUP] Failed to send with status=BUSY -- in my case this is sometimes causing groups of Tradfri lights not to turn on when my motion detection Home Assistant automation tells them to. I'm using the Sonoff ZBdongle-E with darkxst's NCP 7.4.4.0 firmware. I only have 8 groups in total, with the lights in question only being part of 1 group. (There are 31 devices on my network in total.) Do you know if there's anything I can do to prevent this issue from happening, or maybe work around it? Perhaps there's some kind of setting that would retry sending these messages if they fail?

Further info: I just upgraded yesterday from a ConBee II to the Sonoff ZBdongle-E and from Deconz to Z2M. I've already optimized my Zigbee and wifi channels to minimize interference (with Zigbee being on the quietest channel). My Zigbee dongle is also ~2 meters away from any other devices by way of a shielded USB extension cable. I'm running Z2M through the official Home Assistant addon, latest version 1.40.1-1.

Would appreciate any guidance! Thanks 😄

[Nerivec]

When you get the status BUSY, it means all mechanisms to prevent overloading the network have been exhausted (queues are full, as many messages as allowed are currently traveling the network, etc.).
The ZigBee specification restricts "currently traveling" broadcast messages to 8 over a 9-second window. The stack has a queueing mechanism on top, to allow some leeway in that regard.

Check your logs to see if you can identify a trigger for it. Usually it's dimmers (or similar devices that spam a lot of tiny changes in a short amount of time), they were not meant to be used with broadcasts. But it can also be too many automations triggering at or around the same time.
Considering your relatively low amount of devices, I'd say you have a trigger with unintended side-effects that may be spamming your network without you realizing.

You can also try firmware here: https://github.com/Nerivec/silabs-firmware-builder/releases 8.0.2 no longer has the LQI bug, and these firmware have higher configs to allow more queueing and such. Dongle-E firmware file

[chris-1243]

It might not be the right place to post.

I am wondering how ember is managing the IEEE address while restoring a backup. Especially while changing the coordinator.

I asked there

darkxst/silabs-firmware-builder#120 (comment)

and according to this thread, the IEEE address is stored as a token in NV memory. This means, it would be not needed to write it while flashing a firmware.

Second question arises... What about zstack or other driver ? This would remove the pain of re-pairing everything and users would have the possibility to move from one driver to another depending on their needs...

[Nerivec]

z2m ember currently is not messing with the adapter's IEEE (only backup/restore network info). If I'm not mistaken, the ability to manipulate tokens depends on the firmware being compiled with the tokens component, which may or may not be the case depending on the options set by the builder. It would result in varying behavior depending on adapter+firmware. This would add a bit too much randomness to the process I think (especially since all this happens in the background, without any menu or anything, and only logs for feedback). I'll look into it to confirm and see what could be done in a more reliable fashion.

ember-zli should be able to do it via tokens menus though (can backup/restore the entire set of zigbee tokens in nvm3), assuming your firmware supports tokens manipulation.

[Xploder]

Yes I've tried restoring just the network via ember-zli from an ZBDongleE to SLZB-06M and it didn't work. After also restoring NVM3 tokens with the same tool I was back up immediately - very smooth migration!

[chris-1243]

@Nerivec

In order to get a better support of ZGP frames/devices, would you accept to receive one or more PTM device directly from Enocean according this thread ?

[Se7ven]

JFI, rebuilt official multiprotocol plugin with latest SimplicitySDK v2024.6.2 and latest CPCd. Ember protocol works in multiprotocol solution.
https://github.com/Se7ven/silabs-multiprotocol-ember

But I'm not sure about thread since I don't have any thread devices. Although OpenThread Router looks good.

I need some enthusiasts for testing and feedback 😉

[Nerivec]

For anyone interested, the latest release build includes an updated (8.0.2) router firmware for the Sonoff ZBDongle-E.
https://github.com/Nerivec/silabs-firmware-builder/releases/tag/v2024.6.2-update1

Please read the release notes as it describes the adjusted behavior of that new firmware, to try to fix the previous shortcomings.

Feedback welcome (in that repo)!
Others to come...

[tore1980]

I have updated ZBdonle to 8.0.2 [zbdonglee_zigbee_ncp_8.0.2.0_sw_flow_115200.gbl] from 7.4.3, now I can`t see devises or cordinator in Z2M but it does work from HA. Has anyone experienced the same?

Z2M Config:
ort: >-
/dev/serial/by-id/usb-ITEAD_SONOFF_Zigbee_3.0_USB_Dongle_Plus_V2_20220813141617-if00
adapter: ember
baudrate: 115200
rtscts: false

[Nerivec]

It looks like your frontend is not connected to Z2M anymore? Did you check MQTT?

[tore1980]

Yes, MQTT is OK

[Nerivec]

Can you open an issue, with some logs? I don't think that's anything to do with the firmware update, more the Z2M reboot, something looks to have gone wrong with the connection between frontend and MQTT/Z2M. You can tag me.

[tore1980]

You are correct its not related to fw, no errors in log.
The problem was z2m config.yaml, when I replased it with my backup the frontend is back.
Strange that it happened after FW upgrade.