Some Ubisys devices moved to ota.zigbeeOTA and no longer pick up updates #329

sjorge · 2023-06-25T13:09:06Z

Seems a while ago some of the Ubisys devices moved to the ota.zigbeeOTA provider... I don't understand why.

We have a zigbee ota.ubisys that is working fine, this weeks updates are not showing up for all devices using ota.zigbeeOTA but are showing up for those with ota.ubisys.

e.g.

Once the update is install correctly I will switch them all back to ota.ubisys locally and report back if they all pick up the updates.

Once swapepd to ota.ubisys

sjorge · 2023-06-25T13:14:48Z

device	provider	shows update with ota.ubisys	installs update with ota.ubisys
S1	ota.ubisys	yes	yes
S1-R	ota.zigbeeOTA	do not own	do not own
S2	ota.zigbeeOTA	yes	yes
S2-R	ota.zigbeeOTA	do not own	do not own
D1	ota.zigbeeOTA	do not own	do not own
D1-R	ota.zigbeeOTA	do not own	do not own
J1	ota.zigbeeOTA	yes	yes
J1-R	ota.zigbeeOTA	do not own	do not own
C4	ota.zigbeeOTA	yes	yes
H1	ota.ubisys	yes	yes
R0	ota.zigbeeOTA	do not own	do not own

I'll update this table ones done testing

Koenkk · 2023-06-25T13:17:40Z

See #317 for the reason

sjorge · 2023-06-25T13:25:43Z

See #317 for the reason

That doesn't make much sense to me? If the ones fetched via ota.ubisys are the same ones as the ones manually available for download, there should be no difference (at the time the firmware as added, there is obviously a newer one available now)

We should at the very least I think, switch them all over to ota.zigbeeOTA then? I'll still round of my tests with switching them back to ota.ubisys though. I wonder if they user just got unlucky and he checked while ubisys was in the process of publishing them firmwares.

sjorge · 2023-06-25T13:35:34Z

Something that also comes to mind, some devices need a 2 stage update as ubisys swapped firmware formats at some point, if you check the index used by ota.ubysis the older format one is listed as ota1.zigbee, AFAIK the still works when using ota.ubisys but it's probably broken with ota.zigbeeOTA as we cannot add multiple firmwares for a single devices AFAIK.

e.g.

10F2-7B01-0000-0006-01110206-spo-fmd.ota.zigbee 256748 5018ea0bf877e458ce78163e45f425dda7db16fb2190d3464dcb2fa496b0bdbc
10F2-7B01-0000-0006-0192020D-spo-fmd.ota1.zigbee 263324 facc5ac8cdd533b00559df80ea441d8bc8b1170504b4bac818307026b4783c16

IIRC the special logic to deal with that is here:
https://github.com/Koenkk/zigbee-herdsman-converters/blob/143bc6ff7500b3f85c35dd3c5a18f4d17b078de1/src/lib/ota/ubisys.ts#L28-L33

sjorge · 2023-06-25T13:41:54Z

@Koenkk

Some more references as to why we also resole older images:
Koenkk/zigbee-herdsman-converters#4466
Koenkk/zigbee-herdsman-converters#4491

I think the regex to pick the proper image is probably not 100% correct and somehow @WhistleMaster hit case where it returns the wrong image.

I do think we should figure out why it was not working for them and revert to ota.ubisys, as the current ota.zigbeeOTA approach will result in an incomplete upgrade path.

sjorge · 2023-06-25T14:59:32Z

Updates one of each devices with ota.ubisys and they all still seem to work.

Koenkk · 2023-06-25T17:38:20Z

Reverted it!

sjorge · 2023-06-25T17:41:07Z

@WhistleMaster do you still happen to have one of those devices that you haven't updated yet?
If so would you mind adding some extra console.log() to see why the match is failing?

WhistleMaster · 2023-06-25T17:54:29Z

Sure ! Where should I add those extra log ?

sjorge · 2023-06-25T18:08:03Z

https://github.com/Koenkk/zigbee-herdsman-converters/blob/3a30a5f5d1bad50ab985cffee653feeeeb97c168/src/lib/ota/ubisys.ts#L28

Above that line try adding:

logger.debug(`FWMATCH - Device: ${JSON.stringify(device)}`);
logger.debug(`FWMATCH - ImageMatch 1-16: ${parseInt(imageMatch[1], 16)}, ImageMatch 2-16: ${parseInt(imageMatch[2], 16)}, ImageMatch 3-16: ${parseInt(imageMatch[3], 16))}, ImageMatch 4-16: ${parseInt(highestMatch[4], 16)}, ImageMatch hwVer: ${hardwareVersion}`);

It should snow up in the z2m log if you have debug level enabled, also make sure to switch to ota.ubisys while adding this.

There is also a chance it will just work TM now, given Ubisys pushed a new fw earlier this week.

WhistleMaster · 2023-06-26T12:39:55Z

I've added the debug output. I had to remove one ")" for the second line, otherwise I had an error. I get the following when checking for updates:

FWMATCH - Device: {"_events":{},"_eventsCount":0,"ID":3,"_type":"Router","_ieeeAddr":"0x001fee00000056ff","_networkAddress":4510,"_manufacturerID":4338,"_endpoints":[{"_events":{},"_eventsCount":0,"ID":1,"profileID":260,"deviceID":266,"inputClusters":[0,3,4,5,6],"outputClusters":[],"deviceNetworkAddress":4510,"deviceIeeeAddress":"0x001fee00000056ff","clusters":{"genBasic":{"attributes":{"modelId":"S2 (5502)","manufacturerName":"ubisys","powerSource":1,"zclVersion":8,"appVersion":1,"stackVersion":1,"hwVersion":5,"dateCode":"20191127-DE-FB1","swBuildId":"2.0.0"}},"genOnOff":{"attributes":{"onOff":0,"startUpOnOff":255}}},"_binds":[{"cluster":6,"type":"endpoint","deviceIeeeAddress":"0x00124b00257ca56f","endpointID":1}],"_configuredReportings":[{"cluster":6,"attrId":0,"minRepIntval":0,"maxRepIntval":300,"repChange":0}],"meta":{},"pendingRequests":[]},{"_events":{},"_eventsCount":0,"ID":2,"profileID":260,"deviceID":266,"inputClusters":[0,3,4,5,6],"outputClusters":[],"deviceNetworkAddress":4510,"deviceIeeeAddress":"0x001fee00000056ff","clusters":{"genOnOff":{"attributes":{"onOff":0,"startUpOnOff":255}}},"_binds":[{"cluster":6,"type":"endpoint","deviceIeeeAddress":"0x00124b00257ca56f","endpointID":1}],"_configuredReportings":[{"cluster":6,"attrId":0,"minRepIntval":0,"maxRepIntval":300,"repChange":0}],"meta":{},"pendingRequests":[]},{"_events":{},"_eventsCount":0,"ID":3,"profileID":260,"deviceID":1,"inputClusters":[0,3],"outputClusters":[5,6,8],"deviceNetworkAddress":4510,"deviceIeeeAddress":"0x001fee00000056ff","clusters":{},"_binds":[{"cluster":6,"type":"endpoint","deviceIeeeAddress":"0x001fee00000056ff","endpointID":1}],"_configuredReportings":[],"meta":{},"pendingRequests":[]},{"_events":{},"_eventsCount":0,"ID":4,"profileID":260,"deviceID":1,"inputClusters":[0,3],"outputClusters":[5,6,8],"deviceNetworkAddress":4510,"deviceIeeeAddress":"0x001fee00000056ff","clusters":{},"_binds":[{"cluster":6,"type":"endpoint","deviceIeeeAddress":"0x001fee00000056ff","endpointID":2}],"_configuredReportings":[],"meta":{},"pendingRequests":[]},{"_events":{},"_eventsCount":0,"ID":5,"profileID":260,"deviceID":1281,"inputClusters":[0,1794,2820],"outputClusters":[],"deviceNetworkAddress":4510,"deviceIeeeAddress":"0x001fee00000056ff","clusters":{"seMetering":{"attributes":{"multiplier":1,"divisor":1000,"instantaneousDemand":0,"currentSummDelivered":[0,97]}}},"_binds":[{"cluster":1794,"type":"endpoint","deviceIeeeAddress":"0x00124b00257ca56f","endpointID":1}],"_configuredReportings":[{"cluster":1794,"attrId":1024,"minRepIntval":5,"maxRepIntval":3600,"repChange":1}],"meta":{},"pendingRequests":[]},{"_events":{},"_eventsCount":0,"ID":232,"profileID":260,"deviceID":1287,"inputClusters":[0,64512],"outputClusters":[3,25],"deviceNetworkAddress":4510,"deviceIeeeAddress":"0x001fee00000056ff","clusters":{},"_binds":[],"_configuredReportings":[],"meta":{},"pendingRequests":[]},{"_events":{},"_eventsCount":0,"ID":242,"profileID":41440,"deviceID":102,"inputClusters":[33],"outputClusters":[33],"deviceNetworkAddress":4510,"deviceIeeeAddress":"0x001fee00000056ff","clusters":{},"_binds":[],"_configuredReportings":[],"meta":{},"pendingRequests":[]}],"_manufacturerName":"ubisys","_powerSource":"Mains (single phase)","_modelID":"S2 (5502)","_applicationVersion":1,"_stackVersion":1,"_zclVersion":8,"_hardwareVersion":5,"_dateCode":"20191127-DE-FB1","_softwareBuildID":"2.0.0","_interviewCompleted":true,"_interviewing":false,"_skipDefaultResponse":false,"_skipTimeResponse":false,"meta":{"configured":3},"_lastSeen":1687782309297,"_defaultSendRequestWhen":"immediate","_linkquality":36}

Failed to check if update available for 'switch_1' (Cannot read properties of null (reading '4'))

So I just added the following line instead:

logger.debug(`FWMATCH - ${imageMatch}`);

And I get:

OTA ubisys: image found: 10F2-7B2B-0000-0001-01920210-m7b-h10.ota.zigbee
FWMATCH - 10F2-7B31-0000-0006-02110404-spo-fmd.ota.zigbee,7B31,0000,0006,02110404

EDIT: Forgot to add the hardwareVersion output. Here it is:

FWMATCH - 5

sjorge · 2023-06-26T14:45:11Z

OK that's interesting,

You are currently on 2.0.0 which should be image type 0x7b33 (Going to try and find a way to verify this).

I did some digging and the major version changes were:

1.9.1, image type goes to 0x7b23
1.9.2, image type goes to 0x7b33

Assuming the device does indeed report 0x7b33 and a hwVer of 5 it should match one of these I think ({manufacturerCode}-{imageType}-{hardwareVersionMin}-{hardwareVersionMax}-{fileVersion}-{?}-{?})

$ curl -s http://fwu.ubisys.de/smarthome/OTA/release/index | grep -i '7b33'
10F2-7B33-0000-0006-02120404-spo-fms2.ota.zigbee 144894 bdd07ed61f50767f72ab0825cae9c374a854957cd514c3c3646586d702978fca
10F2-7B33-0000-0006-02210412-spo-fms2.ota.zigbee 146942 2f99f07d83c71b703b5781fae617c6b7d5262dbe30ca7fae345024690a401a70

Can you replace L24 -> L35 with this instead:

    imageRegex.lastIndex = 0; // reset (global) regex for next exec to match from the beginning again
    let imageMatch = imageRegex.exec(firmwarePageHtml);
    let highestMatch = null;
    logger.debug(`OTA Ubisys: image type = ${current.imageType}`);
    while (imageMatch != null) {
        logger.debug(`OTA ubisys: image found: ${imageMatch[0]}`);
        if (parseInt(imageMatch[1], 16) === imageType &&
            parseInt(imageMatch[2], 16) <= hardwareVersion && hardwareVersion <= parseInt(imageMatch[3], 16)) {
            logger.debug(`OTA ubisys: potential match = ${imageMatch[0]}`);
            if (highestMatch === null || parseInt(highestMatch[4], 16) < parseInt(imageMatch[4], 16)) {
                logger.debug(`OTA ubisys: new highest match = ${imageMatch[0]}`);
                highestMatch = imageMatch;
            }
        }
        imageMatch = imageRegex.exec(firmwarePageHtml);
    }

And can you then past the debug log entry from when it's looping over the images, i would expect at least 2 loops to happen.

WhistleMaster · 2023-06-26T15:27:57Z

I've made the changes, and I get the following in the log: _zigbee2mqtt_logs.txt

sjorge · 2023-06-26T16:10:24Z

Ok this is weird, your current.imageType returns 0x7B23 and that one only matches

10F2-7B23-0000-0006-0192020D-spo-fms2.ota.zigbee 129022 8cf2e80213f743980d778d10d4899d200862065a2cd29173b4105b51344bfd77

So it seems you somehow have an older bootloader (based on the change logs) with a newer firmware image.

The one you updated with the manually downloaded firmware already, what does that device report for imageType after the update?

WhistleMaster · 2023-06-26T16:18:20Z

Well, I did not had the time to update with the manually downloaded firmware, so all my devices are like that as the PR was reverted.

Is there another way to update them then ?

sjorge · 2023-06-26T16:40:51Z

I see, I was looking at the commit and it basically has the same imageType with the same file. So I'd be a bit surprised if that one matched while via ota.ubisys it didn't.

I guess we could test the following:

offer the newest firmware ignoring the imageType match (seems like a bad idea)
offer the older firmware matching the imageType and see if it wants to downgrade, then it should 🤞 update properly after.

@Koenkk I'm guessing if the image type doesn't match the device would reject it correct? That would rule out option 1.

WhistleMaster · 2023-06-26T17:04:40Z

And by looking at the Firmware build date, all of them are 20191127-DE-FB1 but with Firmware version 2.0.0

sjorge · 2023-06-26T17:11:32Z

build_date never changes on any of mine when updating, I think it's like batch number when they were made.

I have 2x S2, both went through the 3 stage update process, one has date 20191128-DE-FB1 and one has 20200327-DE-FB1.

The weird thing is that yours seems to be reporting an older imageType but already on 2.0.0 fw

You can track the fw/image type changes via the changelog https://www.ubisys.de/en/support/firmware/changelog-s2/

0x7B23 and 2.0.0 never appears anywhere :/

WhistleMaster · 2023-06-26T17:14:17Z

Indeed... that's really weird. Before, all my S2 were used with Deconz and updated with Deconz.

I hope I can find a way to update them with Z2M. If there were a way to "upload" a file directly to it, that could maybe help.

sjorge · 2023-06-26T17:20:02Z

Indeed... that's really weird. Before, all my S2 were used with Deconz and updated with Deconz.

Interesting, I wonder if deconz somehow skipped the bootloader bit of an OTA image, although I'm not even sure that's technically possible (@Koenkk ?)

I hope I can find a way to update them with Z2M. If there were a way to "upload" a file directly to it, that could maybe help.

I don't think we can force a particular firmware, also a question for Koen. Even if we somehow do force send it a file, I think the device would just reject it.

Koenkk · 2023-06-26T17:30:49Z

I don't think we can force a particular firmware, also a question for Koen. Even if we somehow do force send it a file, I think the device would just reject it.

This is possible with the local OTA index: https://www.zigbee2mqtt.io/guide/usage/ota_updates.html

sjorge · 2023-06-26T17:38:20Z

I guess @WhistleMaster can try and force downgrade using http://fwu.ubisys.de/smarthome/OTA/release/10F2-7B23-0000-0006-0192020D-spo-fms2.ota.zigbee which is the only one of the correct image type. Once that it done, the regular OTA should work again. Assuming the device accepts that one.

If that works, would there be a way to have ota.ubisys somehow also have a force option? I think something probably went wrong when they were updated via deconz

WhistleMaster · 2023-06-26T18:08:37Z

Thanks for the help. I've made a local index like that and add it to the Z2M configuration.

[
    {
        "fileVersion": 26345997,
        "fileSize": 129022,
        "manufacturerCode": 4338,
        "imageType": 31523,
        "sha512": "6b59596d69c0049ac8111ddec0edee16c7fe3353900e9d04eef7a3d42a7cda10152883e743b8114b2140957fc417fa64c62b15fc236a4d8d86a10cb786742cf2",
        "url": "http://fwu.ubisys.de/smarthome/OTA/release/10F2-7B23-0000-0006-0192020D-spo-fms2.ota.zigbee",
        "force": true
    }
]

and changing ota: ota.zigbeeOTA for the S2 type in src/devices/ubisys.ts

Restarting Z2M and checking for firmware, I get the information that an update is available. I click on "Update firmware device" and after a bit I get:

Update of 'switch_1' failed (Timeout: device did not request any image blocks)

sjorge · 2023-06-26T18:12:29Z

You still need to compile the .ts to a .js file... I'm not 100% sure on the steps but:

checkout zigbee-herdsman-converter somewhere
npm run build in that dir
change to your zigbee2mqtt folder
`npm link /path/to/zigbee-herdsman-conveter

@Koenkk did I miss a step ?

WhistleMaster · 2023-06-26T18:17:35Z

It looks like that it gets my local index as it says that a firmware is available, isn't it ? But I think that the device does not accept the firmware...

I've tried to change the local index to put the latest version available and I get:

Failed to check if update available for 'switch_1' (No image available for imageType '31523')

So I manually changed the imageType to 31523 instead of 31539 but I get:

Update of 'switch_1' failed (Image type mismatch)

sjorge · 2023-06-26T18:23:35Z

Not sure it picked up the change, as http://fwu.ubisys.de/smarthome/OTA/release/10F2-7B23-0000-0006-0192020D-spo-fms2.ota.zigbee is imageType 31523 / 0x7B23

WhistleMaster · 2023-06-26T18:27:54Z

If I simply revert back to the original ubisys.ts, I get my old error that the firmware is newer and can't be updated even with my local index.

WhistleMaster · 2023-06-27T15:03:37Z

I've written an email to Ubisys support to have some insights on the issue.

By the way, how do you read the imageType on the Z2M web interface ? I've tried on the Dev Console, with Endpoint 232, Cluster OTA and Attribute imageTypeId but I get an error: Error: Cannot read properties of undefined (reading '0')

sjorge · 2023-06-27T16:24:23Z

I'm not sure how to read it, that why I let you add the logging ealier.

Koenkk · 2023-06-27T18:35:08Z

@sjorge those are the correct steps indeed!

WhistleMaster · 2023-06-27T19:33:27Z

I’m using the Docker version, I guess that I can’t really follow the same steps, do I ?

What I don’t understand is why it seems that it takes my local index when modifying the ubisys.ts file through Docker console on my Z2M container ? I can “play” with the local index and it changes the error I get when I try to update.

WhistleMaster · 2023-06-28T13:22:48Z

@sjorge, @Koenkk,

I got a reply from Ubisys:

The component already should have switched to image type 0x7b33 with the 1.9.2 firmware version. 
In fact without changing the OTA file it is not possible to update to 2.0.0 without having the correct image type.

I'm a bit lost as I'm not sure what should be done now. It seems not possible to have a different image type if I have the 2.0.0 firmware. I don't know what to do now and try to update the devices.

What about ignoring the imageType and force the latest firmware ?

It would be great to have a possibility to send a new firmware from the dashboard UI, by selecting it with an input file and a checkbox option to "force".

What are your thoughts on that one ?

sjorge · 2023-06-28T13:57:51Z

So they basically confirm that the combination of requested imageType and firmware version 2.0.0 should be impossible.

If the device is willing to downgrade you could try and install the 1.9.2 firmware and see if it wants to upgrade later on.
But I think you tried that in #329 (comment)

I'm not really sure how to proceed.

WhistleMaster · 2023-06-28T16:15:37Z

So they basically confirm that the combination of requested imageType and firmware version 2.0.0 should be impossible.

Additional answer from Ubisys:

as already mentioned, the image type should have already been changed to 0x7b33 with the update to 1.9.2, i.e. somehow your devices managed to update twice without having the correct image type. This has never been observed and should not be technically possible.

If the device is willing to downgrade you could try and install the 1.9.2 firmware and see if it wants to upgrade later on. But I think you tried that in #329 (comment)

Yes indeed, this is what I tried first.

The problem is somewhere else but I don't know where 😅

github-actions · 2023-07-29T00:37:08Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

sjorge mentioned this issue Jun 25, 2023

After configuration of a Ubisys Device i get a (Status 'INSUFFICIENT_SPACE') Koenkk/zigbee2mqtt#15875

Closed

Koenkk mentioned this issue Jun 25, 2023

Add Ubisys OTA firmwares #317

Merged

github-actions bot added the Stale label Jul 29, 2023

github-actions bot closed this as completed Aug 5, 2023

Some Ubisys devices moved to ota.zigbeeOTA and no longer pick up updates #329

Some Ubisys devices moved to ota.zigbeeOTA and no longer pick up updates #329

Comments

sjorge commented Jun 25, 2023 • edited Loading

sjorge commented Jun 25, 2023 • edited Loading

Koenkk commented Jun 25, 2023

sjorge commented Jun 25, 2023

sjorge commented Jun 25, 2023 • edited Loading

sjorge commented Jun 25, 2023

sjorge commented Jun 25, 2023

Koenkk commented Jun 25, 2023

sjorge commented Jun 25, 2023

WhistleMaster commented Jun 25, 2023

sjorge commented Jun 25, 2023 • edited Loading

WhistleMaster commented Jun 26, 2023 • edited Loading

sjorge commented Jun 26, 2023

WhistleMaster commented Jun 26, 2023

sjorge commented Jun 26, 2023

WhistleMaster commented Jun 26, 2023

sjorge commented Jun 26, 2023

WhistleMaster commented Jun 26, 2023

sjorge commented Jun 26, 2023

WhistleMaster commented Jun 26, 2023

sjorge commented Jun 26, 2023 • edited Loading

Koenkk commented Jun 26, 2023

sjorge commented Jun 26, 2023

WhistleMaster commented Jun 26, 2023

sjorge commented Jun 26, 2023

WhistleMaster commented Jun 26, 2023

sjorge commented Jun 26, 2023

WhistleMaster commented Jun 26, 2023

WhistleMaster commented Jun 27, 2023

sjorge commented Jun 27, 2023

Koenkk commented Jun 27, 2023

WhistleMaster commented Jun 27, 2023 • edited Loading

WhistleMaster commented Jun 28, 2023

sjorge commented Jun 28, 2023

WhistleMaster commented Jun 28, 2023

github-actions bot commented Jul 29, 2023

sjorge commented Jun 25, 2023 •

edited

Loading

sjorge commented Jun 25, 2023 •

edited

Loading

sjorge commented Jun 25, 2023 •

edited

Loading

sjorge commented Jun 25, 2023 •

edited

Loading

WhistleMaster commented Jun 26, 2023 •

edited

Loading

sjorge commented Jun 26, 2023 •

edited

Loading

WhistleMaster commented Jun 27, 2023 •

edited

Loading