[uom] Add unit metadata for NumberItem

[J-N-K]

This is a new approach to solving #3282. It keeps the dimension when the item is created but it separates the state description from the internal unit.

With this PR:

• The unit for the item is either the system default (provided by the UnitProvider, depends on dimension and system of measurement) or the unit configured in the new defaultUnit metadata (takes precedence if present)
• The state description is ONLY used for displaying the values (before it was also used to determine the item's unit)
• States are converted to the item unit before the state is set, for plain Number items units are stripped, DecimalType updates to dimension items are considered to be in the configured unit, incompatible units are rejected.
• Commands are handled the same as states, the unit of QuantityType to a plain item is stripped, the item unit is added to DecimalType if there is a dimension, incompatible units are rejected. In contrast to state updates, the unit for commands is NOT converted.
• The CommunicationManager strips the unit and sends DecimalType if the channel has the accepted item-type Number, it adds the item-unit if the dimension of the item and the channel are the same. Commands with incompatible dimension are rejected. "Guessing" a unit based on the channel type when the item is dimensionless has been removed as it is confusing.
• A new UoM profile is introduced. If a unit is configured for that profile, it is added to a DecimalType value, and a QuantityType value is converted to that unit, if no unit is configured, the unit is stripped from a QuantityType value.

To fully solve #3282 we also need to merge #3141 and check where ItemStateEvent was used before.

[spacemanspiff2007]

• The unit for the item is either the system default

Do I understand this correctly that this will keep e.g. Number:Temperature

• or the unit configured in the new defaultUnit metadata (takes precedence if present)

Wasn't it unit before? defaultUnit sounds like a fall back yet you write this takes precedence and overrides all other settings.
I think unit would be a much better wording (if I understand this correctly).

[J-N-K]

Yes, Number:Temperature is kept. It's less invasive and we can still workaround the limitations we found before. One benefit is that the UI now can propose a unit, because it already knows the dimension.

Regarding the naming: it was always defaultUnit in this branch (which is the direct successor of #3248). It can be easily changed to unit if that is the result of the discussion.

One other thing that could be implemented (independently) is changing the system default unit (which is currently hard-coded, depending on dimension and system of measurement).

[spacemanspiff2007]

Yes, Number:Temperature is kept. It's less invasive and we can still workaround the limitations we found before. One benefit is that the UI now can propose a unit, because it already knows the dimension.

So if I select Number:Temperature the UI can suggest unit="°C instead of selecting unit="°C" directly?
I get that this makes migration easier but it obfuscates the whole UoM concept again and provides no real benefit.
The item unit can't and shouldn't be changed automatically (after initial creation) so what do we use Number:Temperature for?
The current dimension can also be derived from °C so when changing the locale the displayed value can still be changed accordingly.

It can be easily changed to unit if that is the result of the discussion.

Depends on how it works.
If it's the fallback that is used when nothing other is there then defaultUnit is good.
If it overrides everything else and takes precedence it should not have default in the name since my understanding of default is more of a fallback in case nothing else is specified.

[J-N-K]

Regarding defaultUnit/unit it probably depends on the POV. It is the unit that is assumed for values that don't carry a unit (it's therefore the "default"), but it is also the unit that is used to store the value internally.

[lolodomo]

What would be the syntax in items file?

[J-N-K]

@lolodomo Like any other metadata. For using K instead of °C:

Number:Temperature Freezer_Temp "Freezer [%.1f %unit%]" { channel="foo:bar:....", defaultUnit="K" }

[J-N-K]

@spacemanspiff2007

If nothing is configured, the system default (in my case °C) is used. By knowing the dimension, the UI can "validate" anything I configure as unit (e.g. it could reject kg for a Number:Temperature and prevent setting a unit that will later result in issues.

[spacemanspiff2007]

It is the unit that is assumed for values that don't carry a unit (it's therefore the "default"), but it is also the unit that is used to store the value internally.

But if it carries a unit (e,g, °F) it will still be normalized to °C so default does not apply there.
Default would mean the new unit would be °F yet the item will still carry the unit °C.
The field will configure the item unit so it should be named accordingly unit.

If nothing is configured, the system default (in my case °C) is used. By knowing the dimension, the UI can "validate" anything I configure as unit (e.g. it could reject kg for a Number:Temperature and prevent setting a unit that will later result in issues.

Imho you're mixing two things - how the GUI presents the setting and how it actually is configured.
I think it would be nice to have some kind of tree where you move down (e.g. first you select "temperature" which reduces the available values to °F, °C and °K. Then the user selects °F which will internally set unit="°F".

However there is no need for a Number:Temperature item because the unit can be inferred from the item metadata.
This will introduce two places where the user has to configure the same thing: first the item type and then the item metadata.
There should be one place to define this and one place only. There should be at maximum two number item types: one with a uom set and one without a unit and even those two types can be hidden from the user.

Also if I as a user don't explicitly set the unit it might lead to the impression that the unit will change when I e.g. change my system locale which is something we explicitly don't want to do. We neither want to change the normalization value automatically for existing items nor do we want to create the impression that openHAB will behave that way.

[J-N-K]

As I said earlier: if it's unit or defaultUnit is a minor issue and can be changed in seconds. The more important discussion is the general UoM handling.

If the user creates the item from a thing channel with "Add link to item -> Create new item" then the item type is automatically set from the accepted item type of the channel. Nothing that can go wrong there. If the user doesn't want anything special and the system default unit is fine, then nothing else needs to be configured. If he wants to use a different unit, he can configure the unit metadata to use another unit. In this case UI can make suggestions (a list of common units for this dimension) and verify the compatibility of the selected unit.

It's debatable if the unit metadata should be automatically added when the item is created and initialized with the system default unit. I would be open for that, this would also solve the "change system of measurement". But this is IMO a rare case and we'll probably have a 50/50 chance that someone wants to change the item units in that case or not.

[spacemanspiff2007]

I think I understand what you are describing.

from the accepted item type of the channel.

But why doesn't the channel type say Number and provide an additional optional field "unit" which would say "Temperature".
It can be easily included e.g. in the CannelTypeDTO, That way the UI still can provide the functionality you described but there is only once place where the unit is configured for the item and that is the item metadata.

But this is IMO a rare case and we'll probably have a 50/50 chance that someone wants to change the item units in that case or not.

I think explicit is better than implicit and in this case I'd stick to that. It takes a very short time to configure it but it's such an important field that the user should be aware of it. So from the current perspective I'd vote against that.

[J-N-K]

Strictly typed item-types (liked it is now) have some advantages. The channel defines Number:Temperature and exactly that item is created. The user can then change from °C to K or °F but not to kg, because kg is not compatible with the dimension Temperature.

This also applies to textual configuration, if you define an incompatible unit, the item will still work and provide values. Breaking that requires deliberately changing the item-type, not just a typo in the unit (e.g. kW vs. kWh). If you determine the dimension from the unit the dimension will change from Power to Energy and future state-updates fail.

I don't think there is a benefit of using Number with additional information Temperature and unit °C over Number:Temperature and unit °C. It's just another way of defining the same and it is not backward compatible.

[spacemanspiff2007]

If you determine the dimension from the unit the dimension will change from Power to Energy and future state-updates fail.

But would that fail quickly - e.g. if I change the unit from kW to kWh it would be the next item update that would generate an error and the user would see that it was a mistake. Since the normalization unit is something that normally is set up once I don't see the need for that kind of error check.

Also if I am not super fluent in English I'll quickly get confused with Number:Energy and Number:Power (or more exotic ones) but I'll most definitely know that I'll want kWh. The use case for most users will be to think of the unit, then look up the corresponding item type in the docs and blindly use that making that kind of error check obsolete.

I don't think there is a benefit of using Number with additional information Temperature and unit °C over Number:Temperature and unit °C. It's just another way of defining the same and it is not backward compatible.

I think this is a misunderstanding. My suggestion is that the item only defines the unit and scale and the channel defines the item type (e.g. String/Number) and optionally an UoM (in case of Number).
E.g. Number and Temperature is defined for the Channel, °C is defined for the item.
That way there is no redundant information and all the error checks are still possible.

There should be one way to define the uom and one way only. It just makes no sense to say: "This item should have the temperature in °C and yes - I'm really sure it's a temperature". You're also not saying "Today it'll be 15°C and what I just told you was a temperature". Everyone know that already from the °C.
Also you're not saving me from typos in the item name and state either. 😉

[lolodomo]

Just one remark as I see that some of you want to remove the number dimension: we cannot break the setup of 100% of users by making the item syntax not backward compatible...

[J-N-K]

For managed things #3268 would solve the issue. This will also be needed for the approach taken here. Textual configuration will in any case require manual action, but that is exactly what those users want.

[spacemanspiff2007]

Just one remark as I see that some of you want to remove the number dimension: we cannot break the setup of 100% of users by making the item syntax not backward compatible...

We had a huge breaking changes from OH1 to OH2 and from OH2 to OH3 and people figured it out, too.
For a major version number increase people will expect breaking changes and I think it's okay to have those.
Moving forward without change is impossible.
There will be tools to help with the migration, e.g. the CLI upgrade tool @J-N-K mentioned and I offered to write a python script that will help with the migration of the *.items files.
So I think the migration path should be relatively straight forward and more or less painless.
If it's properly documented I see no issue.

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/openhab-4-0-milestone-discussion/145133/158

[spacemanspiff2007]

@J-N-K what are your current thoughts on dropping e.g.
Number:Temperature MyItem {unit="°C"}
in favor of
Number MyItem {unit="°C"}

If you're not sold can you share your doubts so I can better understand them.
I'll try to improve my arguments accordingly.

[J-N-K]

We have discussed that before (see the attached issue) and there have been concerns regarding clarity what the item actually stores. That is why I decided to go on with this PR.

[lolodomo]

I see a big advantage to @J-N-K 's solution, it is (almost fully) backward compatible. @J-N-K : please correct me if I am wrong. Even if the user does not update their items definition in config files, it will continue to work as before. In case the user was not using the default system unit for an item, the only impacts would be the values displayed in charts. No change in UIs. Not sure if rules could be impacted ?

[J-N-K]

@lolodomo The breaking change is that currently we use the state description for determining the unit and if that is not set the system default unit. With this PR the state description is not considered, but the defaultUnit (or unit) metadata and if that is not set the system default unit. That is one of the main points here: decouple internal representation of the value and display value.

So if someone currently uses %.2f K for a temperature item, the item's unit is considered to be K. In the future it is °C (or °F, depending on the locale) if he does not set defaultUnit="K". I have a PR (#3268) that fixes these issues for managed items (by setting the defaultUnit when a unit is found in the state description and prints a warning during upgrade if the unit cannot be determined), but there is not much we can do for textual configuration, these user will need to do it by hand - but they chose to do so.

I don't see any effect on UIs or rules, bindings might be affected depending on my last question in #3282.

[lolodomo]

So if someone currently uses %.2f K for a temperature item, the item's unit is considered to be K. In the future it is °C (or °F, depending on the locale) if he does not set defaultUnit="K".

Yes, I clearly understand that point but I have the feeling this change will be almost fully transparent for the user ? The values displayed in UIs will be the same as before. As you say his rules will not be impacted, the only impact I can guess for the user is that now in charts the item values will be in °C rather than in Kelvin before.

bindings might be affected depending on my last question in #3282.

Not sure what question you are referring. In case we have to update all the bindings, this might be a critical problem !

[J-N-K]

IMO bindings that can be linked to Number items with values that MIGHT have a unit (i.e. all measured values, not integers that represent different states) should support DecimalType and QuantityType, but I'm aware of the fact that this is not the case.

That is why I proposed a profile that strips units if necessary (effectively converting QuantityType to DecimalType. If a binding absolutely needs DecimalType, the user can apply that profile and everything works without code changes.

[spacemanspiff2007]

Yes, I clearly understand that point but I have the feeling this change will be almost fully transparent for the user ? The values displayed in UIs will be the same as before. As you say his rules will not be impacted, the only impact I can guess for the user is that now in charts the item values will be in °C rather than in Kelvin before.

No - the change will not be transparent.
The unit defines the internal representation that will be used when interacting with external systems (e.g. Rest / Persistence). So even if you use Number:Temperature on the first start all items will migrate to Number:Temperature with metadata unit="°C" for European locale.
Number:Temperature actually doesn't matter (from a user perspective) - it's the unit="°C" that is important. This will be set once and then never changed again (automatically). The user has to manually intervene to change it.
While for temperature this is more or less trivial (except for those running some scientific stuff which use °K) for length, pressure, speed and intensity it's not so easy any more and there is a high chance that the automatically picked unit is wrong and will have to be manually corrected.

To specify what values you want to display you have to either set a unit in the state description (e.g "Temperature [%.1f °C]") or if nothing is selected the unit from the metadata field will be used ("Temperature [%.1f]" will fall back to what is defined in the metadata unit=

That's why I am so against keeping Number:Temperature:
It obfuscates what's really important and hides functionality from the user without any real benefit.

[J-N-K]

No, the state description is no longer considered, that is the whole point of the refactoring and what the upgrade tool fixes for managed items.

[lolodomo]

But for UI, the state description is used ???

I can see that 1 (instead of 100) is sued when requesting the icon:

How can fix my item definition to have the proper behaviour ?

[lolodomo]

Should I add unit="%" ?

[lolodomo]

That would mean that item state passed to icon request is the internal state. That is probable.

[maniac103]

As for the Android app, sliders currently don't follow the unit metadata, but TBH, I am not sure what the correct algorithm for value formatting is supposed to be.
Consider the setup from here: I have set TestNumberDim1 to say 60 mm. The result of the item endpoint then is this:

{"link":"http://127.0.0.1:8080/rest/items/TestNumberDim1","state":"60 mm","stateDescription":{"pattern":"%.1f cm","readOnly":false,"options":[]},"editable":false,"type":"Number:Length","name":"TestNumberDim1","label":"Test number dim 1","tags":[],"groupNames":[]}

This already is interesting. The item state has unit mm, but state description says %.1f cm for formatting. When formatting intermediate values in sliders, am I supposed to not format values at all then?

Even more interesting is the sitemap response. Relevant excerpt:

                    {
                        "widgetId": "0200",
                        "type": "Slider",
                        "visibility": true,
                        "label": "Test number dim 1 [60,000000 mm]",
                        "icon": "number",
                        "mappings": [

                        ],
                        "switchSupport": false,
                        "sendFrequency": 0,
                        "minValue": 0,
                        "maxValue": 100,
                        "step": 5,
                        "state": "60",
                        "item": {
                            "link": "http://127.0.0.1:8080/rest/items/TestNumberDim1",
                            "state": "60 mm",
                            "stateDescription": {
                                "pattern": "%.1f cm",
                                "readOnly": false,
                                "options": [

                                ]
                            },
                            "type": "Number:Length",
                            "name": "TestNumberDim1",
                            "label": "Test number dim 1",
                            "tags": [

                            ],
                            "groupNames": [

                            ]
                        },
                        "widgets": [

                        ]
                    },

So the item state matches the item endpoint (as expected), but the widget state is "60". From there it's - AFAICT - not possible to discern the unit the min, max, state and step values relate to, or am I missing something?

(Consequently, the Android app interprets everything as cm, because that's what it can get from the state description)

As much as I like the UoM handling in general, I feel we (as in: app people) need some more pieces of information mapped into the sitemap response here:

• unit for widget state
• probably equally or even more important: widget format string (here: %f mm)

With that, the widget format string could override the state description format string, which would lead to the correct behavior for the test case, I believe.

[J-N-K]

What version did you use for that test?

[maniac103]

Snapshot 3389. I'll update to a newer one and test again.

[J-N-K]

That's very old (the last snapshot is 3479) and I believe way before this PR was merged, can you please try with a recent snapshot?

[maniac103]

Just did just that, but AFAICT, the sitemap response is the same as with 3389:

                    {
                        "widgetId": "0200",
                        "type": "Slider",
                        "visibility": true,
                        "label": "Test number dim 1 [60,000000 mm]",
                        "icon": "number",
                        "mappings": [

                        ],
                        "switchSupport": false,
                        "sendFrequency": 0,
                        "minValue": 0,
                        "maxValue": 100,
                        "step": 5,
                        "state": "60",
                        "item": {
                            "link": "http://127.0.0.1:8080/rest/items/TestNumberDim1",
                            "state": "6 cm",
                            "stateDescription": {
                                "pattern": "%.1f cm",
                                "readOnly": false,
                                "options": [

                                ]
                            },
                            "type": "Number:Length",
                            "name": "TestNumberDim1",
                            "label": "Test number dim 1",
                            "tags": [

                            ],
                            "groupNames": [

                            ]
                        },
                        "widgets": [

                        ]
                    },

... so the points from above (particularly the missing widget format string) still apply.

[J-N-K]

IMO the response is fully valid, the state is 6 cm because there is no unit metadata and the default for length is cm. The sitemap instructs openHAB to use mm

Text item=TestNumberDim1 label="Test number dim 1 [%f mm]"

(and that overrides the state description, which is also correct). Also the label is formatted accordingly, as well as the state. So I guess the only thing that is missing is the unit.

@lolodomo WDYT?

[maniac103]

Well, the state is 6 cm, yes (as visible via item -> state), but

• How does the client know maxValue=100 means mm, not cm?
• How does the client know state=60 means mm, not cm?
• How is the client know the widget format string (%f mm)?

As I said, it's all there, it just needs to be put in the sitemap widget response (and SSE event, in case of the state) ;-)
IMO, it would be ideal to add an (optional) unit field (which would contain mm here) and another (optional) formatPattern field (which would contain %f mm here).

[lolodomo]

Default unit for length is meters, not centimeters.

[J-N-K]

@lolodomo Correct. That's interesting.

The item endpoint reports:

{
  "link": "http://localhost:8080/rest/items/LL",
  "state": "0.05 m",
  "stateDescription": {
    "pattern": "%.1f cm",
    "readOnly": false,
    "options": []
  },
  "metadata": {
    "stateDescription": {
      "value": " ",
      "config": {
        "pattern": "%.1f cm"
      }
    }
  },
  "editable": true,
  "type": "Number:Length",
  "name": "LL",
  "label": "Length",
  "category": "",
  "tags": [],
  "groupNames": []
}

Which looks fine. I added a sitemap:

sitemap default label="My home automation" {
    Frame label="Demo" {
       
		Text item=LL label="LL [%f mm]"
	}
}

and the REST API returns

{
  "name": "default",
  "label": "My home automation",
  "link": "http://localhost:8080/rest/sitemaps/default",
  "homepage": {
    "id": "default",
    "title": "My home automation",
    "link": "http://localhost:8080/rest/sitemaps/default/default",
    "leaf": false,
    "timeout": false,
    "widgets": [
      {
        "widgetId": "00",
        "type": "Frame",
        "visibility": true,
        "label": "Demo",
        "icon": "frame",
        "mappings": [],
        "widgets": [
          {
            "widgetId": "0000",
            "type": "Text",
            "visibility": true,
            "label": "LL [50,000000 mm]",
            "icon": "",
            "mappings": [],
            "state": "50 mm",
            "item": {
              "link": "http://localhost:8080/rest/items/LL",
              "state": "0.05 m",
              "stateDescription": {
                "pattern": "%.1f cm",
                "readOnly": false,
                "options": []
              },
              "type": "Number:Length",
              "name": "LL",
              "label": "Length",
              "category": "",
              "tags": [],
              "groupNames": []
            },
            "widgets": []
          }
        ]
      }
    ]
  }
}

which also looks reasonable to me. The state is correctly set to 50 mm and the item-part still returns 0.05 m. I am a bit puzzled how the result above could be produced by recent code.

[J-N-K]

This should be discussed in a new issue.

[maniac103]

I am a bit puzzled how the result above could be produced by recent code.

'Result above' as in: mine? I'll retry with a new install (as opposed to unpacking the zip over the old install) tomorrow.
Regarding the metadata part of your sitemap response: I assume the stateDescription part automatically derives from that if no other state description is given?

This should be discussed in a new issue.

Will open a new one for that.

[spacemanspiff2007]

• How does the client know maxValue=100 means mm, not cm?

Unitless values should fall back to the defined unit so maxValue should actually be meters in this case.
Imho maxValue="100 mm" should be used

[lolodomo]

Regarding min/max/step values, there is am open issue to add unit. Currently, the default unit for these attributes is the display unit if one is defined.
For the state field in the sitemap event, it is the value in the display unit. For slider widget, the unit is stripped IIRC
The label field contains the value formatted according to state description.

[lolodomo]

I have not read everything in ultra details but I see nothing wrong in the events.

I agree that it could be better to explain the units for min/max(step or even to be able to provide an unit.

[maniac103]

Imho maxValue="100 mm" should be used

maxValue is a number in the JSON response though. Adding it there also has the disadvantage of providing redundancy/ambiguity (think minValue=1cm, maxValue=45mm ... parsing something like this is impossible without reimplementing UoM in the client)

The label field contains the value formatted according to state description.

Yes, but that doesn't help for formatting min/max etc.
Ideally widget state, min, max and step all should use the same unit.

We can continue that discussion in #3624.

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/blockly-quantity-type-gets-lost-when-using-persistence/150470/6

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/modbus-thermostat-stepper-dont-work/149836/7

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/modbus-thermostat-stepper-dont-work/149836/20