Charts is doing strange interpolation if period M or longer is choosen

[der-dag]

Which UI are you reporting an issue for?

• Basic UI
• HABPanel
• HABot
• CometVisu

The problem

Just upgraded my openhab from 4.0.4 to 4.3.2 and now some charts are looking strange. I’m using Chart in BasicUI sitemap:

E.g. I have a chart showing the daily gas consumption of the last month. Values are from rrd4j. Every night the value is updated with the consumption of the previous day.

Chart item=Consum_Gas24h period=M refresh=300000

The chart for one month (period=M) now looks like this, somehow it is interpolating:

Expected behavior

If I switch the chart period to show the values of the past two weeks only chart looks as it was in the older openhab versions and in my eyes it is correctly displayed:

Steps to reproduce

Your environment

runtimeInfo:
  version: 4.3.2
  buildString: Release Build
locale: de-DE
systemInfo:
  configFolder: /opt/openhab4/current/conf
  userdataFolder: /opt/openhab4/current/userdata
  logFolder: /opt/openhab4/current/userdata/logs
  javaVersion: 17.0.13
  javaVendor: Debian
  osName: Linux
  osVersion: 6.1.0-29-amd64
  osArchitecture: amd64
  availableProcessors: 4
  freeMemory: 539401568
  totalMemory: 872415232
  uptime: 242139
  startLevel: 100
addons:
  - automation-jsscripting
  - binding-mqtt
  - persistence-rrd4j
  - transformation-jsonpath
  - transformation-map
  - transformation-regex
  - transformation-scale
  - ui-basic
  - ui-habpanel
clientInfo:
  device:
    ios: false
    android: false
    androidChrome: false
    desktop: true
    iphone: false
    ipod: false
    ipad: false
    edge: false
    ie: false
    firefox: true
    macos: false
    windows: true
    cordova: false
    phonegap: false
    electron: false
    nwjs: false
    webView: false
    webview: false
    standalone: false
    os: windows
    pixelRatio: 1
    prefersColorScheme: light
  isSecureContext: true
  locationbarVisible: true
  menubarVisible: true
  navigator:
    cookieEnabled: true
    deviceMemory: N/A
    hardwareConcurrency: 8
    language: de
    languages:
      - de
      - en-US
      - en
    onLine: true
    platform: Win32
  screen:
    width: 2560
    height: 1440
    colorDepth: 24
  support:
    touch: false
    pointerEvents: true
    observer: true
    passiveListener: true
    gestures: false
    intersectionObserver: true
  themeOptions:
    dark: light
    filled: true
    pageTransitionAnimation: default
    bars: light
    homeNavbar: default
    homeBackground: default
    expandableCardAnimation: default
    blocklyRenderer: null
  userAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101
    Firefox/134.0
timestamp: 2025-01-18T08:39:17.523Z

[lolodomo]

Charts for sitemap UIs are generated by the core framework. So it should not be specific to Basic UI.
As you were on an old version, it is possible that the library version we rely on was updated.

[lolodomo]

The last change of version for XChart (3.1.0) was in OH 3.2. So no recent changes.

[der-dag]

I see that problem is already in OH 4.1.0 and you are right problem seems to be in core framework. I guess problem came up when new sitemap chart period parameter were introduced: openhab/openhab-core@f71ebfb
Before that all the periods durations where calculated using Duration.ofDays like entry("M", Duration.ofDays(30)) now the periods are parsed somehow. I see also in the latest version 4.3.2 if the periods are given in days the chart is as expected:

If I generated the chart using period 'M'
http://localhost:8080/chart?items=Consum_Gas24h&period=M

it's bad:

If I use period '30D'
http://localhost:8080/chart?items=Consum_Gas24h&period=30D

it's good:

But if I used '31D' it's bad again.

@lolodomo Can this issue be moved to core repo or do I have to recreate there?

[lolodomo]

@lolodomo Can this issue be moved to core repo or do I have to recreate there?

I have no rights to do that.
@kaikreuzer or @wborn : please move this issue to core repo

[wborn]

I have no rights to do that.

Me neither as I'm no webui maintainer. 😉

[mherwege]

@lolodomo I am not sure this a a chart issue. It would be interesting to see if the same issue exists with another underlying database. I suspect it may be an rrd4j issue. There have been a number of PR's recently changing the way data is stored and/or retrieved. See:

• [rrd4j] Improve the interpolation workaround openhab-addons#17437
• [rrd4j] Avoid RrdDb.getRrdDef calls openhab-addons#17449
• [rrd4j] Improved the internal data structure openhab-addons#16389
• [rrd4j] Reuse the state for identical values openhab-addons#16379
• [rrd4j] Improve querying the RRD4jPersistenceService openhab-addons#16360

[mherwege]

I have now tried with my production system on OH 4.3, storing the same data in rrd4j and influxdb. I cannot see a noticable difference. For me, all the lines are interpolated, no steps. I tested with period definitions set to: 6M, 2M, M, 31D, 30D, 4W, 2W. There is no difference.
For the tested value, I do persist data on every change, and that is roughly every 15 minutes. I have not done any tweaking on the RRD4J configuration.

[mherwege]

@der-dag I get you have values stored for the day only once? Do you have some rule that also stores the same value at the end of the day? That could explain this kind of visualisation. As you change chart periods, RRD4J would be reading from another aggregation and may ignore these second stored values. The interpolation between datapoints is always linear. But it just doesn't see the other value anymore, so it is not visualized as a step anymore.

openhab/openhab-core#4610 would allow you to configure this.

[mherwege]

@lolodomo Here is another observation.

Here is the core default ChartProvider: https://github.com/openhab/openhab-core/blob/main/bundles/org.openhab.core.ui/src/main/java/org/openhab/core/ui/internal/chart/defaultchartprovider/DefaultChartProvider.java
There is a specific ChartProvider for RRD4J: https://github.com/openhab/openhab-addons/blob/main/bundles/org.openhab.persistence.rrd4j/src/main/java/org/openhab/persistence/rrd4j/internal/charts/RRD4jChartServlet.java

I don't know why that is the case. That may very well be for historic reasons. I don't know if it is used anywhere explicitely, or if it could just be removed.

Both may fulfill the component requirements. The RRD4J version seems to not be in sync with the core version, therefore could generate different results.
It may explain why I don't see a difference, as I have multiple persistence services and the core version probably fulfills the requirements.
The RRD4J version would not support the updated period parameters.

[der-dag]

@der-dag I get you have values stored for the day only once? Do you have some rule that also stores the same value at the end of the day? That could explain this kind of visualisation. As you change chart periods, RRD4J would be reading from another aggregation and may ignore these second stored values. The interpolation between datapoints is always linear. But it just doesn't see the other value anymore, so it is not visualized as a step anymore.

openhab/openhab-core#4610 would allow you to configure this.

In RRD4J I have defined a persistence strategy "everyMinute" for such channels. The value is changed in a "midnight" rule only once a day. I'm using this setup for years now and just observed that my charts looked different after updating OH and then figured out that I could get my (good) old behaviour back if I change chart period from "M" to "30D".

Thanks for pointing me to the new interpolation parameter, I'll try that out.

[mherwege]

Thanks for pointing me to the new interpolation parameter, I'll try that out.

It's not merged yet, and will only be in OH 5.0.0.

In RRD4J I have defined a persistence strategy "everyMinute" for such channels. The value is changed in a "midnight" rule only once a day. I'm using this setup for years now and just observed that my charts looked different after updating OH and then figured out that I could get my (good) old behaviour back if I change chart period from "M" to "30D".

So, it is strange I don't see the same for all my tests. There must be something fundamentally different in our systems.

everyMinute is required for RRD4J, and I have that as well. But for me the values do change within a day.

The test I did was with a Number:Quantity item. Did you test with a pure Number item without unit? There is a difference in aggregation for the default configuration, see: https://www.openhab.org/addons/persistence/rrd4j/#default-datasource. On the time periods evaluated, I don't expect this to be a major difference though.

You don't have any specific RRD4J configuration impacting these default?

[der-dag]

I have Number items only, no units defined and I guess I have no special RRD4J config, my services/rrd4j.cfg is empty. The only difference between our systems is the changes per day for the items.

I have other items that only change once a month and there the charts are fine (with steps) as long as the choosen period is less one year: 365D, 9M are good, 1Y, 2Y are interpolating

[mherwege]

OK, I have been able to reproduce it to some extent with my data, but this time on an hour timescale.

If you look at the image, it looks like I have pure steps when looking at 1h (or 60min), and there are some slopes when I look at 2h (or 120min) time setting. Yours only happens on much longer timescales as your data is updated much less frequently.

In my reading, this is caused by the way RRD4J is querying the data. It looks at data with a specific granularity and applies a consolidation function on it when returning the results. The granularity and consolidation function depend on the time horizon selected and the item type. The charting tool will only interpolate between data points for Number times at the moment. And while it looks like a step in your example, it is because what is returned from RRD4J has consecutive datapoints with constant value, and then a step. Slightly changing the time selection can force RRD4J to retrieve values with another granularity, leading to different results.

The period parameter is converted to a begin- end endate for the query to RRD4J. Also this has an impact. If these dates are more or less aligned to the buckets, the results will be different.

The one change that may have impacted your results is indeed the introduction of the period parameter. A M parameter used to be converted to 30D as a Duration in OH code. It now is converted to a Period of 1 month. The difference is when the begin and end dates are calculated afterwards. Adding a Duration adds the exact amount of nanoseconds for 30 days. Adding a 1 month Period will respect the calendar, daylight savings etc for the base it is being added to. And that gives a different outcome for the chart period, that may lead to a different selection of granularity in the RRD4J datastore.

You could actually check this without charting by just doing a query on the RRD4J with begin and end dates set, and see what values come back. You can use the REST API or persistence extensions for this.

Overall, I don't think this is easy to solve with RRD4J as an underlying DB. I don't see any of this with InfluxDB, that's always interpolated. The interpolation parameter may help for visualization.