[BUG]: Observability data values are incorrect.

[DougBernhardt]

Bug Description

The values in the grafana charts and as viewed in Prometheus do not match the actual values from PowerStore. The values are consistent between prometheus and grafana, but do not match actual values from the array.

Logs

response_1651238945578.zip
https://snapshots.raintank.io/dashboard/snapshot/E1pkwxj5SuOnP0d60CvZOmMiBPrZ8aW0

Screenshots

Additional Environment Information

No response

Steps to Reproduce

Compare Grafana charts (PowerStore volume IO metrics) to PowerStore UI or REST API call.

Expected Behavior

Values should match

CSM Driver(s)

CSM Observability Module for PowerStore 1.1.0 (Initially had issue with values populating, got a dev build to test, data values are populating but are incorrect. Current image is:
xx.xx.xx.xx:5000/csm-metrics-powerstore latest c0690c7c233b6

Installation Type

Helm

Container Storage Modules Enabled

Observability 1.1.0

Container Orchestrator

Kubernetes v1.23.5

Operating System

RHEL 8.5

[shaynafinocchiaro]

Hi @DougBernhardt, can you try updating the polling frequency value in your values.yaml file for deployment to see different data points and update this defect with your findings.

[DougBernhardt]

I updated the polling frequency to 5s. No difference. Values always increase but never decrease. In the example below, the dip is when the modules were uninstalled and installed to update the polling frequency. In reality, the volumes had a load for approx two hours after the change was made and then they had no activity the rest of the time, so the values should have dropped to near zero.
https://snapshots.raintank.io/dashboard/snapshot/GixG1LVhbFToX7svfI9ywd5SEWOjCerW

[shaynafinocchiaro]

@DougBernhardt can you try importing our file system io metrics dashboard to check how those values are rendering with the array that you're utilizing.

[DougBernhardt]

I don't have file features enabled on the array and therefore the CSI file features are not enabled.

[hoppea2]

@DougBernhardt We have reproduced the issue in house and working on RCA

[hoppea2]

Fix will be available 05/24

[hoppea2]