[Question] Does mimir compress consecutive equal data points from one series into one data point

[Poweranimal]

Does mimir compress consecutive equal data points from one series into one data point?

Context:
I try to query metrics with a specific time vector, but don't get any result back.
However, when I increase the time vector, a proper result gets returned.
This only happen with a time vector smaller or equal 5m with some metrics.
If I increase the time vector to 5m1s I get a result.

Example:

irate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}[5m])

-> returns no data

irate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}[5m1s])

-> returns one data point

However, I checked the prometheus config and can confirm that the metrics are scraped every 30s with a timeout of 10s.

Shouldn't I have multiple data points with a time vector of 5m?
This 5m magic number looks to me like some sort of limit somewhere to be enforced.

I look forward to your help.

[johannaratliff]

Hi @Poweranimal, I think this is a matter of the PromQL query. Mimir doesn't aggregate/rollup data points in this way.
So my guess is that it's related to the use of irate.
Does the container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}[5m] query itself return multiple data points?

[Poweranimal]

Hi @johannaratliff, thank you very much for the quick response.
Yes, querying container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!=""} returns multiple data points.

The example query I showcased is part of many queries from predefined dashboards of helm kube-prometheus-stack. I use all of their dashboards (defined here) including their Prometheus recording rules.
All of the defined dashboards show proper graphs, except some panels that use metrics from kublet via /metrics/cadvisor (mostly metrics related to network activity). This seems to be caused by the preset range vectors of 5m or smaller.

I wonder, if the presets are wrong (however, I highly doubt that) or if something is odd regarding the metrics scraping via Prometheus Agent or the metrics processing of Mimir.

[johannaratliff]

Sorry I can't be of further help. I can assure you that the metrics coming from Mimir are intended to be Prometheus compliant including returning all scraped and stored metrics, and would not do any secret rollups on queries to the best of my knowledge. If you figure this out and there is an issue and our end - please file it to let us know!

[Poweranimal]

Thanks for your reply and your help on the matter.
I’ll try my best to figure out, what might be the issue on my side :)

[pracucci]

Does this issue happen only on cAdvisor metrics? If so, then I would recommend to double check the cAdvisor metrics exporter.

A theory: I remember cAdvisor export metrics with timestamp (quite unusual), so the sample timestamp used is the one provided by cAdvisor and not the time when Prometheus does the scraping. If cAdvisor updates the metrics only once every 5m, then you would get the behaviour you're describing, and the 30s scrape interval of Prometheus wouldn't matter.

[Poweranimal]

Hi @pracucci , thank you very much for your response.
That seems to be the problem indeed.

A metric may look like this:

container_cpu_usage_seconds_total{container="POD",cpu="total",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaade6160_63cd_465a_bcfa_baca8c3f227f.slice/docker-cf51bfa13d91fd381790620905aae0de64dac0c28d65b2f7b6678294b66707f9.scope",image="k8s.gcr.io/pause:3.4.1",name="k8s_POD_storage-provisioner_kube-system_aade6160-63cd-465a-bcfa-baca8c3f227f_0",namespace="kube-system",pod="storage-provisioner"} 0.208482875 1655475393125

The last part of the metrics, 1655475393125, is the timestamp in nanoseconds you mentioned.

I tracked the change of the timestamp and it seems to be updated very infrequently and with long delays (more than 5m).

I guess, now I have to figure out how change the update interval of cadvisor in minikube.

Thank you very much for your help!

[Poweranimal]

For the record:
One can change the update interval of cadvisor in kublet with the flag --housekeeping-interval=5s.
If one uses minikube, one needs to add the following flag to the start command: --extra-config=kubelet.housekeeping-interval=5s