Grafana Mimir does not send metrics instantly to s3

[felipefrx]

Metric blocks are stored for a while in the ingester to be sent to s3. I would like to know if there is any parameter for it to send them instantly.

I saw that in Loki there are some parameters related to this, such as:

ingester:
  chunk_idle_period: 1m
  max_chunk_age: 1m

[pstibrany]

No, that's not how Mimir works. Mimir buffers data on local nodes for up to 3 hours, and only then build blocks and uploads them to cloud storage.

The data can be queried via API immediately.

[pstibrany]

How can we avoid these log metrics losses?

Upload to S3!

Once Mimir ingests your metrics, metrics should survive the restart, if Mimir is set up correctly. Are you saying that's not the case?

[flowramps]

This is my current structure!

My values.yaml structure!

mimir-values.txt

Is there any specific configuration of what I configured in my values ​​?

Mimir receives the metrics and sends them to an s3, through common and blocks_storage, but without a persistent volume enabled
and every time the pods restart, we lose about 20 minutes of metrics!

[pstibrany]

Mimir ingesters require persistent volumes, that is where they store data between restarts, while data is not yet in blocks.

[pstibrany]

I see that Mimir helm chart values.yaml file supports disabling of persistent volumes for ingesters. That seems like an oversight, and something we should fix.

[mrkiani98]

Hi @pstibrany

I understand how mimir works in terms of keeping the blocks first in local storage and then ship them to the backend s3 location.
And I also understand that we can use for example AWS EBS volumes or maybe some storage services in on-prem environment, when using Kubernetes clusters, but then we need to think of taking a backup of these block storage services to make sure the metrics are 100% safe.
What can be the suggestions not to lose the metrics when deploying mimir in kubernetes clusters inon-prem or even cloud based production environement?