Metrics delivery reliability

[arnitolog]

Hello,
I have Grafana Agent deployed in flow mode with controller.type: deployment.
I did a small experiment and it looks like Agent can cover the case when Mimir is not reachable for a while. Does it use any local storage to keep metrics that are not deliverable or any kind of queue? If so, what are the default parameters, are they somehow configurable?

[tpaschalis]

Hey there @arnitolog. Yes, Grafana Agent reuses Prometheus' Write-Ahead Log (WAL) implementation to temporarily buffer data during network outages. More information about the data retention and parameters is available on this page. Data is stored on the agent's --storage.path argument which defaults to a data-agent directory. You can influence the remote write behavior during both normal operations and remote_write outages using the queue_config and wal blocks, while defaults are documented on the component's reference page

Currently, the metrics WAL implementation does not work for cases where the agent crashes or restarts; in these cases data is lost but the WAL is only replayed to warm up a cache.

A similar WAL implementation also exists for logs in loki.write, primarily meant for network-based log sources.