-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docs: Hint about potential high cardinality metrics #5603
Conversation
Add warning about potential high cardinality metrics due to the file path being included in some of the metrics. Depending on the setup, such config can result in a very large amount of label values, which is not recommended by Prometheus. This can result in increased storage, slow down of queries, extra costs and so on. Address what has been discussed in #5553.
Thanks for submitting this @m1keil 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
@@ -82,6 +82,13 @@ exposed by Promtail at its `/metrics` endpoint. See Promtail's documentation on | |||
An example Grafana dashboard was built by the community and is available as | |||
dashboard [10004](https://grafana.com/dashboards/10004). | |||
|
|||
## Metrics cardinality | |||
|
|||
Some of the Loki observability metrics are emitted per tracked file (active), with the file path included in labels. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should mention Promtail here explicitly... "per tracked file" only makes sense in the context of Promtail.
Removed redundancy:
Some of the Loki observability metrics are emitted per tracked file (active), with the file path included in labels. | |
Some of the Loki metrics are emitted per tracked file (active), with the file path included in labels. |
A hint about which metrics are emitted per tracked file would be kind to the operator, so they are able to configure the scraping to avoid this issue.
Hi! This issue has been automatically marked as stale because it has not had any We use a stalebot among other tools to help manage the state of issues in this project. Stalebots are also emotionless and cruel and can close issues which are still very relevant. If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry. We regularly sort for closed issues which have a We may also:
We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, |
Not stale. . . |
@KMiller-Grafana can you please re-review? Your requested changes are blocking a merge |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes. Sorry for the delay in my re-review.
What this PR does / why we need it:
Add warning about potential high cardinality metrics due to the file path being included in some of the metrics.
Depending on the setup, such config can result in a very large amount of label values, which is not recommended by Prometheus.
This can result in increased storage, slow down of queries, extra costs and so on.
Address what has been discussed in #5553.
Which issue(s) this PR fixes:
Fixes #5553
Checklist
CHANGELOG.md
about the changes.