Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Hint about potential high cardinality metrics #5603

Merged
merged 2 commits into from
May 20, 2022
Merged

Docs: Hint about potential high cardinality metrics #5603

merged 2 commits into from
May 20, 2022

Conversation

m1keil
Copy link
Contributor

@m1keil m1keil commented Mar 11, 2022

What this PR does / why we need it:
Add warning about potential high cardinality metrics due to the file path being included in some of the metrics.
Depending on the setup, such config can result in a very large amount of label values, which is not recommended by Prometheus.
This can result in increased storage, slow down of queries, extra costs and so on.

Address what has been discussed in #5553.

Which issue(s) this PR fixes:
Fixes #5553

Checklist

  • Documentation added
  • Tests updated
  • Add an entry in the CHANGELOG.md about the changes.

Add warning about potential high cardinality metrics due to the file path being included in some of the metrics. 
Depending on the setup, such config can result in a very large amount of label values, which is not recommended by Prometheus.
This can result in increased storage, slow down of queries, extra costs and so on.

Address what has been discussed in #5553.
@CLAassistant
Copy link

CLAassistant commented Mar 11, 2022

CLA assistant check
All committers have signed the CLA.

@dannykopping
Copy link
Contributor

Thanks for submitting this @m1keil 👍
I'll leave this for @KMiller-Grafana to review

Copy link
Contributor

@kavirajk kavirajk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
@@ -82,6 +82,13 @@ exposed by Promtail at its `/metrics` endpoint. See Promtail's documentation on
An example Grafana dashboard was built by the community and is available as
dashboard [10004](https://grafana.com/dashboards/10004).

## Metrics cardinality

Some of the Loki observability metrics are emitted per tracked file (active), with the file path included in labels.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should mention Promtail here explicitly... "per tracked file" only makes sense in the context of Promtail.

Removed redundancy:

Suggested change
Some of the Loki observability metrics are emitted per tracked file (active), with the file path included in labels.
Some of the Loki metrics are emitted per tracked file (active), with the file path included in labels.

A hint about which metrics are emitted per tracked file would be kind to the operator, so they are able to configure the scraping to avoid this issue.

@stale
Copy link

stale bot commented Apr 29, 2022

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

  • Mark issues as revivable if we think it's a valid issue but isn't something we are likely
    to prioritize in the future (the issue will still remain closed).
  • Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

@stale stale bot added the stale A stale issue or PR that will automatically be closed. label Apr 29, 2022
@KMiller-Grafana
Copy link
Contributor

Not stale. . .

@stale stale bot removed the stale A stale issue or PR that will automatically be closed. label Apr 29, 2022
@dannykopping
Copy link
Contributor

@KMiller-Grafana can you please re-review? Your requested changes are blocking a merge

Copy link
Contributor

@KMiller-Grafana KMiller-Grafana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes. Sorry for the delay in my re-review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Promtail generates very high cardinality prometheus metrics
5 participants