Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm: default dashboards use a job label that doesn't exist #11390

Open
alarobric opened this issue Dec 5, 2023 · 2 comments
Open

Helm: default dashboards use a job label that doesn't exist #11390

alarobric opened this issue Dec 5, 2023 · 2 comments
Labels
area/helm type/bug Somehing is not working as expected

Comments

@alarobric
Copy link

Describe the bug
The default Loki dashboards shipped with the Helm chart fail to load data, due to using a job label that doesn't match the metrics that are collected.
The dashboard is querying metrics like this:
loki_ingester_memory_chunks{cluster="$cluster", job=~"$namespace/(loki|enterprise-logs)-write"}
but the ServiceMonitor is generating metrics with job=namespace/loki, not `job=namespace/loki-write'

To Reproduce
Steps to reproduce the behavior:

  1. Installed loki in EKS using the Helm chart. Installed promtail, grafana, and kube-prometheus-stack alongside.
  2. Open the 'Loki/Chunks' dashboard for example. No data is shown.
  3. Drill into a chart widget, and modify the query to remove -write from the job label. Data shows as expected.

Expected behavior
Default dashboards should work alongside the default ServiceMonitor configuration.

Environment:

  • Infrastructure: EKS
  • Deployment tool: Helm via Terraform

Screenshots, Promtail config, or terminal output

@alarobric
Copy link
Author

I guess this sounds like a related issue to #7657 or #9469

@JStickler JStickler added type/bug Somehing is not working as expected area/helm labels Dec 11, 2023
@HaveFun83
Copy link

we run into the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/helm type/bug Somehing is not working as expected
Projects
None yet
Development

No branches or pull requests

3 participants