I set 'metricsGenerator' to 'true' but no metrics found in prometheus. Please help me.

[pingping95]

Hi.

Im using Grafana tempo , Loki so well.

Im trying to use Grafana tempo Service graph using metrics-generator

My environment :
AWS EKS(Kubernetes) 1.24 Version
Tempo 1.5.0 version (With Helm chart)

And this is my helm-values.yaml

...

  metricsGenerator:
    enabled: true
    remoteWriteUrl: "http://kube-prometheus-stack-prometheus:9090/api/v1/write"

...

  overrides:
    {{- toYaml .Values.tempo.global_overrides | nindent 6 }}
    {{- if .Values.tempo.metricsGenerator.enabled }}
        metrics_generator_processors:
        - 'service-graphs'
        - 'span-metrics'
  metrics_generator:
        storage:
          path: "/tmp/tempo/generator/wal"
          remote_write:
            - url: {{ .Values.tempo.metricsGenerator.remoteWriteUrl }}
    {{- end }}

..

  overrides: {}

but I can`t find any metrics at prometheus. (https://grafana.com/docs/tempo/latest/metrics-generator/service_graphs/#metrics)

How do i troubleshoot ? If need more information, Please tell me.

[yvrhdn]

Hi, what does your configmap look in the end? Especially the overrides section. You have overrides: twice in your configuration, I'm not sure if the latter will overwrite the first.

[pingping95]

@kvrhdn

Hi! Thanks for answering.

This is my tempo configmap. overrieds and metrics_generator.

Is there anything i have to change ?

overrides:
      per_tenant_override_config: /conf/overrides.yaml
      metrics_generator_processors:
      - 'service-graphs'
      - 'span-metrics'
metrics_generator:
      storage:
        path: "/tmp/tempo/generator/wal"
        remote_write:
          - url: http://kube-prometheus-stack-prometheus:9090/api/v1/write


BinaryData
====

Events:  <none>

Ans this is my tempo Logs.

k logs tempo-0 -n monitoring -c tempo | grep -i generator

level=info ts=2023-03-01T10:37:21.395767655Z caller=module_service.go:82 msg=initialising module=metrics-generator-ring
level=info ts=2023-03-01T10:37:21.395886293Z caller=client.go:255 msg="value is nil" key=collectors/metrics-generator index=1
level=info ts=2023-03-01T10:37:21.397083978Z caller=module_service.go:82 msg=initialising module=metrics-generator
level=info ts=2023-03-01T10:37:21.397129229Z caller=basic_lifecycler.go:261 msg="instance not found in the ring" instance=tempo-0 ring=metrics-generator
level=info ts=2023-03-01T10:37:22.212520839Z caller=instance.go:46 tenant=single-tenant msg="creating WAL" dir=/tmp/tempo/generator/wal/single-tenant
level=info ts=2023-03-01T10:37:22.216884414Z caller=db.go:332 tenant=single-tenant component=wal msg="replaying WAL, this may take a while" dir=/tmp/tempo/generator/wal/single-tenant/wal

Thanks.

[yvrhdn]

This looks okay, I'm not sure if the metrics-generator isn't generating any metrics at all or just no service graph metrics.

Can you check the following metrics?

• Bytes received by the metrics-generator:

sum(rate(tempo_metrics_generator_bytes_received_total{}[$__rate_interval]))

• Spans received by the metrics-generator

sum (rate(tempo_metrics_generator_spans_received_total{}[$__rate_interval]))

• Active series in the metrics-generator

sum(tempo_metrics_generator_registry_active_series{})

If these are active, it means the metrics-generator is received some traces and generating some metrics, just not the service graph metrics. In that case you likely have to set SpanKind in your instrumentation as described here: https://grafana.com/docs/tempo/latest/metrics-generator/service_graphs/#how-they-work

[pingping95]

@kvrhdn

Hi! Thanks for replying.

It was my mistake.

When i see logs more, Prometheus remote-write was disabled and Tempo was failed to write metrics to prometheus.

I fixed this issue and resolved.

I`ll also try to upgrade my tempo version v1.5 to v2.0 for more functions. Thanks for assistance!

[yvrhdn]

Glad you got it working! Have fun 🙂

[Shindek77]

could you please let me know that for which tool you have upgraded the version from V1.5 to V2.0

[zalegrala]

It looks like they are referring to the Tempo version @Shindek77 .