How to add service labels?

[tcurdt]

I want to have labels on the metrics from the discovery:

    - job_name: integrations/nodejs
      metrics_path: /api/metrics
      consul_sd_configs:
      - server: '127.0.0.1:8500'
        services:
          - 'backend-production'
          - 'backend-staging'
      # metric_relabel_configs:

I could not find an example or docs on how to match inside Grafana which service the metrics were coming from ('backend-production' or 'backend-staging')

I want the data coming from the "backend-production" service to also have a label added clarifying it is coming from a backend-production service.

The instance label has the host and port - but I also need the service label.
Unfortunately I have not found any information in the docs on how to do that when the discovery is dynamic.

Right now I am getting for example:

backend_foo{instance="10.0.1.2:22026"}
backend_foo{instance="10.0.1.2:22023"}

But what I want is

backend_foo{instance="10.0.1.2:22026", service="backend-production"}
backend_foo{instance="10.0.1.2:22023", service="backend-staging"}

I don't see how to write the relabel_configs/metric_relabel_config to reference the service.

[rfratto]

The Prometheus documentation says that __meta_consul_service is the meta label containing the service name that you can use for relabeling.

relabel_configs:
- source_labels: ['__meta_consul_service'] 
  target_label: service 

[tcurdt]

Thanks! I will give that a try.

I was a bit lost on where to place the relabel config.

[tcurdt]

Should it be relabel_configs or metrics_relabel_configs?

        - job_name: integrations/nodejs
          metrics_path: /api/metrics
          consul_sd_configs:
          - server: '127.0.0.1:8500'
            services:
              - 'backend-production'
              - 'backend-staging'
          # metric_relabel_configs:
          relabel_configs:
          - source_labels: ['__meta_consul_service'] 
            target_label: service 

Is there a schema for current version of the yml?

[rfratto]

It must be relabel_configs. Labels starting with __meta_ are only available prior to scraping. relabel_configs applies to discovered targets prior to scraping, while metric_relabel_configs applies to individual metrics post-scraping.

Is there a schema for current version of the yml?

https://grafana.com/docs/agent/latest/configuration/