Thanos Query Sidecar Endpoint specific metric?

[rgarcia89]

Hi, I am currently alerting when the GRPC Client Error Rate for one of the Thanos Query Endpoints is rising. However, in this alert I am not seeing which endpoint exactly is causing the issue. Does anyone know if there is a metric which I could use, in order to obtain these information?

Current alert definition

        - alert: ThanosQueryGrpcClientErrorRate
          annotations:
            description: Thanos Query {{$labels.job}} is failing to send {{$value | humanize}}% of requests.
            runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosquerygrpcclienterrorrate
            summary: Thanos Query is failing to send requests.
          expr: |
            (
              sum by (job) (rate(grpc_client_handled_total{grpc_code!="OK", job=~".*thanos-query.*"}[5m]))
            /
              sum by (job) (rate(grpc_client_started_total{job=~".*thanos-query.*"}[5m]))
            ) * 100 > 5

[saswatamcode]

The grpc_method and grpc_service methods should give you a clue! Try to group by those labels.

[rgarcia89]

@saswatamcode what I am looking for to be honest are metrics that make me link the issue to the specific sidecar endpoint which is causing it