Tempo Trace Filter

[andreivaloshyn]

Hello!
We are using Istio to produce traces.
Tempo shows a lot of useless "metrics" traces such as

{
  "value": "http://10.33.60.105:8081/metrics",
  "key": "http.url"
},
{
  "value": "Prometheus/2.39.1",
  "key": "user_agent"
},
{
  "value": "server",
  "key": "span.kind"
},
{
  "value": "server",
  "key": "span.kind"
},
{
  "value": "sidecar~xxxx.svc.cluster.local",
  "key": "node_id"
}

OR

{
  "value": "http://10.33.58.109:8080/actuator/prometheus",
  "key": "http.url"
}

is there a way how we can reject such "Prometheus" traces with Tempo ?

[joe-elliott]

I don't know much about configuring Istio, but at Grafana we use Jaeger remote sampling to set per endpoint sampling. This is supported by Jaeger clients as well as some OTEL clients.

Alternatively for single span traces like /metrics they would likely be easy to identify and drop using a processor in the otel collector. Something like: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md

[andreivaloshyn]

Thank you!