Is there any ability to persist metrics based on trace duration?

[jSwanson99]

Not spanmetrics, which are great in their own way but sometimes it is more beneficial to have metrics for an entire trace. For example in the below websocket / kafka soup trace:

I really just want to know how long the whole thing took, it would make alerting much cleaner if there is some ability to scrape trace levll metrics as well as span level metrics

[jSwanson99]

I am aware of the trace metrics queries, but my concern with those is

1. I really just want prometheus metrics in mimir, for the purpose of simplicity and history
2. I don't think these support alerting (yet ? )

[joe-elliott]

The challenge for trace level metrics will always be that it's impossible to know when a trace is received in its entirety to correctly calculate the metrics. Additionally buffering traces at scale to make this decision is costly.

Having said that if we are very specifically targeting total trace duration you would not need to store the full trace. Perhaps a map of trace ids to earliest span start time and latest span end time? Then, after some time had passed, assume it is as good as it would get and write a metric to prometheus?

Challenges:

• For long running traces you'd have to keep this data for quite a long time. This is a challenge across Tempo as a whole though so I don't think it's out of question.
• The data point would be considered after all spans are received and some timeout expires. This would delay how its written to the metrics system and may cause confusion. If a long trace (minutes/hours/days?) was processed it would impact the metric at the end of the trace. Perhaps no big deal?

Ultimately I'm not opposed to a new metrics generator processor that performs this calculation if someone wanted to attempt the PR.

[jSwanson99]

I am interested in contributing to the project, I would just need some guidance in getting started (could you point me towards some docs) and I would not have the speediest turnaround time.
I think the "metric at the end of the trace" is whats important to me, since my goal with this is simpler alerting for event based arch:

• Use trace level metrics to out-of-the-box alert on what's slow
• Configure more specific spanmetrics based alerts for specific internals that you care about

Something related to this question though, is I am not understanding the numbers coming from my service graph processor. I would expect my spanmetrics to have less of a "realistic" number in this example, since it has all span kinds muddled together; I would have expected my service graph to be the more realistic depiction of latency, since I thought it would be matching the producer/consumer spans together. The latency of these traces was around ~8-10 seconds at this time, so the spikes in the spanmetrics make sense, but I don't get why the service graph doesnt budge. Am I mistaken in how the service graph is working / does this seem right to you:

[joe-elliott]

I would just need some guidance in getting started (could you point me towards some docs)

This exists which is probably mostly up to date. I would add a new processor here: https://github.com/grafana/tempo/tree/main/modules/generator/processor

This processor would have a PushSpans method like this. In this method you would record the earliest start time of a trace and the latest end time. Then every so many seconds you'd review the stored values. If a trace had not received spans in N seconds then we'd consider it complete and observe the metric.

I would not have the speediest turnaround time.

No worries

The latency of these traces was around ~8-10 seconds at this time

I can't say for sure, but i noticed you mention matching "producer/consumer" spans and you are graphing the client side of the service graph. Perhaps try the server side? If its async queue based exchange then I would expect the producer side to remain steady even while the consumer side increased in duration.