How to fix Distributor to Metrics Generator queue full

[Chelsea-Barraball]

Hey guys!

We're getting a lot of these errors from our distributer component to our metrics generator:

level=error msg="failed to push traces to queue" tenant=single-tenant err="failed to push data to queue for tenant=single-tenant and queue_name=metrics-generator: queue is full"

I went onto the Tempo documentation Troubleshoot Metrics Generator section, and found the metrics that I can use to view the queue push/failures, but there are no recommendations for potential fixes. I also can't find any configuration options for the distributer's queue. Can someone point me in the direction of how to begin tackling this issue?

Thank you!

[mdisibio]

Hi, the error queue is full means that the metrics-generator is not able to consume data as fast as it is received by the distributor. The distributor has a small in-memory buffer of requests for the generator, and when it is full this error is recorded and the data is skipped.

There are several factors involved and we can try a few things. There isn't one answer.

• Scale up the generators. There isn't specific guidance on how many are needed since trace data and processing can vary a lot, but easy to try.
• Increase the tenant metrics_generator_ring_size. This is how many generators the tenant's traffic is split among. If you have 5 generators, and metrics_generator_ring_size=2, then tenant's data is only going to 2 of them. Increasing this means there are more generators available to consume the data. However this also increases the final number of active series.
• If the volume is spiky, you can increase the metrics_generator_forwarder_queue_size. But be aware that this also increases distributor memory requirements.

[jcarres-mdsol]

Wouldn't having more than one generator prevent having proper values for traces_service_graph_request* metrics? As these need to see all spans.

[yvrhdn]

When we run multiple instances of the metrics-generator we load balance spans by their trace ID. This ensures each generator receives all spans related to a trace, which is required for the service-graphs processor as you rightfully noticed.
The downside of this is that each generator will have roughly the same cardinality, so the amount of replicas is a direct factor of the amount of active series generator in total.

[michmour]

Hi, the error queue is full means that the metrics-generator is not able to consume data as fast as it is received by the distributor. The distributor has a small in-memory buffer of requests for the generator, and when it is full this error is recorded and the data is skipped.

There are several factors involved and we can try a few things. There isn't one answer.

• Scale up the generators. There isn't specific guidance on how many are needed since trace data and processing can vary a lot, but easy to try.
• Increase the tenant metrics_generator_ring_size. This is how many generators the tenant's traffic is split among. If you have 5 generators, and metrics_generator_ring_size=2, then tenant's data is only going to 2 of them. Increasing this means there are more generators available to consume the data. However this also increases the final number of active series.
• If the volume is spiky, you can increase the metrics_generator_forwarder_queue_size. But be aware that this also increases distributor memory requirements.

We are experiencing the same issue described above. I attempted to address it by scaling up the metrics-generator to 2 pods and setting the ring_size to 2 in the tempo.yml file but didn't solve the issue. Is this the recommended approach to resolve these issues?

[jcarres-mdsol]

We ended up solving this by having 6 metrics generator pods running