How can we record specific traces 100% of the time when sampling is enabled?

[travisbell]

Hey everyone,

Been playing around with this general idea, and I just can't seem to figure out how it's supposed to be implemented. I have sampling enabled via the following ENV vars:

OTEL_TRACES_SAMPLER="parentbased_traceidratio"
OTEL_TRACES_SAMPLER_ARG="0.1"

Which is working fine but I have the need for some specific traces to be recorded 100% of the time.

How does one go about doing this?

[arielvalentin]

The only real supported way of doing this is using Tail Sampling in the OTel Collector:

• https://opentelemetry.io/docs/concepts/sampling/#tail-sampling
• https://opentelemetry.io/blog/2022/tail-sampling/

The OTel Probabilistic Sampling Processor also has some backward compatibility with OpenTracing sampling.priority but it only works capturing all details of a trace across services if you use a single collector or if you add that attribute to every single span related to that trace.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor#probabilistic-sampling-processor

[travisbell]

Ah, I see. Right, if I'm head sampling, it's too late for the library to make that decision since it never processes the spans to begin with. By moving to tail sampling, we move where the decision is being made.

Ok, thanks Ariel. 👍🏼