Support delayed sampling

[william-tran]

Requirement - what kind of business use case are you trying to solve?

I'd like to make sampling decisions according to service and operation name by leveraging remote controlled sampler.

Problem - what in Jaeger blocks you from solving the requirement?

Some instrumentation frameworks create a root span e.g. at servlet level, and use the HTTP method as the operation name; but then further refine the operation name as the request progresses through the stack, e.g. Spring MVC, which updates the operation name to the controller method. This prevents me from creating more specific sampling rules.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Implement delayed sampling along the lines of https://github.com/jaegertracing/jaeger-client-go/tree/v2.24.0#delayed-sampling

Any open questions to address

• We could start by moving flags into their own class like in All spans of a trace share sampling state jaeger-client-go#443, and having the same instance of SamplingState passed to child SpanContexts as they are created. Are there any issues with thread safety that need to be accounted for?
• Should we address a few of these metrics issues while we're at it?
  • Rethink client metrics for sampled/unsampled spans/traces jaeger#1862
  • Tag metrics with service, operation name, and certain span tags #716

See also:

• [tracking] Delayed sampling jaeger#1861
• Feature request: sampling based on tags. #682

[yurishkuro]

I think it's reasonable to approach this piecemeal starting with shared SamplingState. Thread safety issues are the same as always. In the Go client we opted to avoid using mutexes for synchronization and instead use spinning lock principle via atomic CAS operations. It's worth doing a benchmark first to see what's more appropriate for Java. Note that the sampling state is read very frequently, for every span write operation.

[william-tran]

Update on this:

I'm working on something internally first to validate the approach I'm taking. It doesn't go as far as the go client with a shared sampling state, but sample can be called on setOperationName AND on child context creation. This is to achieve something similar to https://github.com/openzipkin/openzipkin.github.io/wiki/secondary-sampling, but there's no ttl on the secondary sampling decision or reconnection or replacement of the unsampled parent with the one last sampled. I'm also fixing the concurrency bottleneck described here as there will be lots more calls to sample now.

[cptkng23]

@william-tran , @yurishkuro
Is this the issue to subscribe to for delayed sampling? If so, are there any plan to implement it in jaeger-java-client?