Use Bagge to link two spans across different threads

[elysweyr]

Hello,

TL;DR: Using auto-instrumented (Java Agent) Spring I want to schedule a job from (BusinessLogicService) which is afterwards processed at an undefined point of time in an asynchronous manner in (CustomJobScheduler).
The two spans involved can be correlated using correlationId.

My approach:

Context ctx = Context.current();
Baggage baggage = Baggage.current();
baggage = baggage.toBuilder().put("correlationId", correlationId).build();
baggage.storeInContext(ctx);

But I don't see any baggage data in my Jaeger Frontend. What am I missing here?
I've had a deep dive through the documentation - that's how I found the OpenTelemetry.get() approach.

Long Version:
Project is using java agent instrumentation. In addition to that I added
opentelemetry-sdk, opentelemetry-sdk-trace, opentelemetry-exporter-otlp, opentelemetry-sdk-logs, opentelemetry-api-logs, opentelemetry-sdk-metrics, opentelemetry-sdk-trace, opentelemetry-sdk-extension-autoconfigure to my pom.xml.

A configuration class was added with the following bean definitions:

@Bean
public OpenTelemetry openTelemetry() { return GlobalOpenTelemetry.get(); } 

@Bean
public TracerProvider tracerProvider() { return openTelemetry().getTracerProvider(); }

@Bean
public ContextPropagators contextPropagators() { return openTelemetry().getPropagators(); }

Finally the code below is applied to BusinessLogicService and CustomJobScheduler.

My approach:

Context ctx = Context.current();
Baggage baggage = Baggage.current();
baggage = baggage.toBuilder().put("correlationId", correlationId).build();
baggage.storeInContext(ctx);

One more question: Is there any way to configure the automatic span configuration to add more intermediate spans?

Thanks for your time! I appreciate any hints/tips which help me as a first-time OTEL user.

[jkwatson]

when you make this call: baggage.storeInContext(ctx); you're ignoring the return value, which is the context with the baggage in it. You need to use that context to wrap any further code that gets executed if you want to have the baggage available when needed.

[elysweyr]

Thanks for your answer! In my further research I found out that Baggage is not the correct concept I need to use here.

Basically there is no direct connection between the parts which schedule and execute jobs. Therefore I can’t propagate any context. The scheduler executes jobs (every 10s), the business logic service simply queues jobs and returns.

I stumbled across this discussion thread #4668 (comment)
which leads to the conclusion that I need to store the spanId as well as traceId from the span which queues the Job into some kind of cache identified by the jobId (either plain in-memory or redis).

The job scheduler will then retrieve the spanId and traceId using the jobId from the mentioned cache and constructs manually a SpanContext. Further job execution takes place inside a new span.

Is there anything else I need to keep in mind? (Attributes, flags which represent this indirect execution flow)

Thanks in advance and sorry for the confusion!

[elysweyr]

I've created a SpanContext using an externally stored traceId and spanId using createFromRemoteParent() and linked the Span in the job batch processor to this external SpanContext (Business Logic Service). The batch job processor's trace shows up a span with a reference to a the business logic service's trace. Is there any possibility to encapsulate this in one single trace to rule them all? It would be very neat to have one single traceId which allows tracking of this job processing. I'm asking this as the business logic trace ("entry point") doesn't contain these job batch processor spans.

Thanks in advance!

[jkwatson]

sounds like you've got a good plan. I don't know if we have any semantic conventions to model this particular structure. @jack-berg any thoughts?

[jack-berg]

Is there any possibility to encapsulate this in one single trace to rule them all? It would be very neat to have one single traceId which allows tracking of this job processing. I'm asking this as the business logic trace ("entry point") doesn't contain these job batch processor spans.

It sounds like you've properly associated the child span (i.e. the batch job processor span) with the parent span / trace (i.e. the span / trace initialized in the business logic service). Is the tool you're using to visualize the traces showing something different? Or perhaps I'm misunderstanding your goal?

[elysweyr]

I'll set up a demo project and link it here when ready