W3C Baggage Propagation not working with OpenTelemetry instrumented code in a Python Lambda

[gromanek]

I have a python Lambda set up with the DD Lambda extension which extracts both OpenTelemetry trace context and W3C style baggage from the Lambda event with the following code. The Lambda spans are showing up in DataDog correctly and the trace context extraction seems to be working because they are showing up properly linked to other spans outside of the Lambda, but the baggage propagation isn't working - I wasn't sure if this is because that isn't supported or user error on my part.

The code looks like this:

        tracer = trace.get_tracer("request-canceler")
        ctx = TraceContextTextMapPropagator().extract(carrier=event)
        ctx2 = W3CBaggagePropagator().extract(carrier=event, context=ctx)
        token = context.attach(ctx2)
        try:
            with tracer.start_as_current_span(
                "request-canceler.process_request_canceler_event", ctx2
            ) as span:

Where event is the JSON passed into the Lambda and looks like this (trace context and baggage at the bottom)

{
  "tracking_id": "test_id",
  "reference_id": "test_ref_id",
  "site_id": "02800",
  "request_source": "hca",
  "request_type": "standard",
  "traceparent": "00-01dac916538653cf9f1bd24d71ab8426-cea3886ffb1f9cd4-01",
  "baggage": "reference_id=8542331,tracking_id=c245bc0c-cf00-434e-bd84-25415d7dcc0c,main_reason=State+DDS,secondary_reason=,major_class=GOV,requester_type=Disability,stat=0,site_id=02800,source_id=,emr=hca,health_system=hca,channel=HCA"
}

This is how we handle baggage propagation in all of our non-Lambda code and it works there, having trouble figuring out why it isn't working in this context. Here's the code for how we're setting up OTel in the Lambda + relevant environment variables we have configured.

if os.getenv("TRACING_ENABLED", "false").lower() == "true":
    endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    provider = TracerProvider()
    otlp_exporter = OTLPSpanExporter(endpoint=endpoint)
    span_processor = BatchSpanProcessor(otlp_exporter)
    provider.add_span_processor(span_processor)
    provider.add_span_processor(BaggageSpanProcessor(ALLOW_ALL_BAGGAGE_KEYS))
    trace.set_tracer_provider(provider)


@datadog_lambda_wrapper
def lambda_handler(event: dict[str, str], context: dict[str, Any]) -> None:
    request_canceler = RequestCanceler(RequestCancelerConfig())
    request_canceler.process_request_canceler_event(event)

Env vars

    DD_ENV : "staging"
    DD_API_KEY_SECRET_ARN : var.dd_api_key_secret
    DD_LAMBDA_HANDLER : "start.execute_lambda"
    DD_SITE : "ddog-gov.com"
    DD_TRACE_OTEL_ENABLED :"true"
    DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT : "localhost:4317"
    TRACING_ENABLED : "true"
    OTEL_EXPORTER_OTLP_ENDPOINT : "http://localhost:4317"
    DD_PROFILING_ENABLED : "false"
    DD_SERVICE : "request-canceler"

Anything jumping out that I'm missing for baggage propagation to work? Is that just not a supported feature?

[purple4reina]

Sadly, it's just not a supported feature at this time. I think though it should be possible for you to do this extraction manually inside of your function though. Would you like help with that?

[gromanek]

Gotcha, thank you! Appreciate the quick reply, I think I can set up the manual extraction 👍

[astuyve]

Thanks folks! I'm gonna close this issue as it's related to the datadog-lambda-python library and not this project.

Best!
AJ