Support for AWS X-Ray

[sibethencourt]

Is your feature request related to a problem? Please describe.
X-Ray has its own collector implementation that requires a specific format for traceIds. Because of this, the otlp exporter is not able to transfer traces to X-Ray. This is the log error:

debug	awsxrayexporter@v0.66.0/awsxray.go:100	Error translating span.	{"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "invalid xray traceid: 970d7933a1b683da293e20e8047f2468"}

It is related to this bug: open-telemetry/opentelemetry-collector-contrib#2396

Describe the solution you'd like
Provide a way to define a custom traceId generator, AWS already offers one: AwsXrayIdGenerator

I need support for custom Id generators: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#id-generators should allow us to fix the issue.

Describe alternatives you've considered
I tried to look up the code and see if I can find a way to override the Id generators, but I haven't been able to do it so far. Right now the alternatives are either replacing X-Ray or not using sleuth-otel.

Additional context
More context here: open-telemetry/opentelemetry-collector-contrib#1646

Because the error I received and the comments in other forums it seems very clear what is the issue, but I haven't been able to confirm it yet, since I haven't found a way to workaround the problem.

[sibethencourt]

Managed to get it working but had to copy some code to override one of the beans:

I had to copy this function in my project and add the IdGenerator https://github.com/spring-projects-experimental/spring-cloud-sleuth-otel/blob/main/spring-cloud-sleuth-otel-autoconfigure/src/main/java/org/springframework/cloud/sleuth/autoconfig/otel/OtelAutoConfiguration.java#L91

    @Bean
    fun otelTracerProvider(spanLimits: SpanLimits, spanProcessors: ObjectProvider<MutableList<SpanProcessor>>, spanExporterCustomizer: SpanExporterCustomizer, spanExporters: ObjectProvider<List<SpanExporter>>, sampler: Sampler, resource: Resource, spanProcessorProvider: SpanProcessorProvider): SdkTracerProvider {
        val sdkTracerProviderBuilder = SdkTracerProvider.builder().setResource(resource).setSampler(sampler).setSpanLimits(spanLimits)

        val processors = spanProcessors.getIfAvailable { mutableListOf() }

        processors.addAll(spanExporters.getIfAvailable { ArrayList() }.stream()
            .map { e -> spanProcessorProvider.toSpanProcessor(spanExporterCustomizer.customize(e)) }
            .collect(Collectors.toList()))

        processors.forEach(Consumer { spanProcessor: SpanProcessor? ->
            sdkTracerProviderBuilder.addSpanProcessor(
                spanProcessor!!
            )
        })
        return sdkTracerProviderBuilder
            .setIdGenerator(object : IdGenerator {
                override fun generateSpanId() = AwsXrayIdGenerator.getInstance().generateSpanId()
                override fun generateTraceId() = AwsXrayIdGenerator.getInstance().generateTraceId()
            })
            .build()
    }

Added this dependency to the project

    implementation("io.opentelemetry.contrib:opentelemetry-aws-xray:1.21.0")

[marcingrzejszczak]

Support was there but for the deprecated AWS XRay text maps. It's enough for you to pass the propagation type to be AWS. You can read more about this here https://docs.spring.io/spring-cloud-sleuth/docs/current/reference/html/howto.html#how-to-change-context-propagation.

With this commit 4fc825b we're adding support for the contrib class from the io.opentelemetry.contrib:opentelemetry-aws-xray-propagator dependency

[JulianDoe]

It seems, that the bug is still out there. We used the version 1.1.1 which should fix it, but we run in the same problem as @sibethencourt. We used his workaround to overwirte the bean with the correct AwsXrayIdGenerator which also worked in our case.

We followed your comment @marcingrzejszczak and just pass the propagation type via the application.properties as the documentation describes it but nothing came through x-ray. Is there more to be configured to generate the correct aws ids?

[marcingrzejszczak]

Have you added the io.opentelemetry.contrib:opentelemetry-aws-xray-propagator dependency? If that's the case can you debug the CompositeTextMapPropagator class to see why AWS is not getting picked?

[JulianDoe]

Yes, I added io.opentelemetry.contrib:opentelemetry-aws-xray-propagator dependency to the project. The debug message is as follows

However, I don't know why besides AWS the other propagators were found/added, but aws is there however the traces not coming through.
Our application config yml is as follows:

spring:
  mandatory-file-encoding: UTF-8
  jackson:
    serialization:
      WRITE_DATES_AS_TIMESTAMPS: false # prevents the issue in HELP-47285, printing dates in timestamps in our response json
  main:
    allow-circular-references: true
  sleuth:
    otel:
      config:
        trace-id-ratio-based: 1
    propagation:
      type: "AWS"

[marcingrzejszczak]

Can you actually try to put a debugger to the CompositeTextMapPropagator class to see why AWS is not getting picked?

[JulianDoe]

The debug showed that the AWS is as a type configured (see the debug variables in the screenshot) and the first AWS propagator will be selected for the mapping (also see screenshot).

[marcingrzejszczak]

So it looks that the AwsXrayPropagator should be properly picked. Have you created an AwsXrayPropagator as a bean and configured it so that it follows your requirements?

[epit787]

Hi, I try to take over today for my colleague as he is ill today.
Maybe I get this wrong, but the AwsXrayPropagator has nothing which could be configured. But I tried to add the propagator as bean:

@Bean
public AwsXrayPropagator awsXrayPropagator() {
    return AwsXrayPropagator.getInstance();
}

But also with no effect. Can you elaborate on how to configure the AwsXrayPropagator?
Thank you in advance