Datadog exporter drops trace payloads when trace_buffer is full

[songy23]

Component(s)

exporter/datadog

What happened?

See https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6627251378/job/18001822042?pr=28396#step:5:188 for an example

2023-10-24T13:10:43.382Z warn datadogexporter/zaplogger.go:43 Payload in channel full. Dropped 1 payload. {"kind": "exporter", "data_type": "metrics", "name": "datadog"}

Collector version

mainline

Environment information

No response

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[kallangerard]

@songy23 does this actually fix this issue? The main issues I'm seeing are:

• The exporter does not return any failure to the processor, so as far as the pipeline is concerned, the spans were submitted successfully
• I believe this means that there's no otelcol telemetry to show this
• why is the default behaviour unbuffered?
• What should it be set to and under what conditions? In other words how do we actually use it?

I'm not sure if this is correct behaviour, regardless of a parameter. It seems like we should receive an ExportResult failure value as per the trace spec?

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#exportbatch

[songy23]

@kallangerard Unfortunately the change is from our upstream dependency library. It is not in the collector-contrib code base. The dependency library does not return any error when traces are dropped, the only indicator is the log message Payload in channel full. Dropped 1 payload. So when you see this log message, you may consider setting trace_buffer to a higher value.

We have relayed this issue to the dependency library to change the default behavior to back pressure instead of dropping.