[datadogexporter/datadogconnector] OpenTelemetry Missing Tags in APM Stats while upgrading to 0.95.0

[JM89]

Component(s)

No response

What happened?

Description

I am publishing traces from EKS Services to DataDog using otel/opentelemetry-collector-contrib:0.94.0.
I have been trying to upgrade to the latest version, but could not go beyond 0.95.0 without breaking most of our alerting system which relies heavily on APM Metrics and specific tags being present.

I went through reusing the Datadog Connector as described here and here; and “trace.Microsoft.AspNetCore.server.hits” APM metrics now appear in DataDog, but without any of the custom tags available in the APM traces. I can see a number of options in the datadog connector (e.g. peer_tags) but none worked. The tags service, env and resource_name seems to go through, but not “host” or anything custom.

Is this behavior expected? Can these tags be available the same way they used to be? Is there a better way of doing this?

Steps to Reproduce

Upgrade the OpenTelemetryCollector sidecar from 0.94.0 to 0.95.0 and reuse datadog connector as configured below. Note that this was tried with v0.113.0 as well.

Expected Result

The computed APM Metrics contain the same tags as the APM traces.

Actual Result

The computed APM Metrics contain only service, env, and resource_name. The tag "host" is set to "none" and custom tags are not available.

Collector version

v0.95.0

Environment information

Environment

• AWS EKS
• .NET Applications publishing traces.
• OpenTelemetry Collector running as sidecar.

OpenTelemetry Collector configuration

receivers:
      otlp:
        protocols:
          grpc:
          http:
processors: 
  batch:
    timeout: 10s
connectors:
    datadog/connector:
exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: ${env:DD_SITE}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog/connector]
    traces/2:
      receivers: [datadog/connector]
      processors: [batch]
      exporters: [datadog]
    metrics:
      receivers: [datadog/connector]
      processors: [batch]
      exporters: [datadog]

Log output

No response

Additional context

No response

[jackgopack4]

Can you share a sample trace with the tags applied? Also have you opened a ticket with Datadog support?

Also, there have been some improvements/fixes relating to tagging between versions 0.95.0 and the current version 0.114.0 so if you are able to try a newer version that might be helpful as well.

[JM89]

Sorry for the late reply, didn't have a chance to do more testing until today.

Can you share a sample trace with the tags applied?

What is the simplest way to share a sample trace? I need to anonymize it... However, please keep in mind that the issue does not appear on tags: the trace tags didn't change between datadog-exporter and datadog-connector for APM stats.

Also have you opened a ticket with Datadog support?

No, it was rapidly isolated to a difference in behavior between the datadog-exporter and datadog-connector so I thought this repository was the best place to ask the question. In fact, after some tests with 0.94.0 and the datadog-connector, the tags aren't there as well so it's not so much about the version, more like the tags are propagated differently between datadog-connector and datadog-exporter v0.94.0 during APM stats computation. Do you know if this is expected behavior?

Also, there have been some improvements/fixes relating to tagging between versions 0.95.0 and the current version 0.114.0 so if you can try a newer version that might be helpful as well.

Yes, I tested with 0.113.0 as mentioned in the question. I also tested with the 0.116.1, after the communication with the peer tags changes. The tags still do not appear.

---

Since the host is set to "none", is there a way to enrich at least this one manually using a processor? k8sattribute does not fit the purpose here since it's a sidecar setup. The processor attributes didn't add any tags anywhere either.

[jackgopack4]

if you can open the support ticket that will allow our engineers to view sample traces from the support tool, that's why I suggested it

[JM89]

I have now opened a DataDog support ticket. I am still very interested to understand the difference between the APM stats computations by the DataDog exporter v0.94.0 and the connector in the meantime? Especially around this host tag? Seems like this is not an expected behaviour.

[parkedwards]

Update - my issue may not be the same as the OP; I was placing the attribute on the wrong context

#36272 (comment)

---

i am also running into this issue. our attributes processor is used to set a key/value tag onto our otlp traces and statistics we send to DD

    processors:
      attributes/env:
        actions:
          - action: insert
            key: env
            value: ${env}
...
        traces:
          receivers: [otlp]
          processors: [batch/dd, attributes/env]
          exporters: [datadog/connector]
        traces/dd-with-sampling:
          receivers: [datadog/connector]
          processors: [batch/dd, attributes/env, tail_sampling]
          exporters: [datadog/external]
        metrics/dd:
          receivers: [datadog/connector]
          processors: [batch/dd, attributes/env]
          exporters: [datadog/external]

however, only our traces have the resulting env tag. the statistics do not have this tag populating. we're on -contrib:0.116.1