-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[datadogexporter/datadogconnector] OpenTelemetry Missing Tags in APM Stats while upgrading to 0.95.0 #36329
Comments
Can you share a sample trace with the tags applied? Also have you opened a ticket with Datadog support? Also, there have been some improvements/fixes relating to tagging between versions 0.95.0 and the current version 0.114.0 so if you are able to try a newer version that might be helpful as well. |
Sorry for the late reply, didn't have a chance to do more testing until today.
What is the simplest way to share a sample trace? I need to anonymize it... However, please keep in mind that the issue does not appear on tags: the trace tags didn't change between
No, it was rapidly isolated to a difference in behavior between the
Yes, I tested with 0.113.0 as mentioned in the question. I also tested with the 0.116.1, after the communication with the peer tags changes. The tags still do not appear. Since the host is set to "none", is there a way to enrich at least this one manually using a processor? k8sattribute does not fit the purpose here since it's a sidecar setup. The processor |
if you can open the support ticket that will allow our engineers to view sample traces from the support tool, that's why I suggested it |
I have now opened a DataDog support ticket. I am still very interested to understand the difference between the APM stats computations by the DataDog exporter v0.94.0 and the connector in the meantime? Especially around this host tag? Seems like this is not an expected behaviour. |
Update - my issue may not be the same as the OP; I was placing the attribute on the wrong context i am also running into this issue. our
however, only our traces have the resulting |
Pinging code owners for exporter/datadog: @mx-psi @dineshg13 @liustanley @songy23 @mackjmr @ankitpatel96 @jade-guiton-dd. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label. |
Pinging code owners for connector/datadog: @mx-psi @dineshg13 @ankitpatel96 @jade-guiton-dd. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label. |
@parkedwards The reason why the attributes processor is not added to your trace stats is that the OTLP metrics which are exported via the datadog connector are not the trace stats, but contain a tag which contains the serialised trace stats payload. If you want the env tag on your stats computed in dd connector, you will need to set resource attribute |
Hi, Unfortunately, the support ticket does not help much to progress on this issue. I would deeply appreciate any technical assistance with this configuration as well as some context on the expected behavior. I have done a few more tests, and so far, I have managed to find a way to get the cluster name and host back by trial and error, but no custom metrics go through for the trace metrics apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: opentelemetry-collector
labels:
kustomize.toolkit.fluxcd.io/substitute: disabled
spec:
mode: sidecar
image: docker/otel/opentelemetry-collector-contrib:0.117.0
resources:
requests:
cpu: 10m
memory: 46Mi
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: vault-secrets
key: datadog-api-key
- name: DD_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: DD_CLUSTER_NAME
value: cluster
- name: DD_TEAM
value: team_x
- name: DD_HOSTNAME
value: $(DD_NODE_NAME)-$(DD_CLUSTER_NAME)
config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
resourcedetection:
detectors: [system, eks]
timeout: 15s
override: false
eks:
resource_attributes: # No effect noticed
k8s.cluster.name:
enabled: true
system:
resource_attributes: # No effect noticed
host.name:
enabled: true
host.id:
enabled: false
batch:
timeout: 10s
resource/custom:
attributes: # No effect noticed
- action: insert
key: cluster_name
value: coming_from_resource
- action: insert
key: test_two
value: coming_from_resource
- action: insert
key: k8s.cluster.name
value: coming_from_resource
- action: insert
key: host.ip
value: coming_from_resource
- action: insert
key: host.name
value: coming_from_resource
attributes/custom:
actions:
- action: insert
key: test
value: coming_from_attr_custom # This does not appears in trace.Microsoft.AspNetCore.server.hits but appears in otlp custom metrics and traces + aspire
- action: insert
key: topicname
value: coming_from_attr_custom # This does not appears in trace.Microsoft.AspNetCore.server.hits but appears in otlp custom metrics and traces + aspire
transform:
metric_statements: &statements
- context: resource
statements:
- set(attributes["datadog.host.name"], "coming_from_transform") # This appears in trace.Microsoft.AspNetCore.server.hits
- set(attributes["kube_cluster_name"], "coming_from_transform") # This does not appear in trace.Microsoft.AspNetCore.server.hits but appears in otlp custom metrics and traces + aspire
- set(attributes["k8s.cluster.name"], "coming_from_transform_two") # This appears in trace.Microsoft.AspNetCore.server.hits under cluster_name
- set(attributes["cluster_name"], "coming_from_transform") # This does not appear in trace.Microsoft.AspNetCore.server.hits but appears in otlp custom metrics and traces + aspire
- set(attributes["datadog.host.use_as_metadata"], true)
- set(attributes["datadog.test_three"], "coming_from_transform") # This does not appear in trace.Microsoft.AspNetCore.server.hits but appears in otlp custom metrics and traces + aspire
- set(attributes["team"], "coming_from_transform") # This does not appear in trace.Microsoft.AspNetCore.server.hits but appears in otlp custom metrics and traces + aspire
- set(attributes["datadog.team"], "coming_from_transform_two") # This does not appear in trace.Microsoft.AspNetCore.server.hits but appears in otlp custom metrics and traces + aspire
- set(attributes["topicname"], "coming_from_transform") # This does not appear in trace.Microsoft.AspNetCore.server.hits but appears in otlp custom metrics and traces + aspire
- set(attributes["service.name"], "coming_from_transform") # This appeared before in trace.Microsoft.AspNetCore.server.hits under `service`, and can be overriden
trace_statements: *statements # Use the same statements as in metrics
log_statements: *statements # Use the same statements as in metrics
exporters:
datadog/exporter:
api:
key: "${API_KEY}"
site: datadoghq.eu
metrics:
resource_attributes_as_tags: true
otlp/aspire:
endpoint: aspire-otlp:18889
tls:
insecure: true
connectors:
datadog/connector:
traces:
peer_tags: ["cluster_name", "test", "test_two", "k8s.cluster.name", "host.ip", "host.name"] # No effect noticed
resource_attributes_as_container_tags: ["cloud.availability_zone", "cloud.region"] # No effect noticed
compute_stats_by_span_kind: true # No effect noticed
service:
pipelines:
traces:
receivers: [otlp]
processors: [resourcedetection, transform, attributes/custom, resource/custom, batch]
exporters: [datadog/connector, datadog/exporter, otlp/aspire]
metrics:
receivers: [datadog/connector, otlp]
processors: [resourcedetection, transform, attributes/custom, resource/custom, batch]
exporters: [datadog/exporter, otlp/aspire] I can see the documentation suggests that span tags should not be there, where can I get the full list of tags available by default? are the resource tags also subject to this explicit "filtering"? can this behavior be overridden in any way? Reference doc:
Thank you very much, PS: parkedwards Based on what I have seen so far, have you tried the transform processor yet? processors:
transform:
metric_statements: &statements
- context: resource
statements:
- set(attributes["deployment.environment.name"], "new env")
trace_statements: *statements # Use the same statements as in metrics
log_statements: *statements # Use the same statements as in metrics
service:
pipelines:
traces:
receivers: [otlp]
processors: [transform, batch]
exporters: [datadog/connector, datadog]
metrics:
receivers: [datadog/connector, otlp]
processors: [transform, batch]
exporters: [datadog] |
so there are a few things happening here.
As such, throwing resource/attributes/transform processors at the metrics won't affect the APM stats/calculated metrics, and even throwing them at the traces coming in won't add the tags to the calculated metrics unless they are one of the types of tags mentioned in the doc. Which brings me back to the "second primary tag." There is currently a method to do this, call it a "beta" or "workaround," but we can set either If you follow the steps here and here, and set the Sorry this has taken so long to resolve; hopefully this brings you and anyone else dealing with this issue closer to resolution. |
Hi @jackgopack4, Thank you very much for the context and your detailed explanations. In summary, there is currently no way to pass custom tags in the APM Stats Payload, which means we need to migrate all live DD monitors based on APM metrics with custom tags before we can upgrade to the newer version of the OpenTelemetry collector. If this conclusion is exact, I'll suggest an update of the documentation in the support ticket to cover the difference in tagging logic here: https://docs.datadoghq.com/opentelemetry/guide/migration/ if that's okay. Out of curiosity for number 3, is this the name of the "throwaway" OTLP metric Best, |
Correct, there is a workaround to pass a second primary tag as detailed above but only a singular tag.
That's a good point, it could use some clarification. I am not certain the previous functionality was actually intentional, but we will make sure to clarify it.
Correct, although that could always be subject to change down the road. |
Thank you very much for your help, |
Component(s)
No response
What happened?
Description
I am publishing traces from EKS Services to DataDog using
otel/opentelemetry-collector-contrib:0.94.0
.I have been trying to upgrade to the latest version, but could not go beyond 0.95.0 without breaking most of our alerting system which relies heavily on APM Metrics and specific tags being present.
I went through reusing the Datadog Connector as described here and here; and “trace.Microsoft.AspNetCore.server.hits” APM metrics now appear in DataDog, but without any of the custom tags available in the APM traces. I can see a number of options in the datadog connector (e.g. peer_tags) but none worked. The tags service, env and resource_name seems to go through, but not “host” or anything custom.
Is this behavior expected? Can these tags be available the same way they used to be? Is there a better way of doing this?
Steps to Reproduce
Upgrade the OpenTelemetryCollector sidecar from 0.94.0 to 0.95.0 and reuse datadog connector as configured below. Note that this was tried with v0.113.0 as well.
Expected Result
The computed APM Metrics contain the same tags as the APM traces.
Actual Result
The computed APM Metrics contain only service, env, and resource_name. The tag "host" is set to "none" and custom tags are not available.
Collector version
v0.95.0
Environment information
Environment
OpenTelemetry Collector configuration
Log output
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: