40x & 50x Emissary Traces not categorised as ERRORS in Datadog

[pkalantri-ebsco]

Describe the bug
We have 2 emissary ingresses running in our EKS cluster.
They are successfully integrated with a Datadog Agent pushing traces to our DataDog Backend.

We are now receiving the traces/spans in DataDog but their statuses are not being classified correctly - everything is classified as "OK".
According to Datadog support, this is an issue with how the Ambassador team sends the attributes with their traces.
This is the response we have gotten from DataDog Support :

...
It looks like the root issue is that Ambassador has made a mistake when implementing our tracing library that is causing error spans to not be properly marked. 
To help you if you're interested in filing a bug report with Ambassador, my reasoning is below:
 
Currently, spans submitted by the Ambassador are being marked as errors, but are not following our conventions for how to mark the errors. 
Spans from Ambassador are setting the span attribute @error to "true", where we expect the span tag status to be set to error and the error span tag to be set to 1 (you can see the difference between span tags and attributes [here](https://docs.datadoghq.com/tracing/trace_collection/tracing_naming_convention/#span-tags), to clarify further). 
To fix this on their end, they can use the set_error method to automatically follow our conventions for flagging errors which you can see in our documentation on this here.
...

Links sent by DataDog:
https://docs.datadoghq.com/tracing/trace_collection/tracing_naming_convention/#span-tags
https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/ruby/dd-api/#setting-errors-on-a-span

To Reproduce
Steps to reproduce the behavior:

1. Install Emissary - version below
2. Install Datadog Agent - version below
3. Connect traces to Datadog Backend via DD Agent
4. Check Traces

Expected behavior
We are supposed to see traces with HTTP error code 50x & 40x as errors. Instead they are being ingested as "OK" similar to 200 status codes.

Versions (please complete the following information):

• Ambassador Emissary: v8.9.1
• Datadog Agent: v3.70.1
• Kubernetes environment - EKS v1.30

Please help. Thanks!

[umarizulkifli]

emissary-ingress 3.9.1 is on envoy v1.27.2 which have this issue
tested using emissary-ingress 3.10.0-rc.0 which on envoy v1.30.3 fixed the issue

[pkalantri-ebsco]

I see the last General Release has been Nov 20th 2023.
Do we know if there is any plan to make 3.10.0-rc.0 as a GR? Any timeline on that as we do not want to use RC in PROD.

[kflynn]

There is indeed a plan to make a proper 3.10.0 GA, with a goal of that happening this month.