Update to latest tracer is breaking - cannot exempt downstream service from header propagation.

[mrtristan]

Describe the bug
There was a change in how header propagation makes its way to downstream services as of the latest tracer versions. (Propagate trace tags to downstream services (horizontal propagation) (https://github.com/DataDog/dd-trace-dotnet/pull/2897)) I do not see a way to exempt certain downstream services from this feature. We have certain external services (think massive banks) that are throwing errors at the presence of unexpected headers. This same complaint was made on the JS side and implemented there. DataDog/dd-trace-js#1601

Calling this as a bug in that we were functional until updating to the latest tracer version (2.12). Just installed 2.12 to a bunch of servers that I need to go manually downgrade as I need to inventory what external services are now rejecting our requests. Took a pretty big time investment to diagnose this across a few team members.

Added clarifications:
servers were at various versions, all bumped to latest agent version of 7.37.1.1 and tracer version of 2.12.0 - errors were observed - situation was resolved by only downgrading tracer to version 2.11.0

outbound http call is executing in a netstandard2.0 class library that's running in a full framework 4.7.2 webapi app

[pierotibou]

Hello, thanks for raising this and sorry for the inconvenience it has caused.

The PR you've pointed to introduced a bug indeed. When there were no tags to propagate (and there aren't any yet), we shouldn't have added the header. Unfortunately, the setting that we have added to disable the feature falls into the same bug so there's no way with this version to disable the feature.

Our next version shall fix that and setting DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH to 0 should prevent the issue. We'll state that clearly in the release notes.

[mrtristan]

Thanks @pierotibou. I've confirmed the before/after on 2.11 vs 2.12 where the only difference in outbound headers is that empty tags header.

However, is there a mechanism by which I can stop all DD headers from latching on to this external service call? (think like a domain blacklist). That's in my mind the missing feature that most other tracing libraries support. I tried searching for the concept within DD but came up empty.

Does something similar to the JS item linked above exist in dotnet-land?

[pierotibou]

Hello,
Regarding the Feature request part of the issue, we're discussing internally if we could add this feature across the board with the same behaviour/configuration for each tracer. But indeed, it makes sense to us to add the feature. I'll update on this when we have more visibility on when we would schedule this work.

[mlstubblefield]

Another nice to have here might be being able to specify which hosts you want to have the headers propagated to. IE: if I know the server is not running datadog and isn't maintained by my organization, I probably don't want to propagate.

[andrewlock]

Thanks for the feedback @mlstubblefield, I believe that's something that's being actively considered right now, and applies to all the context propagation, not just this one🙂

[andrewlock]

@mrtristan we have just released version 2.13.0 of the tracer, which should fix this issue. I'll close this now, but feel free to re-open if you run into any problems, thanks.