Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OpenTracing] Fix context propagation between OpenTracing and Datadog #518

Merged

Conversation

delner
Copy link
Contributor

@delner delner commented Aug 20, 2018

When using ddtrace OpenTracer in conjunction with an app already instrumented by ruby-rack-tracer, we were seeing some disjoint traces. Spans generated by ddtrace out of the box instrumentation was not ending up parented by the OpenTracing integration parent span (in this case, Rack), resulting in multiple traces that should have appeared as one.

To fix this, the pull request does two things:

  • For distributed tracing oriented OT integrations, they pass SpanContext objects as child_of. We need to make sure the Datadog::Context object that comes from this is set on the tracer, such that the following generated Datadog span inherits the proper trace info (trace ID, parent ID.) It was not actually using this context properly before.
  • For the first OpenTracing span created, regardless of being preceded or followed by Datadog spans, it was not resolving a Datadog context (because it was trying to derive it from a non-existing OT parent span), and thus passing nil to the start_span method which would create a brand new context that would not be re-used on subsequent spans. We now make sure its passing a Datadog::Context to start_span.

@delner delner added bug Involves a bug core Involves Datadog core libraries labels Aug 20, 2018
@delner delner self-assigned this Aug 20, 2018
@delner delner requested review from pawelchcki and palazzem August 20, 2018 21:42
# effectively ends up "assigned to a scope", by virtue of being added to the
# Context. Hence, it would behave more like an active span, which is why it
# should only be here.
unless child_of || ignore_active_scope
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block is responsible for allowing OpenTracer spans to append to traces that were created by Datadog auto-instrumentation. e.g. datadog.span --> open_tracer.span.

@delner delner force-pushed the bugfix/opentracing_context_between_tracers branch from 3d223ff to fab0f89 Compare August 22, 2018 18:58
# This is mostly for the benefit of any out-of-the-box tracing from Datadog,
# such that spans generated by that tracing will be attached to the OpenTracer
# parent span.
datadog_tracer.provider.context = span.datadog_span.context
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is responsible for allowing Datadog auto-instrumentation spans to append to traces that were created by OpenTracer. e.g. open_tracer.span --> datadog.span .


private

def inherited_span_context(parent, ignore_active_scope: false)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a refactor from #start_span.

@delner delner force-pushed the bugfix/opentracing_context_between_tracers branch from fab0f89 to 0612ae0 Compare August 22, 2018 19:06
@delner
Copy link
Contributor Author

delner commented Aug 23, 2018

Made some significant changes to how context is resolved and written to. This should be ready for review.

child_of = if scope_manager.active
scope_manager.active.span.context
else
SpanContextFactory.build(datadog_context: datadog_tracer.call_context)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Contributor

@pawelchcki pawelchcki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@delner delner changed the title [OpenTracing] Fix Tracer not using child_of SpanContext properly [OpenTracing] Fix context propagation between OpenTracing and Datadog Aug 23, 2018
@delner delner added this to the 0.16.0 milestone Aug 23, 2018
@delner delner mentioned this pull request Aug 23, 2018
Copy link
Member

@Kyle-Verhoog Kyle-Verhoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, I had a nearly identical approach with the Python tracer.

@@ -204,6 +228,58 @@ def current_trace_for(object)
end
end
end

context 'preceded by a Datadog span' do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

it { expect(child_datadog_span.trace_id).to eq(parent_datadog_span.trace_id) }
end

context 'followed by a Datadog span' do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@@ -51,6 +51,30 @@ def current_trace_for(object)
end

context 'for a nested span' do
context 'when there is no active scope' do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@delner delner merged commit 287cc51 into feature/opentracing Aug 24, 2018
@delner delner deleted the bugfix/opentracing_context_between_tracers branch August 24, 2018 13:29
delner added a commit that referenced this pull request Sep 13, 2018
…#518)

* Fixed: OpenTracer::Tracer not using child_of SpanContext properly.

* Changed: OpenTracer to append its spans to Datadog auto-instrumentation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Involves a bug core Involves Datadog core libraries
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants