Custom Traces + Generated spans from integration

[willyyang]

Is it possible to use the generated spans, from the integrations for GraphQL, as parent / child spans for other custom traces ? in the graphql resolvers, I make gRPC calls out to a golang server.. I want to see the entire trace.

flame graph from Hapi GraphQL plugin integration.

[rochdev]

This is definitely possible, and should happen automatically for any supported module.

To do it manually, you have to use the scope manager to get the scope and then add your own instrumentation.

In your case for example, you would have to grab the span from the active scope in the resolver, and then add a span for the gRPC client, and propagate the context to the next service.

function resolve (obj, args) {
  const scope = tracer.scopeManager().active()
  const childOf = scope.span()
  const span = tracer.startSpan('grpc.request', { childOf })
}

Since HTTP2 is not supported at the moment out of the box, you would also have to manually propagate the context using tracer.inject() in the client and tracer.extract() in the server.

[rochdev]

Created #311 and #312 to support this out of the box.

[rochdev]

Support for gRPC has landed in #311.

For completeness, here is how to do this manually with the new tracer.trace method.

function resolve (obj, args) {
  // tracer.trace automatically handles the span lifecycle, propagation, and errors
  tracer.trace('grpc.request', () => {
    // your code
  })
}

[joneshf-cn]

We're trying to do something like the above, but it doesn't seem to connect the Spans. In particular, we use the mysql package, and it's automatically instrumented by the dd-trace package. The Spans of our application are making it to DataDog. The Spans of the mysql package are making it to DataDog. But, neither Spans connect to the same Trace.

This is the function we use to generate a span:

function startSpan(tracer, name) {
  const scope = tracer.scope();
  const span = scope.active();
  if (span != null) {
    return tracer.startSpan(
      name,
      {
        childOf: span.context(),
      }
    );
  }

  return tracer.startSpan(name);
}

And we end up with Spans for service: foo and service: foo-mysql being reported in completely separate Traces. Any idea what we might be doing wrong?

[rochdev]

@joneshf-cn Can you share the code using this function? Is it only your custom spans that are not connected or is it also auto-instrumentation?

[joneshf-cn]

Can you share the code using this function?

Unfortunately, we can't share the actual code. It's some closed source code. It's also a little bit more complicated since we're calling this from PureScript and there's no DataDog package for PureScript, so we have to interact with it through the PureScript FFI. I'm about 80% sure there's something funky going on with our PureScript FFI.

I can try to make as close a reproduction as possible in plain JavaScript though. Hopefully that can fill in the missing pieces.

We have some functions to work with dd-trace:

// file: datadog.js
const ddTrace = require("dd-trace");

exports.finish = function(span) {
  span.finish();
};

exports.initTracer = function(options) {
  return ddTrace.init(options);
};

exports.startSpan = function(tracer, name) {
  const scope = tracer.scope();
  const span = scope.active();
  if (span != null) {
    return tracer.startSpan(
      name,
      {
        childOf: span.context(),
      }
    );
  }

  return tracer.startSpan(name);
};

We have some kind of router that handles endpoints:

// file: router.js
exports.handle = function(tracer, endpoint) {
  const span = datadog.startSpan(tracer, endpoint);

  // hit MySQL, hit other Services, etc.

  datadog.finish(span);
};

We initialize the tracer in our main:

// file: main.js
const datadog = require("./datadog.js");
const router = require("./router.js");

function main() {
  const tracer = datadog.initTracer({
    service: "foo",
  });

  // Other initialization

  router.handle(tracer, "/index.html")
}

main();

Hopefully that kind of explains how we're using dd-trace. As mentioned, it's a bit more complicated due to both needing to wrap it in a PureScript FFI, and how our application has grown over the years, but that's the gist of it. Does anything jump out as obvious mistakes?

Is it only your custom spans that are not connected or is it also auto-instrumentation?

I'm not entirely sure if it's only custom Spans, only something with the mysql plugin, or something with both. We get the automatic instrumentation of dns and tcp, so we end up with a single Trace that has Spans for service: foo-dns, and service: foo-tcp. But we don't get anything else on those Traces.

More concretely, when we make a request to our foo service, we get Traces for the following DataDog Services:

• foo-dns
• foo-mysql
• foo-tcp
• foo

The foo-dns and foo-tcp Services always seem to connect each others Spans through a single Trace. The foo-mysql and foo Services never seem to connect to each other, nor to either of foo-dns or foo-tcp.

[joneshf-cn]

I just re-read the getting started instructions. The second step might be the thing we're doing not quite right:

2. Import and initialize the tracer either in code or via command line arguments. The Node.js tracing library needs to be imported and initialized before any other module.

Once you have completed setup, if you are not receiving complete traces, including missing URL routes for web requests, or disconnected or missing spans, confirm step 2 has been correctly done. The tracing library being initialized first is necessary for the tracer to properly patch all of the required libraries for automatic instrumentation.

When using a transpiler such as TypeScript, Webpack, Babel, or others, import and initialize the tracer library in an external file and then import that file as a whole when building your application.

I think maybe we're not actually initializing before anything else in the actual code. In particular, this part:

When using a transpiler such as TypeScript, Webpack, Babel, or others, import and initialize the tracer library in an external file and then import that file as a whole when building your application.

I'll see if we can't move things around so we end up doing that initialization first.

[joneshf-cn]

Yep. That was it! We now have

• service: foo
• service: foo-dns
• service: foo-mysql
• service: foo-tcp

Spans all connected to the same Trace.

We even got a new set of packages instrumented!

• service: foo-fs
• service: foo-http-client

Sorry for bringing up this old Issue.