Named tracers design/configuration questions

[lmolkova]

I was going through named tracers otep and api and sdk specs here and I still have some questions regarding named tracers API and motivation behind.
It affects some decisions we make on how to instantiate and configure tracers/factories.

@z1c0 I hope you can help me understand this.

Filtering

• why users cannot disable the instrumentation library that is not working properly? They presumably explicitly onboarded onto this library, so they can disable it? Is it only runtime filtering scenario?
• How filtering is going to be done? By SpanProcessor? Isn't it too late to filter out (especially if library crashes the app) - span is already collected and perf is already impacted. If earlier - should we create noop-tracers for bad libraries?
• There is some piece of configuration that is missing on the TraceFactory today that will configure/enable this filtering and I wonder if it affects other parts (how we instantiate factories and how they behave) and can introduce breaking changes later.

Different settings

• let's say I want different sampling/processors/etc for different libraries. Do I create multiple factories with different configurations?
• If so, do we need common registry of tracers between multiple factories? Otherwise, I need to know which factory to get tracer from.
• If it's one factory - how do I provide different samplers for different tracers

It would really be great to have some end-to-end scenario where/how named tracers are going to be used and also list of things that need to happen to make filtering and different settings feasible.

[z1c0]

Thanks for looking into the named tracers otep and spec @lmolkova.
Let me try and answer your questions:

Filtering

• This is indeed intended as a runtime setting. There might be cases where users of a library do not have access to the source code or are not allowed to (immediately) change deployed applications, when a (tracing-related) issue occurs.
• Filtering can be done on multiple levels. If a "full SDK" is implemented, this can e.g. even happen on a "Tracer" level. We (Dynatrace) are currently focusing on considering that "tracer name" information on an Exporter (SpanProcessor) level.
  It might not be possible to avoid application crashes that way. The problem this solves however, is the sending of bogus, "spam" or potentially harmful trace data to a backend system.
• This configuration is intended to be vendor SDK specific. The default SDK will not care about filtering or enabling/disabling tracers.

Different settings

• Creating multiple factories for different processors/exporters/... is the suggested way.
• There is no common tracer registry planned. Why is it required to know which factory to get a tracer from?

For general motivation of this topic, the issue on "components" might be interesting: #10
One of the main concerns of named tracers is to make sure that this "component" information is consistently provided on span data.

[lmolkova]

@z1c0 thanks for the update, it really hepls!

So, what I hear is that

• default SDK implementation does not require any filtering or different configuration for tracers.
• there is no specification/expectation/scenario that tells how filtering should be done
• vendor-specific SDK implementation has full freedom to decide and implement the configuration layer and approaches for filtering.

Is my understanding correct?

If so, it sounds like for the default SDK:

• implementing filtering does not make sense - users configure libraries explicitly anyway. it's just hard to foresee how it might be used, so it's too risky to implement esp for 1.0.

• it only makes sense to have one configuration for all tracers - multiple factories are hard to manage

  • you need to keep all processors and exporters somewhere to shut them down along with other disposables
  • if you need to obtain the same tracer more than once, you have to keep factory you got it first from (with all specific configuration)
  • there is just one global configuration, so whoever uses it, becomes exception from different configurations (unless you share registry between factories).

• it just feels like that if this feature is ever going to be part of default SDK implementation, it needs more APIs and behavior defined - now it is very loose.

So, my question is:
Today in C# we do have Factory in the SDK that supports getting different tracers. And considering all the above, I wonder why?

• This is not required feature for 1.0
• This is not the scenario that is going to be used with default SDK (filtering not possible), different configurations are very hard to use and too advanced (and there is just one global).
• It feels like more changes are coming to the spec around it, so would it make sense to only support one default tracer in the default SDK to leave more room for future improvements and API changes. It just affects how we create Factories and configure them and if this API requires change this may be breaking, So I'd like to avoid it.

[z1c0]

The specification is currently not up to date with the Tracer/TracerFactory concepts, when it comes to "shared configuration objects" (processors, exporters, ...).
However, there is an issue (and a soon-to-come PR) that adresses that:
#304

Regarding your questions:

• I don't think this is not required for 1.0. As soon as someone implements a SpanProcessor, Exporter, Propagator let alone a "full SDK", this is very much required and the only way to match Span data to the Tracer that produced it.
• I have to disagree here, since the "default SDK" (using a custom Exporter though) is a very good example where named tracers (and the resulting "component-ids" on spans) are useful, even required.
• I don't think that's a good idea, since this would render the whole idea behind named tracers useless. It's crucial that implementing SDKs handle the naming aspect correctly when tracers are created, so that components further down (Exporters, ...) can rely on having correct ResourceLibrary (name + version) information set on spans.

[lmolkova]

I see that for default SDK the value of named tracers boils down to resource stamped on spans.
Default SDK only ensures they are stamped, cannot do much about it, it's fully up to the exporter to use.

Right?

So next question, how do we envision exporters use this information? From the Application Insights side, we do have a similar concept and we use it for debugging (something went wrong, what module collected this telemtery and which version). Are there other use cases (for default SDK)?

[lmolkova]

Also, it seems specification does not require SDK to implement named tracers:

A library, implementing the OpenTelemetry API may also ignore this name and return a default instance for all calls, if it does not support "named" functionality (e.g. an implementation which is not even observability-related). A TracerFactory could also return a no-op Tracer here if application owners configure the SDK to suppress telemetry produced by this library.

Is this part of the spec should change too?

[z1c0]

I see that for default SDK the value of named tracers boils down to resource stamped on spans.

Correct: the main value of named tracers is associating resource (hence tracer) information with spans.
At Dynatrace e.g, we plan to use this information in propagators / exporters to be able to apply custom configuration (on/off, throttling, ...) to specific tracers (actually the spans produced by them) before sending those to the server.

Also, it seems specification does not require SDK to implement named tracer

Maybe the wording in the specification is too soft here. The key part though is the required attribute for the tracer's name. The sentence about SDKs being allowed to ignore this name was really intended for the mentioned non-observability-related implementations (e.g. the LoggingTracer sample we have in the .NET repository). But I see how this can be confusing. I will double down on the mandatory aspect here.

[z1c0]

@lmolkova @SergeyKanzhelev Just noticed that this is still open. Can we close it?

[lmolkova]

@z1c0 sure, thanks for the explanation!