-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Prototype] Distributed Tracing #7026
Comments
Thanks @Gaganjuneja the general approach sounds reasonable, few question which are not clear to me:
On more general note, we have a few core mechanism to distribute the work: actions/listeners, tasks manager and thread pools (at least). Those should be the ones taking care of establishing the initial boundaries (spans fe) which could be augmented later on. Using the |
Thanks @reta for reviewing the approach and providing the valuable feedback.
You are right, the spanID is already there but the point is we don’t want to expose the OTel spans out of the framework and keep them in a Map. So now in order to create a unique key for this map we need some unique identifier. Name should be the same for all the traces representing the same logical unit of work. Essentially this is just for the SpanMap key only.
Tracers only keep the spans once they are closed for flushing based on the type of processor like BatchSpanProcessor for some duration, etc. But It doesn't keep the full span hierarchy. Anyways the traceid is the same so that they can reconstruct the hierarchy at the data store. We need to persist the span reference somewhere. It could be either return the span to callers, SpanMap or ThreadContext. There are a couple of reasons why I chose SpanMap over others. But yes I agree there is an overhead in maintaining the SpanMap.
We can discuss the benefits vs overhead and take a call. One other possible solution could be to expose a custom wrapped spans which contains the parent span and some additional details.
I have also thought about this and tried even passing the tracer object to the classes through search context, even if you look at the sample implementation I started from TaskManager itself and SearchOperationListener. But while doing this I realized that some low level classes like ContextIndexSearcher.java on the search path don’t even have access to the SearchContext and even if we go one level till Lucene then, we will unnecessarily end up with passing the tracer object everywhere. Somehow, when I think much about adding traces the closest thing comes to my mind is logging. Yes testing is difficult for static constructs but should we be really caring about the testing span start/end (again similar to logs). I am just writing my thoughts here, we can discuss this and decide, but yes both are doable. |
Thanks @Gaganjuneja I believe we should start with as simple API as possible: provider Tracer API to fence the OTel / other impl, wrap OTel spans (we actually never need to refer to IDs directly), no need for additional maps etc. Those won't bring much value (at least initially) but incur overhead for questionable benefits (my opinion).
That's the Tracer API should abstract away but the difference is that Tracer has state (current trace / span), whereas loggers are usually stateless and could be injected as static instances. Traces and spans do not need to be exposes as is but Tracer should at least provide the ways to start new ones.
In this case we probably should provide Tracer as injectable service (like all other services).
There are a number of production ready libraries to learn from, they cover very complex async / reactive / ... paradigms so we could learn from them: |
Thanks @reta for your follow up.
I agree, to start with we can avoid keeping the map and tracer should return the wrapped span, which the caller can use to end the span. This caller will be having the span reference and we need not to maintain this in a map. Now the tracer's startTrace and endTrace method would look something like this.
Agree, tracer as mentioned above will provide the methods to start and end spans/traces.
A Tracer instance will be initialized at startup and we need to make this instance available to the classes where traces need to be added. Now, there are 2 ways.
your thoughts? |
Thanks @Gaganjuneja , sounds like a good start |
Thanks @Gaganjuneja for detailing the approach. Few questions/suggestions:
Thoughts? |
Agreed @backslasht. |
I have added some framework requirements for Span context propagation which needs to be incorporated in |
@reta I am teaming up with @Gaganjuneja on this change and have raised a draft PR. Please review when you get chance. |
Distributed tracing is defined in this RFC #6750. In the continuation of this, I prototyped the actual interface and implementation to see how it works? We mostly focused on the below aspects in this prototype and rest should be built on top of it.
Abstractions are clearly defined to hide the implementation framework like OTel so that the tracing users need not be aware about the implementation. We would be exposing the well defined simple interface to the developers and hiding the implementation. Developers can add the traces by simply adding the start trace and end trace.
Code should not be polluted. If we write a lot of code for tracing then it will be hard to find/focus on the actual logic. We will be providing the methods to start/end traces and most of the plumbing work will be taken care of by the framework.
A trace may be linked to a parent and one parent trace could have multiple children. Most of the tasks/operations run in async mode in the OpenSearch cluster. These async tasks may run on the same thread, different threads in the same thread pool, different thread pool and even on different nodes. In this case the parent context needs to be propagated to the child so that child trace and parent trace can be linked together to show the telemetry information either at parent or / as well as at child level. One classical example for this would be a search request.
Example - Let’s assume there is an OS cluster of 3 data nodes. It contains one Index with 3 shards distributed over 3 nodes (node1, node2, node3). Now we need to trace this request at multiple code points like, RestAction, QueryPhase, IndexSearcher and Fetch phase then the physical view of the query would somewhat look like.
Let’s say we are tracing all these places so we need to propagate the current parent to the following tasks/operations automatically as the current context might not be aware of the parent.
There are 2 ways this context propagation can be done.
OpenSearch ThreadContext - Opensearch custom ThreadContext provides out of the box feature to propagate the context to forked threads on the same/different threadpool or even the network calls to the other nodes. We can simply update the parent info in the thread context and it will be available to the children. ThreadContext will persist the context of immediate parent only.
1. Out of the box support.
2. Context propagation can totally be abstracted out in the framework itself.
1. Slight maintenance overhead in maintaining the span cleanup logic and updating the ThreadContext.
Opentelemetry- Opentelemetry also provides the support for context propagation but it requires an exposure of Otel code to the user. More details on context propagation through OTel can be found here [RFC] Performance metrics framework #6533 .
1. Pros
2. Cons
Recommended Approach - Both approaches have their own pros and cons but fundamentally approach-1 OpenSearch ThreadContext gives us more control and clear abstractions. Though it requires some extra code maintenance overhead but considering the pros it’s recommended.
Sample Tracing Interface ## (Not prod ready code just for understanding purposes)
Levels - Level is an OpenSearch specific concept which OpenTelemetry doesn’t support yet(OpenTelemetry github issue for Levels). It will work similar to logging levels. It allows developers to add detailed spans with appropriate tracing levels. While debugging any issue they should be able to dynamically enable the granular tracing levels for sometime. There is one strong assumption we will have to make, the level of a parent can't be higher ordered than the child so that it shouldn't get into a situation where parent span is filtered out based on the level and child still exists; it will lead to a parent child linking issue and the child will be orphaned.
Sample Implementation
Important Points of implementation.
TracerFactory
TracerFactory will help in initialising the tracer objects and return the instance so that we need to pass the tracer instance everywhere in the code. Users can simply get the instance from TracerFactory and start the instrumentation.
Sample tracing in the search flow
Snippets of instrumentation for the above example.
Sample tracing output
The text was updated successfully, but these errors were encountered: