OpenTelemetry Tracing Request Manager

[hannahhoward]

What

We want to start adding basic tracing into graphsync. The request manager is simpler cause each request starts with a context. Our main challenge is setting up tracing across go routines.

How open telemetry works

OpenTelemetry is an API for tracer that can be backed by one or more actual libraries to export data to a tracing service. Fortunately, in Graphsync, we are NOT concerned with configuring data export -- that is left to the consumer of go-graphsync. However, we would like, if someone sets up a program with open telemetry tracing configured, for graphsync to provide useful trace information within the context whatever tracing the calling program has setup.

You can find a basic OpenTelemtry example here

We'll tackle the RequestManager first.

Requirements

First, we need to create a tracer at the top level in the main implementation at start time and pass it down to the request manager.

The call we use is documented here -- we can simply name our instrumentation "graphsync".

Now, we want to properly instrument a request.

• On the client side, we should start a span inside the call to Request using the passed in context
  (https://pkg.go.dev/go.opentelemetry.io/otel/trace#Tracer)
• make sure to defer span.End()
• Extract the span context for this span (https://pkg.go.dev/go.opentelemetry.io/otel/trace#SpanContextFromContext)
• Pass the span context (optionally marshal to JSON) in newRequestMessage (https://github.com/ipfs/go-graphsync/blob/main/requestmanager/messages.go#L85)
• Pass the span context to newRequest in the server (https://github.com/ipfs/go-graphsync/blob/main/requestmanager/server.go#L52)
  • after request ID is assigned, create a context from the request manager context that has the span context (https://pkg.go.dev/go.opentelemetry.io/otel/trace#ContextWithSpanContext)
  • create a new span, passing the context just created, request.ID & other peer in creation of span for WithAttributes
  • generate the local request context from the created context so you have span information (https://github.com/ipfs/go-graphsync/blob/main/requestmanager/server.go#L74)
  • add the span to the inProgressRequestStatus (https://github.com/ipfs/go-graphsync/blob/main/requestmanager/server.go#L75)
• In requestTask (https://github.com/ipfs/go-graphsync/blob/main/requestmanager/server.go#L96):
  • When generating the traversal context (https://github.com/ipfs/go-graphsync/blob/main/requestmanager/server.go#L116) first create a context from request manager context that has the span context of the in progress request span -- use ContextWithSpan (https://pkg.go.dev/go.opentelemetry.io/otel/trace#ContextWithSpan)
• In terminateRequest, end the span for the request

[hannahhoward]

Note: this is a basic setup just to insure we have two traces inside and outside the go-routine. We will likely add more tracing to the request manager later