Distributor always initialises new trace when using http

[manohar-koukuntla]

Describe the Issue

I am sending request to mimir distributor using http, I am generating traceid in my application and passing it along the request, but I am unable to get the whole trace with the traceid generated by my application.

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (SHA or version) -> 2.6.0
2. Perform Operations(Read/Write/Others) -> Read

Expected behavior

I should be able to see the whole trace with the traceId generated by my application

Environment

• Infrastructure: Kubenetes
• Deployment tool: Jsonnet

Additional Context

here is my tracing setup

package observability

import (
	"context"
	"app/config"
	"app/log"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/jaeger"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
	"go.opentelemetry.io/otel/trace"
)

func InitTracer(ctx context.Context, cfg *config.Config) (trace.Tracer, func(), error) {
	exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(cfg.UpstreamConfig.JaegerCollectorAddress)))
	if err != nil {
		return nil, nil, err
	}

	tracerProvider := sdktrace.NewTracerProvider(
		sdktrace.WithSampler(sdktrace.TraceIDRatioBased(cfg.UpstreamConfig.JaegerTraceSampleRatio)),
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(newResource()),
	)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
	otel.SetTracerProvider(tracerProvider)
	tracer := otel.Tracer(config.AppName)
	return tracer, func() {
		if err := tracerProvider.Shutdown(ctx); err != nil {
			log.Error("stopping tracer provider: %s", err)
		}
	}, nil
}

func newResource() *resource.Resource {
	return resource.NewWithAttributes(
		semconv.SchemaURL,
		semconv.ServiceNameKey.String(config.AppName),
		semconv.ServiceVersionKey.String("0.0.1"),
	)
}

This the part where I am making http request to distributor

func (c *UpstreamClient) sendProtobuf(ctx context.Context, url string, data []byte, compression util.Compression) (*http.Response, error) {
	log.Debug("sending POST request to %s using Content-Type %s", url, receivers.ApplicationProtobuf)
	buf, err := util.CompressRequest(compression, data)
	if err != nil {
		log.Error("error compressing with gzip")
		return nil, err
	}
	ctx, span := otel.Tracer(config.AppName).Start(ctx, url, trace.WithSpanKind(trace.SpanKindClient))
	defer span.End()

	clientTrace := otelhttptrace.NewClientTrace(ctx, otelhttptrace.WithoutSubSpans(), otelhttptrace.WithoutHeaders())
	ctx = httptrace.WithClientTrace(ctx, clientTrace)
	req, _ := http.NewRequestWithContext(ctx, "POST", url, buf)

	req.Header[receivers.ContentTypeHeader] = []string{receivers.ApplicationProtobuf}
	switch compression {
	case util.Snappy:
		req.Header[receivers.ContentEncHeader] = []string{"snappy"}
	case util.Gzip:
		req.Header[receivers.ContentEncHeader] = []string{"gzip"}
	}
	return c.client.Do(req)
}

[dimitarvdimitrov]

Have you seen this page on configuring tracing for mimir? https://grafana.com/docs/mimir/latest/operators-guide/configure/configure-tracing/

[manohar-koukuntla]

Yeah, I have enabled tracing for mimir, both my app and mimir use same grafana agent service.