Trace context (traceparent) not propagated from controller to agent pods

[syn-zhu]

Summary

W3C TraceContext headers (traceparent, tracestate) are not propagated from the kagent controller to agent pods, breaking distributed tracing across the controller-agent boundary.

When an upstream service (e.g., AgentGateway) sends a request with a traceparent header to the kagent controller, the trace context is lost when the controller forwards the request to the agent pod via the A2A client. This results in disconnected traces — the upstream service's spans appear under one trace ID while the agent's spans appear under a completely different trace ID.

Root Cause

There are two independent issues:

1. Go Controller: A2A client drops incoming HTTP headers

The A2ARequestHandler in go/internal/httpserver/auth/authn.go (line 82-117) only injects X-User-Id and Authorization headers into the outgoing request to agent pods. The trpc-a2a-go A2A client constructs a new HTTP request from the JSON-RPC payload, so the original traceparent/tracestate headers from the incoming request are not carried over.

Flow where headers are lost:

Incoming HTTP request (with traceparent)
  → A2A Server (trpc-a2a-go) receives request with headers ✅
  → PassthroughManager.OnSendMessage() — only passes protocol.SendMessageParams (no HTTP headers)
  → A2AClient.SendMessage() — constructs NEW HTTP request
  → A2ARequestHandler — only adds auth headers, NOT trace headers ❌
  → Agent pod receives request WITHOUT traceparent

2. Python Agent: No W3C TraceContext propagator configured

In python/packages/kagent-core/src/kagent/core/tracing/_utils.py, the configure() function sets up FastAPIInstrumentor but does not call set_global_textmap_propagator(TraceContextTextMapPropagator()). Even if the traceparent header reached the agent pod, the Python OTEL SDK would not extract it.

Note: The OTEL_PROPAGATORS environment variable does not automatically configure the propagator in the Python OTEL SDK — explicit code is required.

Suggested Fix

Go Controller (authn.go)

Store trace headers from the incoming request in context, then inject them in A2ARequestHandler:

// In middleware.go or similar — capture trace headers into context
type traceHeadersKey struct{}

func traceHeaderMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        if tp := r.Header.Get("traceparent"); tp != "" {
            ctx = context.WithValue(ctx, traceHeadersKey{}, map[string]string{
                "traceparent": tp,
                "tracestate":  r.Header.Get("tracestate"),
            })
        }
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

// In A2ARequestHandler — inject trace headers into outgoing request
if headers, ok := ctx.Value(traceHeadersKey{}).(map[string]string); ok {
    if tp := headers["traceparent"]; tp != "" {
        req.Header.Set("traceparent", tp)
    }
    if ts := headers["tracestate"]; ts != "" {
        req.Header.Set("tracestate", ts)
    }
}

Python Agent (_utils.py)

Add propagator setup after TracerProvider configuration:

from opentelemetry.propagate import set_global_textmap_propagator
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# Inside configure(), after trace.set_tracer_provider():
set_global_textmap_propagator(
    CompositePropagator([TraceContextTextMapPropagator()])
)

Reproduction

1. Deploy kagent with tracing enabled (OTEL_TRACING_ENABLED=true)
2. Send a request with a known traceparent header:

   curl -H "traceparent: 00-aaaabbbbccccddddeeee111122223344-1234567890abcdef-01" \
     -H "Content-Type: application/json" \
     -X POST http://<gateway>/api/a2a/<namespace>/<agent>/ \
     -d '{"jsonrpc":"2.0","method":"message/send","params":{"message":{"role":"user","parts":[{"kind":"text","text":"hello"}]}},"id":"test"}'

3. Check traces in your OTEL backend — the agent spans will have a different trace ID than aaaabbbbccccddddeeee111122223344

Environment

• kagent version: latest main branch (as of 2026-02-13)
• Upstream: AgentGateway (sends traceparent headers correctly)
• OTEL backend: Langfuse (via OTEL Collector)
• Python OTEL SDK: opentelemetry-sdk
• trpc-a2a-go: used for A2A client/server in Go controller

[syn-zhu]

Update: Root Cause is Missing appProtocol Annotation on Agent Service

After further investigation, the controller being in the A2A data path is itself unintended. The kagent controller is a control plane component — it should only watch CRDs and manage agent deployments/services.

How it's supposed to work

AgentGateway's A2A plugin (a2a_plugin.go:38-68) discovers A2A backends by looking for Kubernetes Services with appProtocol: kgateway.dev/a2a. When it finds one, it routes requests directly to that Service — no controller proxy needed.

What's actually happening

In adk_api_translator.go:529-541, when kagent creates a Service for an agent pod, it does not set the appProtocol field:

Ports: []corev1.ServicePort{{
    Name:       "http",
    Port:       dep.Port,
    TargetPort: intstr.FromInt(int(dep.Port)),
    // No AppProtocol set!
}},

Because the agent Service lacks appProtocol: kgateway.dev/a2a, AgentGateway cannot discover it. Instead, requests end up routed through the controller's passthrough proxy (:8083), which creates new HTTP requests and drops trace headers.

Proposed fix

Add the appProtocol annotation to the agent Service port in adk_api_translator.go:

Ports: []corev1.ServicePort{{
    Name:        "http",
    Port:        dep.Port,
    TargetPort:  intstr.FromInt(int(dep.Port)),
    AppProtocol: ptr.To("kgateway.dev/a2a"),
}},

This single change would:

1. Enable AgentGateway to discover agent Services directly
2. Route A2A requests straight to agent pods (bypassing the controller)
3. Fix the trace disconnection (traceparent flows directly from AgentGateway to agent pod)
4. Keep the controller as a pure control plane component

The Python propagator fix (adding set_global_textmap_propagator(TraceContextTextMapPropagator()) in _utils.py) would still be needed for the agent pod to properly extract the incoming traceparent header.