feat: add distributed tracing support #4745

siavashs · 2025-11-15T12:24:12Z

Add tracing support using otel to the the following components:

api: extract trace and span IDs from request context
provider: mem put
dispatch: split logic and use better naming
inhibit: source and target traces, mutes, etc. drop metrics
silence: query, expire, mutes
notify: add distributed tracing support to stages and all http requests

Note: inhibitor metrics are dropped since we have tracing now and they
are not needed. We have not released any version with these metrics so
we can drop them safely, this is not a breaking change.

This change borrows part of the implementation from #3673
Fixes #3670

Signed-off-by: Dave Henderson dhenderson@gmail.com
Signed-off-by: Siavash Safi siavash@cloudflare.com

TODO list

Demo

Alertmanager receiving alerts from a "sender" and tracing it all the way to an Aggregation Group inside Dispatcher

Note that Sender here is a custom Cloudflare proxy which acts as a limiter, it is included as an example to show distributed tracing in action

Trace of Dispatcher flushing alerts and notifications failing to be sent to PagerDuty

siavashs · 2025-11-15T12:43:17Z

I'll implement the Prometheus notifier changes in a new draft PR based on prometheus/prometheus#16355
So if the above PR got merged then it would be much easier to rebase the tracing changes.

thompson-tomo

Some feedback on attribute naming based on Open telemetry exposure in semconv. Might be worthwhile to add an alerting.platform.name attribute to the spans which stores alert manager.

Is there interest in having these signals defined in the open telemetry signal registry (semantic conventions)?

dispatch/dispatch.go

notify/notify.go

dispatch/dispatch.go

SuperQ · 2025-11-15T14:39:25Z

config/config.go

 	return json.Marshal(result)
 }
+
+// TODO: probably move these into prometheus/common since they're copied from


This is probably a good idea, what do you think @ArthurSens?

I can prepare a PR for common later this week.

I would not be against it :)

api/v2/api.go

dispatch/dispatch.go

notify/notify.go

dispatch/dispatch.go

thompson-tomo · 2025-11-18T02:13:08Z

notify/notify.go

+		trace.WithAttributes(attribute.String("alerting.notify.integration.name", i.name)),
+		trace.WithAttributes(attribute.Int("alerting.alerts.count", len(alerts))),
+		trace.WithSpanKind(trace.SpanKindClient),


Is there an easy way to add additional integration details potential some from https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client-span in particular server.*, note I assume this span represents the outbound call to a notification system hence the client span kind.

otelhttp generates a bunch of spans for each http request so I think it is redundant to add more info to the parent span, unless if we want to disable those and construct only one custom span:

If we are going to also have a http span then perhaps this should be internal.

thompson-tomo · 2025-11-18T02:25:34Z

dispatch/dispatch.go

+			attribute.String("alerting.alert.name", alert.Name()),
+			attribute.String("alerting.alert.fingerprint", alert.Fingerprint().String()),
+		),
+		trace.WithSpanKind(trace.SpanKindConsumer),


check this one

Suggested change

trace.WithSpanKind(trace.SpanKindConsumer),

trace.WithSpanKind(trace.SpanKindInternal),

Producer is here https://github.com/prometheus/alertmanager/pull/4745/files#diff-dfa3fcb58b56a2b2a5e995bf37d4f4cd18e64db2835e51a2b077fdc8ec26e7b4R253

Missed that one, are these outgoing calls as per the test mentioned in #4745 (comment)

No, it is this case:

deferred execution (PRODUCER and CONSUMER spans).

Yes but it still needs to be out going based on other information in the spec. Anyway I have raised open-telemetry/opentelemetry-specification#4758 to get further clarification.

siavashs · 2025-11-21T16:11:02Z

CI failures will be fixed after #4761 is merged.

OGKevin

Some minor ✨comments

provider/provider.go

examples/webhook/echo.go

tracing/tracing.go

Add tracing support using otel to the the following components: - api: extract trace and span IDs from request context - provider: mem put - dispatch: split logic and use better naming - inhibit: source and target traces, mutes, etc. drop metrics - silence: query, expire, mutes - notify: add distributed tracing support to stages and all http requests Note: inhibitor metrics are dropped since we have tracing now and they are not needed. We have not released any version with these metrics so we can drop them safely, this is not a breaking change. This change borrows part of the implementation from prometheus#3673 Fixes prometheus#3670 Signed-off-by: Dave Henderson <dhenderson@gmail.com> Signed-off-by: Siavash Safi <siavash@cloudflare.com>

siavashs force-pushed the feat/tracing branch 2 times, most recently from 178f982 to 22bfc0c Compare November 15, 2025 12:34

siavashs added component/notify component/api component/dispatch labels Nov 15, 2025

siavashs self-assigned this Nov 15, 2025

siavashs force-pushed the feat/tracing branch from 22bfc0c to c1d9a3c Compare November 15, 2025 12:37

siavashs mentioned this pull request Nov 15, 2025

Instrument with OTel tracing #3673

Closed

thompson-tomo reviewed Nov 15, 2025

View reviewed changes

SuperQ reviewed Nov 15, 2025

View reviewed changes

siavashs force-pushed the feat/tracing branch from c1d9a3c to 7b93382 Compare November 15, 2025 15:39

thompson-tomo reviewed Nov 16, 2025

View reviewed changes

siavashs force-pushed the feat/tracing branch 4 times, most recently from 14b79bd to 47747ad Compare November 17, 2025 21:16

thompson-tomo reviewed Nov 18, 2025

View reviewed changes

siavashs force-pushed the feat/tracing branch 9 times, most recently from c1482c7 to 5882778 Compare November 21, 2025 16:00

siavashs marked this pull request as ready for review November 21, 2025 16:28

siavashs requested a review from SuperQ November 21, 2025 16:31

siavashs requested review from Spaceman1701, grobinson-grafana, thompson-tomo and ultrotter November 21, 2025 16:32

siavashs force-pushed the feat/tracing branch 3 times, most recently from 6ba9be4 to 37231f9 Compare November 22, 2025 12:17

siavashs mentioned this pull request Nov 23, 2025

Add AlertLifeCycleObserver that allows consumers to hook into Alert life cycle #3461

Open

OGKevin reviewed Nov 25, 2025

View reviewed changes

provider/provider.go Show resolved Hide resolved

examples/webhook/echo.go Outdated Show resolved Hide resolved

tracing/tracing.go Show resolved Hide resolved

siavashs force-pushed the feat/tracing branch from 37231f9 to bea0dba Compare November 26, 2025 10:07

siavashs force-pushed the feat/tracing branch from bea0dba to b65bdbf Compare November 27, 2025 08:37

	trace.WithSpanKind(trace.SpanKindConsumer),
	trace.WithSpanKind(trace.SpanKindInternal),

feat: add distributed tracing support #4745

Are you sure you want to change the base?

feat: add distributed tracing support #4745

Conversation

siavashs commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO list

Demo

Alertmanager receiving alerts from a "sender" and tracing it all the way to an Aggregation Group inside Dispatcher

Trace of Dispatcher flushing alerts and notifications failing to be sent to PagerDuty

Uh oh!

siavashs commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thompson-tomo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

siavashs Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

siavashs commented Nov 21, 2025

Uh oh!

OGKevin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

siavashs commented Nov 15, 2025 •

edited

Loading

siavashs commented Nov 15, 2025 •

edited

Loading

siavashs Nov 18, 2025 •

edited

Loading