make inhibitions part of the gossip

[OGKevin]

tldr

If clustering is enabled, AM should be considered ready when gossip has settled, this is currently not the case. I'm wondering if this is a bug or intended behavior.

Description

I'm investigation failed inhibitions and some other alertmanager issues that we're facing, and think that waiting for gossip to settle might be an improvement that could reduce these weird behaviors when an instance is restarted.

When alertmanager starts, it immediately report itself as ready and starts serving traffic without having a complete picture of what is going on in the cluster. This can lead to unexpected behavior, such as inhibitions being missed, or duplicate alerts being send out etc.

Code Snippets

Here we can see that there is a method

alertmanager/cluster/cluster.go

Lines 667 to 672 in e252c2e

	// Settle waits until the mesh is ready (and sets the appropriate internal state when it is).
	// The idea is that we don't want to start "working" before we get a chance to know most of the alerts and/or silences.
	// Inspired from https://github.com/apache/cassandra/blob/7a40abb6a5108688fb1b10c375bb751cbb782ea4/src/java/org/apache/cassandra/gms/Gossiper.java
	// This is clearly not perfect or strictly correct but should prevent the alertmanager to send notification before it is obviously not ready.
	// This is especially important for those that do not have persistent storage.
	func (p *Peer) Settle(ctx context.Context, interval time.Duration) {

that gets called in a go routine during startup

alertmanager/cmd/alertmanager/main.go

Lines 326 to 343 in e252c2e

	// Peer state listeners have been registered, now we can join and get the initial state.
	if peer != nil {
	err = peer.Join(
	*reconnectInterval,
	*peerReconnectTimeout,
	)
	if err != nil {
	logger.Warn("unable to join gossip mesh", "err", err)
	}
	ctx, cancel := context.WithTimeout(context.Background(), *settleTimeout)
	defer func() {
	cancel()
	if err := peer.Leave(10 * time.Second); err != nil {
	logger.Warn("unable to leave gossip mesh", "err", err)
	}
	}()
	go peer.Settle(ctx, *gossipInterval*10)
	}

The cluster struct, also provides a method that reports if gossip has settled or not

alertmanager/cluster/cluster.go

Lines 582 to 590 in e252c2e

	// Return true when router has settled.
	func (p *Peer) Ready() bool {
	select {
	case <-p.readyc:
	return true
	default:
	}
	return false
	}

However, this method is not check during stratup nor during /-/ready.

Related Issues

• Alerts that should be inhibited fire on Alertmanager reload/restart #4064
• /-/ready reports ready before cluster gossip has settled #3026

[OGKevin]

After some additional code reading, it seems actually that before an alert is processed for inhibition etc, it goes through GossipSettleStage. Which ensures that gossip is settled before the alert is passed to the next pipeline.

alertmanager/notify/notify.go

Lines 525 to 532 in e252c2e

	func (n *GossipSettleStage) Exec(ctx context.Context, _ *slog.Logger, alerts ...*types.Alert) (context.Context, []*types.Alert, error) {
	if n.peer != nil {
	if err := n.peer.WaitReady(ctx); err != nil {
	return ctx, nil, err
	}
	}
	return ctx, alerts, nil
	}

[grobinson-grafana]

I'm surprised this is affecting inhibitions, as those are in the configuration file, so should be independent of gossip settle. Could the issue with inhibitions be related to something else, like a race condition between evaluated rules (see https://www.grobinson.net/best-practices-for-avoiding-race-conditions-in-inhibition-rules.html).

The GossipSettleStage sounds correct to me as well. Perhaps you could share some debug logs that show duplicate notifications and we can investigate further?

[OGKevin]

I've spend more time reading the code, and I think what we might be suffering from is the fact that Inhibitions are not gossiped to peers.

So there could be a scenario where peer0 is the only member receiving an alert that is in charge of inhibiting others alerts, while peer1 doesn't get this alert due to unimportant reasons.

Subsequently, all peers receive an alert that should be inhibited, but peer0 is the only one muting the alert as it received the inhibitor, while peer1 did not. Its also not in nflog, so peer1 sends out the notification.

I'm in the process of creating a lab to do some testing and validation/debunking of assumptions that we have.

---

The reason I started looking at gossip was due to our assumptions that cluster peers were not having a similar state. And with the combination of restarting an instance, leads to missed inhibitions, the assumption was that alertmanager is making decisions before getting enough info from its peers.

As during startup is the ideal window of a peer to not receive alerts, as it hasn't been discovered yet(added to dns discovery for example).

---

I don't have debug logs 😔, but hopefully my lab test case would produce enough tangible info.

[grobinson-grafana]

I've spend more time reading the code, and I think what we might be suffering from is the fact that Inhibitions are not gossiped to peers.

Inhibitions aren't gossiped, they're in the configuration file. Are you talking about silences by any chance?

So there could be a scenario where peer0 is the only member receiving an alert that is in charge of inhibiting others alerts, while peer1 doesn't get this alert due to unimportant reasons.

For Alertmanager to work correctly, it is essential all alerts are delivered to all replicas. If peer1 isn't getting the same set of alerts as peer0 this is a problem and will cause the behaviour you're seeing. Make sure your Prometheus server is configured to send alerts to all peers.

Subsequently, all peers receive an alert that should be inhibited, but peer0 is the only one muting the alert as it received the inhibitor, while peer1 did not. Its also not in nflog, so peer1 sends out the notification.

See above.

[OGKevin]

I know inhibitions are defined in configuration file, what I am specifically saying is, if peer 1 receives an alert that is a source matcher, it should gossip this to the rest.

inhibit_rules:
  - source_matchers:
      - alertname = "alert1"
    target_matchers:
      - alertname = "alert2"

If prometheus is sending both of these alerts, and for some reason(e.g. network timeout, reboot, unfortunate evaluation timing, glitch in the matrix) peer2 only gets alert2, peer2 will send a notification.
While if peer1's Inhibitor gossiped to the cluster that it has an alert that needs to inhibit others, then other peers have the whole picture to make decisions.

For Alertmanager to work correctly, it is essential all alerts are delivered to all replicas.

This is not realistic in high volume/big scale deployments. There will be scenarios where a request doesn't reach one of the instances due to reasons.

If a peer is going to make a decision, imo it should have all the info needed to make it or don't. Since inhibitions are used to make decisions, gossiping the fact that they've received an alert which will be used to make inhibition decisions, is a good improvement. And if they can't contact peers, decision making should be halted.

---

Or as mentioned #4064 (comment) persisting would cover the restart issue, but not the issue of missing a request, hence gossiping that there is an inhibition match might be a better solution as it covers both scenarios.

---

This is practical example:

POST http://localhost:9093/api/v2/alerts
Content-Type: Application/json
@alert1.json

POST http://localhost:9093/api/v2/alerts
Content-Type: Application/json
@alert2.json

POST http://localhost:9095/api/v2/alerts
Content-Type: Application/json
@alert2.json

This is something that could happen in a real deployment, e.g. network timeout which is configured at 30s, which means localhost:9095 never gets alert1, or at least after 30s it might get it, but then its too late, as it already sent the notification.

[grobinson-grafana]

I know inhibitions are defined in configuration file, what I am specifically saying is, if peer 1 receives an alert that is a source matcher, it should gossip this to the rest.

inhibit_rules:
  - source_matchers:
      - alertname = "alert1"
    target_matchers:
      - alertname = "alert2"

If prometheus is sending both of these alerts, and for some reason(e.g. network timeout, reboot, unfortunate evaluation timing, glitch in the matrix) peer2 only gets alert2, peer2 will send a notification. While if peer1's Inhibitor gossiped to the cluster that it has an alert that needs to inhibit others, then other peers have the whole picture to make decisions.

In general you want both the inhibiting alert and the inhibited alert to be evaluated in the same rule group in Prometheus, and the inhibiting rule to be ordered before the inhibited rule in the YAML file. This ensures that the inhibiting rule is evaluated before the inhibited rule.

If an inhibiting rule is supposed to inhibit alerts across multiple rule groups, the inhibiting rule can be duplicated for each rule group where it is needed in Prometheus and Alertmanager will deduplicate them. This helps ensure (although does not guarantee) that Prometheus sends both alerts in the same request, so either all are received, or none of them are.

For Alertmanager to work correctly, it is essential all alerts are delivered to all replicas.

This is not realistic in high volume/big scale deployments. There will be scenarios where a request doesn't reach one of the instances due to reasons.

There are reasons it works this way, I can go into them if you're interested. If you are finding that some alerts are getting lost in the network, usually the recommendation is to increase your group_wait and group_interval timers to allow for them to be retried and eventually delivered.

If a peer is going to make a decision, imo it should have all the info needed to make it or don't. Since inhibitions are used to make decisions, gossiping the fact that they've received an alert which will be used to make inhibition decisions, is a good improvement. And if they can't contact peers, decision making should be halted.

Keep in mind the gossip mechanism is not strongly consistent, so if you already have network instability, your adjacent peers may not know about the inhibition even with gossiping. Perhaps gossiping inhibitions could help somewhat, but for most use cases the advice above eliminates almost all problems with inhibitions.

[grobinson-grafana]

That all said, if you still feel that gossiping inhibitions would help eliminate some of these issues, please feel free to open a pull request for this. I would ask if you can also include some data showing before and after behavior so we can evaluate its effectiveness.

[OGKevin]

In general you want both the inhibiting alert and the inhibited alert to be evaluated in the same rule group in Prometheus, and the inhibiting rule to be ordered before the inhibited rule in the YAML file. This ensures that the inhibiting rule is evaluated before the inhibited rule.

Yea I've read this in your blog post 👍🏾, unfortunately that's not viable at our scale(So I've been told). Reasoning being that we would end up with huge groups and would need to put some engineering effort into a sane way of duplicating general inhibitions to every team that depends on it.

All in all, I've written all my findings internally and we will have a discussion and decide on next steps. ATM the options are:

• deploy small clusters closer to clients to limit network agony
• try https://grafana.com/docs/mimir/latest/references/architecture/components/alertmanager/, as it seems to replicate everything.
• run custom patch of vanilla AM with inhibition part of gossip
• A combination of the above

If 3 is picked, I would be honored to send a patch upstream!

[grobinson-grafana]

No worries! Just FYI, Mimir doesn't replicate inhibitions, so at least that bit of Mimir won't give you any benefits over vanilla Alertmanager. It does however use it's own middleware to fan out requests from Prometheus to the correct replicas, so that may help your problem, but ultimately the fanned out requests still need to go over the network.

[OGKevin]

After some more research, I think a good first step is to patch Alertmanager to precisely keep track of alert state changes and how long an alert lasted in a state.

We have some home grown analytics that looks at Alertmanager from the outside by calling the alerts api and see alert states. This tool also gets alert notifications from AM as a receiver.

We compare these 2 data points: did it receive a webhook notification from an alert that is supposed to have been suppressed by inhibition --> inhibition failed.

However this data is not accurate enough to provide such analytics. It currently contains a lot of false positives. This data in combination with user complaints lead to the creation of this issue(didn't know the data was inaccurate) 😆.

I've an idea to patch AM and add a state change aware Marker implementation and a new subscription to alerts.
Write this info in an internal (embedded) DB with a new API endpoint that returns this data.

This would allow for saner alert notification debugging. E.g. I received notification at time X, what were the sates of all participating Alertmanagers at the time for this specific fingerprint. Was one of them missing data to make the right decision? Was it misconfiguration? etc.

This information could also support the conclusion that gossiping alert states or inhibitions would improve the experience. And therefore can also be used as data to accept the patch of gossiping about inhibitions.

Now my question is, would a patch that adds analytics capability for AM be accepted? Obviously the first implementation would be Cloudflare specific, but we can then work on making it generic and fit for upstream contribution if determined that it provides value. It would be nice to know if the general idea is acceptable, to weigh our options. Do we potentially maintain a fork or go a whole different route?

I believe such feature, when enabled, could significantly help debug alerting pipeline.

[grobinson-grafana]

We compare these 2 data points: did it receive a webhook notification from an alert that is supposed to have been suppressed by inhibition --> inhibition failed.

Hmm, what are some examples of the false positives here? I would have thought that would have worked, but I might be missing something.

Now my question is, would a patch that adds analytics capability for AM be accepted?

It sounds like what you are proposing is something like audit logging? To answer the question, the proposal is too vague as it stands to give a yes/no answer.

However, I think tracing would be great. There is a PR here that we could continue with #3673. Would that be suitable for your use case?

[OGKevin]

False positives occur due to the nature of the response received when making an API call to /alerts. The API provides a snapshot of alert states at the moment the call is made. However, to make accurate comparisons, we need the state as it existed in the past—specifically, at the time the webhook was sent.

When the API is slow to respond, the data becomes less reliable because state changes may have occurred during the delay.

The tool queries the /alerts endpoint once per minute. This means that state changes occurring between scrapes or during response delays are lost. Reducing the scrape interval feels risky, as Alertmanager is already under high load.

Consider a situation where an alert transitions through states as follows:

Active → Firing → Suppressed

At time t2, we receive a webhook indicating the "Firing" state. However, by the time we scrape AM at t3, the state has already transitioned to "Suppressed." The tooling would incorrectly interpret this as a failed inhibition because it missed the intermediate transition to "Firing."

The tool writes these scrape results to an append-only table, which introduces additional challenges. Here's an example of a webhook that potentially should have been suppressed:

examining alert states for this fingerprint:

We have multiple matching entries where webhook_sent_at falls between startsAt and endsAt. Since this log is append-only, the startsAt and endsAt are not modified accordingly.

This situation could mean one of two things:

• Consensus Across AM Instances: All AM instances agreed that the state transitioned from "Suppressed" to "Firing," and a webhook was sent. However, due to stale data, the query still matches a suppressed state.
• Partial State Change: Only one AM instance lacked sufficient data to make the correct decision, resulting in inconsistent behavior.

By ensuring every state change is recorded, it becomes much easier to distinguish between scenarios:

• If all AM instances agree on a state change to "Firing," it likely indicates a user configuration error or similar issue.
• If only one or a few AM instances reflect a state change, improving gossiping mechanisms could help propagate more accurate information across instances.

Recording state changes provides historical data that eliminates reliance on timing for API calls. With this history available, we can always retrieve the necessary information to draw accurate conclusions.

---

I do not think tracing will help with keeping track of state changes, it will help in understanding why the state changed by following the path it took in the code base.

---

A simplified version of my proposal is: make a new marker implementation that records state changes in DuckDB.
The schema would look something like this:

The new marker, which will just a wrapper around memMarker for now, would write to the alert_state table:

func (s *StateAwareMarker) SetInhibited(alert model.Fingerprint, alertIDs ...string) {
	s.marker.SetInhibited(alert, alertIDs...)

	if by, isInhibited := s.marker.Inhibited(alert); isInhibited {
		s.stateAppender.AppendInhibited(alert, by)

		return
	}

	s.stateAppender.Append(alert, AlertStateActive)
}

And the new subscriber to the Alerts topic would be in charge of populating the information in the other tables.

Finally, a new API endpoint will be added to basically return the data that is in the DB. So that I can change our tooling to consume this new API instead of /alerts. So that we have more accurate data to determine if inhibition failed or not.

The way I see this in upstream AM is that this would be behind a CLI flag that you can enable, and as a user you either consume the data from the API and put it in something like clickhouse or some other externally hosted DB for analytics. Or we can build a whole new page in AM that shows some analytical stats. Or just state changes for a particular fingerprint etc.

Does this make it more concrete?