AlertManager Sends alert notification when the alert in a group gets resolved

[vivek425ster]

Whenever an alert in a group gets resolved alertmanager sends a notification of an alert with the remaining alerts in a group. This notification is not of a resolved alert but it is of the remaining alerts.

For example if i have 4 alerts(A, B, C, D) firing in a group. I get a notification with 4 alerts firing. Now if an alert 'C' gets resolved. I get a notification with 3 alerts(A, B, D) firing after group_interval period. Ideally this shouldn't happen since i have already received a notification of these alerts (A, B, D) firing. We should only get a notification if any new alert is added in a group.

• Alertmanager version:
  0.12.0

• Prometheus version:
  2.0

For config file:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/prometheus-users/4nibd10rJ_0/eqKZ_1gSAQAJ

[simonpasquier]

IMO this is the intended behavior, at least this is what users currently rely on. If many people ask for it, maybe there could be an optional parameter telling AlertManager to send notifications only about new firing alerts.

Having said that, the description of the group_wait parameter could be improved to say that it encompass firing and resolved alerts.

[brian-brazil]

group_wait should have no impact on resolved alerts, as it delays the start of a group which is from firing alerts.

The OP's desired behaviour is correct though, the user already knows that ABD are firing so there's no need to tell them again before the repeat_interval.

[simonpasquier]

In that case, the issue is valid since currently the DeDup stage will pass the alerts to the Retry stage if the current set of resolved alerts (eg [D]) isn't a subset of the nflog entry (eg []). See

alertmanager/notify/notify.go

Lines 493 to 495 in 9b10aca

	if !entry.IsResolvedSubset(resolved) {
	return true, nil
	}