#3513: Add GroupMarker interface

[grobinson-grafana]

This commit adds a new GroupMarker interface that marks the status of groups. For example, whether a group is muted because or one or more active or mute time intervals. It renames the existing Marker interface to AlertMarker to avoid confusion.

It is based on #3791.

[grobinson-grafana]

In the original version of GroupMarker I had:

type GroupMarker interface {
	// Muted returns true if the group is muted, otherwise false. If the group
	// is muted then it also returns the names of the time intervals that muted
	// it.
	Muted(groupKey string, fingerprint model.Fingerprint) ([]string, bool)
	...
}

but then I realized we didn't need to store the fingerprint at all.

The reason here is that active and mute timings work against routes. That means either all alerts in an aggregation group are suppressed because of time intervals, or none of them are. It is not possible to have two alerts A and B in the same aggregation group, where A is muted from a mute time interval and B is not.

[grobinson-grafana]

We have found an issue where because groupKey is not guaranteed to be unique, it's possible for two (or more) different groups to have the same groupKey. In such cases, where one group is muted and another group is not, both will be marked as muted.

[grobinson-grafana]

The following YAML shows a configuration containing two routes with the same groupKey. This happens because the groupKey is calculated using:

1. The matchers in the route
2. The group_by labels (default is empty)

And both routes have the same matchers and group_by labels.

receivers:
  - name: test1
  - name: test2
route:
  receiver: test1
  routes:
    - receiver: test1
      matchers:
        - foo=bar
    - receiver: test2
      matchers:
        - foo=bar
      mute_time_intervals:
        - name: weekends

The reason a user might have such a configuration is to mute notifications on the weekends, but still send webhooks to an issue tracker.

[grobinson-grafana]

I added a TODO in the source code to make it clear that this is not how I want the interface to look, but fixing this is blocked due to #3817 that we will fix in another PR.

[grobinson-grafana]

I renamed this interface to AlertMarker so it could be better differentiated from GroupMarker.

[grobinson-grafana]

Since routeID will be removed in the future I have chosen to just concatenate the two strings together.

[grobinson-grafana]

I added routeID to avoid the issue of non unique group keys from causing groups to be incorrectly marked as muted. See #3817 for more information.

[gotjosh]

Overall looks good, but please see my comment.

[gotjosh]

This claims to store the state of the group but its only attribute is mutedBy is this correct?

[grobinson-grafana]

Yes! I copied it from here:

// AlertStatus stores the state of an alert and, as applicable, the IDs of
// silences silencing the alert and of other alerts inhibiting the alert.

[gotjosh]

Don't we need to aggregate all the time-intervals that this has been muted by?

[grobinson-grafana]

We don't – the reason is that the Mutes method we just merged returns them all:

alertmanager/timeinterval/timeinterval.go

Lines 37 to 53 in dacbf00

	func (i *Intervener) Mutes(names []string, now time.Time) (bool, []string, error) {
	var in []string
	for _, name := range names {
	interval, ok := i.intervals[name]
	if !ok {
	return false, nil, fmt.Errorf("time interval %s doesn't exist in config", name)
	}
	for _, ti := range interval {
	if ti.ContainsTime(now.UTC()) {
	in = append(in, name)
	}
	}
	}
	return len(in) > 0, in, nil
	}

.

[gotjosh]

How is this gargage collected? As in, how does the marker now that it no longer needs to store the status of an alert.

[grobinson-grafana]

See #3793 (comment).

[gotjosh]

Thanks.

[gotjosh]

LGTM

Thanks for clarifying my doubts!