Display "natural" grouping (and make groups linkable)

[beorn7]

If I understand correctly, the new UI only allows custom grouping by the labels given in the Group tab. This implies that all alerts will be grouped by the same grouping label set.
The problem with that is that the alertmanager config allows different groupings for different alerts. That's the grouping that is relevant for sending out actual notifications. But this grouping is neither available in the UI, nor can it be re-created.
For a proper link-back from a notification to the alert group in AM, we need that grouping anyway.

If I'm missing any already existing UI element/feature, please let me know.

[beorn7]

After talking to @w0rm , some conclusions:

The arbitrary grouping in the UI is neat if you are in a position where you want to review the totality of alerts. However, as the person on-call for something, you really want to see the one alert group that has just sent you a page. Or you want to see all the alert groups currently sending you alerts (which could be non-paging alerts). As described in #800 , filtering by receiver helps a lot, but it is really confusing if you cannot easily group the alerts in the same way as it is relevant for alerting. In any sufficiently large organization, you'll have very different groupings present, so even if you know the groupings of your alert, you couldn't even reproduce them on the same page right now because the UI only allows the same groupings for all displayed alert.

My suggestion:

In the grouping tab, offer an option Configured notification grouping. If you select that, the grouping by the given label will be grayed out, and the grouping displayed will be the one resulting from the configured routing tree. (Note that this will include duplication of alerts that are in multiple notification groups.) To implement that, each alert needs to also contain a field "groupIDs" with unique IDs of the notification groups it belongs to. This will allow the UI to group.

In addition, it would be good to be able to directly query alerts by groupID. In that way, we could finally link directly to the notification group that caused the notification in question. (This is something that never worked properly in Alertmanager, but was always highly sought after. Workflow: You get paged. You find a link in your page that leads you directly to the Alertmanager UI with that notification group open, which might have accumulated more alerts in the meantime.)

[beorn7]

@stuartnelson3 FYI.

[beorn7]

Discussed with @w0rm and @stuartnelson3 in person, once more.

The above essentially stands, plus the additional difficulty that we want to display the grouping labels in blue. Those can be different for each group. Idea so far is to have an additional endpoint where you query a groupID and get back the grouping labels for that ID. Other implementations are possible, of course, including returning the grouping labels with each alert already. (The suggested additional field groupIDs would then become a map of n groupIDs, each mapping to m grouping labels.)

[beorn7]

Another aspect to consider: An alert group (IIUC) stays around even if the last alert in it has ceased firing (so that it can accept new alerts if they show up within the group_interval (?)). Since those "empty" groups have an impact on notification behavior, it would be good to make them visible in the UI. I believe the "configured grouping" view asked for in this issue could also include that.

[brian-brazil]

Those should only be kept around until the end of the current group_interval. After that we'd be waiting for the group_wait again. Given that we currently only update the UI on the end of an interval, I would expect that we're currently doing the right thing here. When that changes then this would become a consideration.

[beorn7]

That's not how it works precisely. But let's discuss this in detail if somebody is concretely working on this.

Update after re-reading this: What @brian-brazil said above is most likely precisely how it works. But we really have to change the UI updating anyway.

[beorn7]

@stuartnelson3 based on recent in-person discussions, perhaps we should give this more than priority/Pmaybe? (And of course, ideally we could make this a priority at SC, too…)