Notification Escalation / Delay Notifications

[aslmx]

⚠️ Please verify that this feature request has NOT been suggested before.

• I checked and didn't find similar feature request

🏷️ Feature Request Type

Other

🔖 Feature description

We're using mattermost notificaiton right now in parallel to eMail notification.
I was wondering whether it would be nice to have the possibility to first notify via non intrusive means (like mattermost channel) and if the outage is not resolved in a specified time, to escalate the notification to some more intrusive means, like SMS or Call (whatever).

This could be useful for groups of people, like in a weekend Oncall scenario where one person is on duty to keep systems up.
If something breaks, only he will be woken up in the night.
But think about the Bus factor?
If that person does not react (for whatever reason), you might want to wake additional people or escalate the matter.

It could also just be something uncritical and during the day you have Mattermost opened, but you don't monitor mattermost on the weekends...

There is many ways this can be used.

✔️ Solution

For this we would need UI features to be able to specify multiple notifications (thats already there) but for each monitor (or group of monitory) we would need to be able to specify per notification by how long it should be delayed.

I.e.

Notification 1
type Mattermost
delay: 0

Notification 2
type eMail
delay: 15

Notification 3
type SMS / Call
delay: 30

Notification 3
type eMail to Managers (or similar ;)) 
delay 60

❓ Alternatives

Maybe there is an even better way to design this. The main purpose of this request is to have the possibility of multiple notifications channel, but don't fire them at the same time.

📝 Additional Context

No response

[ablizno]

Related to this, I think we need an option to delay notifications in general. What if you want the service marked as down in the dashboard for historic reasons, so you can see if a service is constantly going down for say 1min. But don't want to be notified unless something is down for say 10 minutes.

[Mellowlynx]

+1, For a general delay option.
Extra point for ability to set delay according to a tag or something like that.
So a PRIO tag can be a direct notification, while a NORM tag will generate the notification only after 30+ of down time.

[davinkevin]

+1 on this, connected to Pager duty, we have a lot of notification just for something being down during the check… and the resolution during the next test.

AlerManager and other tool provide such a feature, if the system under monitoring is not working as expected X times in a row, then the tool sends the notification. It could be wonderful to have this behaviour in Uptime Kuma.

[kmcbride3]

If this is implemented, I recommend that #2791 be implemented at the same time.

If a delay/escalation mechanism is established, I expect that in most cases, you wouldn't want to trigger a set of delayed or escalated notifications when services are restored.