Better grafana to Slack alerts

[rgaudin]

With grafana, we now have access to alerts on HTTP 50x errors (on for all 50x) and one for 500 on all our containers. It's a good thing but the way we get those alerts is not ideal:

• we get a message after a varying delay of up to 5mn
• we get a resolved message once the grafana condition for alert is gone.
• the messages are very long and useless, with just the namespace burried inside the extended version of the message

The most annoying to me is the resolve message that creates noise. The fact that I have to switch context to go to slack, select channel and click on the message to find out about the namespace (and infer the culprit container) may look like nothing to care about but it's actually a huge blocker to react on issues.

Uptime robot alerts very clear and useful in comparison. From the notification preview I already know what's going on an can decide whether it requires immediate action or not.

With grafana, I can't so I mostly decide that I'll look at it when I get a chance…

---

---

[benoit74]

I can only agree 👍