Pass Go's time function into the alertmanager templates to be able to do time calculations

[aboukhal]

Proposal

Currently, it's not possible to do time calculations when sending alerts to Slack for example. Explicitly converting to another time zone or calculating a duration between startsAt and now would be a tremendous help. There is nothing on Slack/Mattermost's side to do any conversions or calculations there.

This would be easily solved by passing the default time functions into the template at this location.

alertmanager/template/template.go

Line 170 in 680568b

var DefaultFuncs = FuncMap{

[grobinson-grafana]

We added date and timezone functions here.

[aboukhal]

This is awesome. Although, since so little from time is passed through, calculations are still not possible.

[grobinson-grafana]

Perhaps I misunderstood, what calculations would you like to do?

[aboukhal]

As I wrote initially:

or calculating a duration between startsAt and now

In an alert I would like to create text, such as "started 3 hours, 22 minutes ago". This is especially helpful for a repeated message, to see that this is ongoing instead of new, or for the resolved message, to have a clear documentation of how long an issue was present.

[grobinson-grafana]

As I wrote initially:

or calculating a duration between startsAt and now

In an alert I would like to create text, such as "started 3 hours, 22 minutes ago". This is especially helpful for a repeated message, to see that this is ongoing instead of new, or for the resolved message, to have a clear documentation of how long an issue was present.

Oh OK. You want a humanize function / or ago function. I think we can add that!

[grobinson-grafana]

Keep in mind the started at text will be relative to the time the notification was sent. So if the alert started at 09:00 UTC, and the notification was sent at 10:00 UTC, then the text would be "started one hour ago". However, suppose you don't see the notification until 11:00 UTC, the text will still say "started one hour ago" even though it actually started two hours ago.

[bryanasdev000]

Related:

#3717

#3720

prometheus/common#627

[aboukhal]

Keep in mind the started at text will be relative to the time the notification was sent. So if the alert started at 09:00 UTC, and the notification was sent at 10:00 UTC, then the text would be "started one hour ago". However, suppose you don't see the notification until 11:00 UTC, the text will still say "started one hour ago" even though it actually started two hours ago.

Yes, this is expected and is consistent with how chatops usually displays things.

[aboukhal]

Is there a specific design decision why you don't want to pass though the entire time through, instead of covering the use cases one by one? That way the end user would be able to do anything that is covered by Go's time functions.

[grobinson-grafana]

Hi! 👋 Most of the functions in time are methods, so you should be able to use them. We don't pass all of the functions in because there are some we don't want to add to templates like time.Sleep.

[freak12techno]

I want to add time.Since and pretty formatting a duration and filed an issue on that here: #3717
We agreed to move humanizeDuration from prometheus to common package to reuse it here, so I did this prometheus/common#627 but it's still in review.
After this is done and merged, we can import it here and use it here as well.

Can someone from Alertmanager team review it? (@grobinson-grafana maybe?)

[grobinson-grafana]

I can do a review! 🙂 I can't merge it as I'm not an Alertmanager maintainer, but I'll let @gotjosh know when I've reviewed it for him to look at too.

[aboukhal]

I believe this issue is fixed by #3863. I would love to test it - is there a schedule for the next release? If it works as advertised, I'll happily close this ticket myself.
Thanks for your effort, @freak12techno

[freak12techno]

@aboukhal yes, you can build from source to test it. As per the new release, I am not an Alertmanager maintainer, so I do not know. @gotjosh can you maybe ping someone or answer here?

[aboukhal]

It's... challenging to build external projects in our environment to say the least. Otherwise I would have been more actively involved in this issue from the start. So unfortunately I can only test it once it's released.