Multiple services for PagerDuty alert

[oldmantaiter]

Some users might want alerts to take different escalation paths (eg in Pagerduty to multiple rotations), so adding support to reference "named" alert handlers of the same type would be useful.

I was thinking I could attempt to tackle this when I get some time, or at least open a discussion on the best way to accomplish this (while staying in the style guidelines of the project).

Very cool project by the way, really like the DSL and how it is structured.

EDIT: Not sure if this already exists, haven't had a whole bunch of time to go through the code. So if it does, please ignore this request, I was basing this off configuration examples.

[nathanielc]

Do you want to provide some examples of how the DSL would look for 'named' alert handlers? I like the idea, if we can decide on a DSL structure we like I can point you in the right direction for implementing a solution. Thanks

[oldmantaiter]

@nathanielc It would be along the lines of how the slack channel is configured. So it looks to be something that is just pagerduty specific at this point.

Based on VictorOps and how that is specified, it could either be done that way (adding another function to be chained in the DSL) or in the config.

As for examples:

Main config

[pagerduty.<name>] # Defaults to 'default'
  enabled = true
  service-key = "xxxxxxx"

DSL

.alert()
  .pagerDuty("<name>")

This would allow a bit of readability when configuring which schedule to escalate to. As I said above, we could also just have a chained function like routingKey for VictorOps and pass in the service key for Pagerduty in there, but that is less readable due to the random nature of the service key.

Let me know your thoughts. I have no problem with either way, but potentially adding the ability for "named" outputs that could be referenced (not only for Pagerduty) might be of use?

[nathanielc]

I like the design. The configuration will probably change to something like:

[pagerduty]
  enabled = true
  [pagerduty.schedules]
    schedule_name = "service_key"
    another_schedule  = "different_service_key"

Which would match with a config structure like:

type Config struct {
    // Whether PagerDuty integration is enabled.
    Enabled bool `toml:"enabled"`
    // The PagerDuty API URL, should not need to be changed.
    URL string `toml:"url"`
    // Whether every alert should automatically go to PagerDuty
    Global bool `toml:"global"`
    // Schedules name -> service key
    Schedules map[string]string
}

Is schedule the right name? I am not very familiar with PagerDuty so I could be using the wrong name here.

This change for PagerDuty should be pretty easy, just need to update the both pipeline/alert.go and 'alert.go and then the service in services/pagerduty.

[oldmantaiter]

Sounds good. For PagerDuty, it would be called a service, which gets mapped to an escalation policy, which can be mapped to multiple schedules.

I'll start taking a look soon into adding this. Still getting familiar with the project so want to make sure I can adequately test.

[mattfinlayson]

This is working fine for us in production, let me know if if needs any more love.

[oldmantaiter]

@savagegus I have not had the time to eval Kapacitor fully yet, so if it works for you I'll take your word for it and close out this issue. Thanks! This will make it easier when we get around to using it.