[RFE] Add detector notification_policy for BigPanda

[drissamri]

Is your feature request related to a problem? Please describe.
Add a notification policy to forward detector alerts from SignalFX to BigPanda

Describe the solution you'd like
Additionally next to the team notification policy I'd like to add a BigPanda policy too

locals {
  notifications = {
    critical  = ["Team,${var.sfx_team_id}"]
    major     = ["Team,${var.sfx_team_id}"]
    minor     = ["Team,${var.sfx_team_id}"]
    warning   = ["Team,${var.sfx_team_id}"]
    info      = ["Team,${var.sfx_team_id}"]
  }

# https://github.com/claranet/terraform-signalfx-detectors/tree/master/modules/integration_aws-lambda
module "signalfx-detectors-integration-aws-lambda" {
  for_each         = var.lambda_functions
  source           = "claranet/detectors/signalfx//modules/integration_aws-lambda"
  version          = ">= 1.0.0, < 2.0.0"
  environment      = each.value.env
  notifications    = local.notifications
  teams            = [var.sfx_team_id]
  prefixes         = [each.value.function_name]
  filtering_custom = "filter('aws_function_name','${each.value.function_name}') and filter('env','${each.value.env}')"
}

Additional context
https://docs.splunk.com/Observability/admin/notif-services/bigpanda.html#step-3-add-a-bigpanda-integration-as-a-detector-alert-recipient-in-observability-cloud

[xp-1000]

Hello @drissamri

notifications variable of modules is an object composed of one attribute for each severity available on signalfx.
It allows to automatically bind notifications of all detectors of a module according to one rule depending on the severity (see https://github.com/claranet/terraform-signalfx-detectors/wiki/Notifications-binding). Notice it is still possible to override this rule per detector basis.

these attributes are free and you can configure the value you want while you respect the signalfx provider format: https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/resources/detector#notification-format

in your case and from your example, you should be able to use something like:

locals {
  alerts_notifications = "Team,${var.sfx_team_id}","BigPanda,${var.bigpanda_credentialsid}"
  notifications = {
    critical  = local.alerts_notifications
    major     = local.alerts_notifications
    minor     = local.alerts_notifications
    warning   = local.alerts_notifications
    info      = ["Team,${var.sfx_team_id}"]
  }

does it make sens ?

[xp-1000]

I can misunderstand but in your case I think the real problem is the lack of bigpanda integration support in the signalfx provider. So you basically cannot retrieve the bigpanda credentials id from Terraform.

In contrast, PagerDuty has:

• a datasource to retrieve id from an existing, already manually configured from the webui integration: https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/data-sources/pagerduty_integration
• a resource to handle the configuration of the integration directly from Terraform and use its ID as any terraform resources: https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/resources/pagerduty_integration

In general, we (at Claranet) use a Terraform module which take care of configuring the alerting integration and we reuse its ID in notifications variables for detectors in this repo. For example for pagerduty we have the following output https://github.com/claranet/terraform-signalfx-integrations/blob/master/alerting/pagerduty/outputs.tf#L13 which is ready to be append in notifications list.

For BigPanda I am afraid you will have:

• either hardcode the id of your integration you can find manually from webui ou api
• or create an issue or a PR on https://github.com/splunk-terraform/terraform-provider-signalfx to implement a datasource/resource for BigPanda integration

[xp-1000]

About your usage of for_each to import module for each lambda functions you have in variable.

I would like to suggest to give a look to https://github.com/claranet/terraform-signalfx-detectors/wiki/Tagging-convention.
SignalFx and the modules are designed to automatically "discover" resources and apply detector for each of them.

So I recommend in general to use a tagging convention (either the default on this repo or your custom one) and rely on this policy to import only once the module which will monitor all functions respecting this convention.

From this way you have a dynamic monitoring and any new function added to your aws account (with the right tags of course) will be added to monitoring without the need to update you variable list manually.

That said you can have plenty of good reasons that I ignore to do that and it works but I generally do not "multi instance" a same module until I need to have different configuration for different resources (thresholds, notifications recipients ...): https://github.com/claranet/terraform-signalfx-detectors/wiki/Guidance#Multi-instantiation

[drissamri]

I can misunderstand but in your case I think the real problem is the lack of bigpanda integration support in the signalfx provider. So you basically cannot retrieve the bigpanda credentials id from Terraform.

In contrast, PagerDuty has:

• a datasource to retrieve id from an existing, already manually configured from the webui integration: https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/data-sources/pagerduty_integration
• a resource to handle the configuration of the integration directly from Terraform and use its ID as any terraform resources: https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/resources/pagerduty_integration

In general, we (at Claranet) use a Terraform module which take care of configuring the alerting integration and we reuse its ID in notifications variables for detectors in this repo. For example for pagerduty we have the following output https://github.com/claranet/terraform-signalfx-integrations/blob/master/alerting/pagerduty/outputs.tf#L13 which is ready to be append in notifications list.

For BigPanda I am afraid you will have:

• either hardcode the id of your integration you can find manually from webui ou api
• or create an issue or a PR on https://github.com/splunk-terraform/terraform-provider-signalfx to implement a datasource/resource for BigPanda integration

Thank you for your quick response, I will take this up with our SignalFX support people to see if we can get this integrated inside their terraform module! Thank you. 🥇

[drissamri]

About your usage of for_each to import module for each lambda functions you have in variable.

I would like to suggest to give a look to https://github.com/claranet/terraform-signalfx-detectors/wiki/Tagging-convention. SignalFx and the modules are designed to automatically "discover" resources and apply detector for each of them.

So I recommend in general to use a tagging convention (either the default on this repo or your custom one) and rely on this policy to import only once the module which will monitor all functions respecting this convention.

From this way you have a dynamic monitoring and any new function added to your aws account (with the right tags of course) will be added to monitoring without the need to update you variable list manually.

That said you can have plenty of good reasons that I ignore to do that and it works but I generally do not "multi instance" a same module until I need to have different configuration for different resources (thresholds, notifications recipients ...): https://github.com/claranet/terraform-signalfx-detectors/wiki/Guidance#Multi-instantiation

I did not know that, thanks for letting me know. I have forwarded this to our company team owning our terraform set up to take a better look at this! 👍