Argument idea: ignore_silences

[alexturek]

https://github.com/terraform-providers/terraform-provider-datadog/issues/41 is affecting my team right now, causing our plans to fill up with things that are never applied.

The root cause of this is that our oncalls may mute monitors from the Datadog UI, and then the next time we generate a plan, terraform intends to remove these mutes, but can't.

Even if #41 is fixed, we'd like to enable muting-in-the-datadog-UI to just work without getting written by terraform apply. So my team's idea is to remove application of the silenced argument, with a ignore_silences argument.

Silencing a Datadog monitor has a couple of parameters, what you're silencing, and for how long. (ref the resource docs) What I'd want to ignore is mutes of a specific scope.

Suggested interface

resource "datadog_monitor" "my_service_monitor" {
  silenced {
    "service:A" = "0"
  }
  ignore_silences = ["service:B", "*"]
}

What this example would do is:

1. Enforce that the monitor with scope service:A is muted forever. Terraform would get the monitor into that state (pending datadog_monitor - Removal of 'silenced' resource argument has no practical effect #41).
2. Ignore if scope service:B or * (the generic scope) is muted, either permanently or temporarily. Never try to apply changes to these scopes, or show them in the plan.

This would enable an oncall to mute the entire monitor for a brief time (e.g. if they're doing planned downtime) or mute service:B for a long time, while it's under development.

Terraform Version

current

Affected Resource(s)

Please list the resources as a list, for example:

• datadog_provider

[alexturek]

Welp, just learned about lifecycle.ignore_changes. For anyone who finds this later:

resource "datadog_monitor" "my_monitor_name" {
  
  # Ignores if you silence the monitor from the UI
  lifecycle {
    ignore_changes = [
      "silenced"
    ]
  }
}

[glasser]

It would be nice if there was a way to do this by default for all datadog_monitor's, though I suppose waiting for the future plugin that removes silenced will work too.