datadog_monitor - Removal of 'silenced' resource argument has no practical effect

[saj]

Terraform Version

• Terraform v0.11.1
• provider.datadog v1.0.2

Affected Resource(s)

• datadog_monitor

Terraform Configuration Files

Start with something like this:

resource "datadog_monitor" "foo" {
  name = "..."
  type = "metric alert"
  ...

  # The significant bit.
  silenced {
    "*" = 0
  }
}

then remove the silenced argument:

resource "datadog_monitor" "foo" {
  name = "..."
  type = "metric alert"
  ...
}

Expected Behavior

• Terraform removes the silence.
• Terraform reaches steady state. (All resources should be in-sync.)

Actual Behavior

Terraform appears to remove the silence but this never actually happens:

module.redacted.datadog_monitor.redacted_a: Modifying... (ID: 3724916)
  silenced.%: "1" => "0"
  silenced.*: "0" => ""
module.redacted.datadog_monitor.redacted_b: Modifying... (ID: 3724915)
  silenced.%: "1" => "0"
  silenced.*: "0" => ""
module.redacted.datadog_monitor.redacted_c: Modifying... (ID: 3724914)
  silenced.%: "1" => "0"
  silenced.*: "0" => ""
module.redacted.datadog_monitor.redacted_d: Modifying... (ID: 3724917)
  silenced.%: "1" => "0"
  silenced.*: "0" => ""
module.redacted.datadog_monitor.redacted_c: Modifications complete after 1s (ID: 3724914)
module.redacted.datadog_monitor.redacted_d: Modifications complete after 1s (ID: 3724917)
module.redacted.datadog_monitor.redacted_a: Modifications complete after 1s (ID: 3724916)
module.redacted.datadog_monitor.redacted_b: Modifications complete after 1s (ID: 3724915)

Apply complete! Resources: 0 added, 4 changed, 0 destroyed.

Subsequent terraform plan invocations show the resources as out-of-sync, and subsequent terraform apply invocations end in the same result.

Workaround:

1. Navigate to https://app.datadoghq.com/monitors#downtime
2. Manually remove the silences.

Steps to Reproduce

1. Create a new Datadog monitor using Terraform. The monitor is successfully created with a global silence.
2. Modify the Datadog monitor using Terraform. Try to remove the silence created in the previous step. This will have no effect. Terraform will never reach steady state on its own.

[jamesmharvey]

If you don't want to use the web interface, another workaround is to set the silenced timestamp to some (non-zero) timestamp in the past.

[pdecat]

It seems an explicit call to the https://api.datadoghq.com/api/v1/monitor/<MONITOR_ID>/unmute endpoint is required to effectively unmute the monitor's muted scopes: https://docs.datadoghq.com/api/#unmute-a-monitor

[bal2ag]

I wanted to make a comment that when this release was shipped (1.6.0) many of our monitors for which we were managing downtimes manually through the UI (required since the "silenced" argument wasn't working) suddenly were un-muted. I know this was an unintended consequence but it's causing us some difficulties as all of our manually managed downtimes were suddenly removed. Furthermore some monitors started failing to apply as they received a 400 response from Datadog (along the lines of a monitor already being unmuted).