AWS: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue

[iloveicedgreentea]

Please include the following information in your report:

    Resource ID: aws_cloudwatch_metric_alarm.fleet-scale-down
    Mismatch reason: attribute mismatch: alarm_actions.1437824865
    Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"alarm_actions.1437824865":*terraform.ResourceAttrDiff{Old:"", New:"arn:aws:autoscaling:us-east-1:REDACTED:scalingPolicy:7f2a81b4-fb1d-440a-9d81-fcc9a2bdf4e6:resource/ec2/spot-fleet-request/sfr-7b43f06a-49dc-4edd-8029-374e3ed1a2ca:policyName/scale-down-staging", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "alarm_actions.#":*terraform.ResourceAttrDiff{Old:"0", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface {}(nil)}
    Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"alarm_actions.#":*terraform.ResourceAttrDiff{Old:"0", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "alarm_actions.1809487444":*terraform.ResourceAttrDiff{Old:"", New:"arn:aws:autoscaling:us-east-1:REDACTED:scalingPolicy:ba5ae522-5db0-4f83-96f9-d6570b59df71:resource/ec2/spot-fleet-request/sfr-02fbb2d2-c36a-48e7-ae43-3a498bf24324:policyName/scale-down-staging", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface {}(nil)}```

```* provider.archive: version = "~> 1.1"
* provider.aws: version = "~> 2.1"
* provider.template: version = "~> 2.1"

I noticed when it creates these resources due to the fleet ID changing, it changes the alarm actions from 0 to 1 and.... it should just not do that.

Deleting the resource had no effect. I believe I will need to manually remove it from the state as it created fine the first time.

[iloveicedgreentea]

Removing it from the state did not fix the issue. Resource was made initially but now it refuses to create.

Edit: I deleted the fleet, auto scaling target via the cli, then tried again. Now the resources are created. Its possible there is some provider bug when it needs to recreate the target/fleet request instead of just making a new one (I manually cancelled it)

[apparentlymart]

Hi @iloveicedgreentea,

This class of error normally happens when a provider produces a final result that conflicts with what was planned. In order to see which resource type the bug actually originates in, could you please share the configuration of your aws_cloudwatch_metric_alarm.fleet-scale-down resource?

I'm expecting to see an interpolation inside your alarm_actions setting that refers to some other resources's attribute, in which case the bug here should originate from the implementation of the resource you are referencing: the provider for that resource is producing an inaccurate plan, which contradicts its result after apply.

[iloveicedgreentea]

Hey thanks for the response. This alarm scales on a custom metric

resource "aws_cloudwatch_metric_alarm" "fleet-scale-down" {
  alarm_name          = "fleet_scale-${var.environment}-down"
  alarm_description   = "Scales down fleet"
  alarm_actions       = ["${aws_appautoscaling_policy.fleet_scale_down.arn}"]
  comparison_operator = "LessThanOrEqualToThreshold"
  evaluation_periods  = "5"
  metric_name         = "service-total-pending-task-count"
  namespace           = "${var.clustername}"
  period              = "60"
  statistic           = "Average"
  threshold           = "0"

  dimensions {
    "Number of Pending Tasks" = "Count"
  }
}

resource "aws_appautoscaling_policy" "fleet_scale_down" {
  name               = "scale-down-${var.environment}"
  policy_type        = "StepScaling"
  resource_id        = "spot-fleet-request/${aws_spot_fleet_request.main.id}"
  scalable_dimension = "ec2:spot-fleet-request:TargetCapacity"
  service_namespace  = "ec2"

  step_scaling_policy_configuration {
    adjustment_type         = "ChangeInCapacity"
    cooldown                = 10
    metric_aggregation_type = "Average"

    step_adjustment {
      metric_interval_upper_bound = 0
      scaling_adjustment          = -2
    }
  }

  depends_on = ["aws_appautoscaling_target.fleet"]
}

I'm expecting to see an interpolation inside your alarm_actions setting that refers to some other resources's attribute

resource_id changes every time because the fleet reservation valid_until is not set, therefore the time changes and that makes the reservation get recreated. I assume this is what is screwing up the process here.

[apparentlymart]

Thanks for the extra details!

I'm not familiar with the aws_appautoscaling_policy resource type but it seems that changes to the resource_id cause the arn to change because the resource id is embedded as part of it:

arn:aws:autoscaling:us-east-1:REDACTED:scalingPolicy:ba5ae522-5db0-4f83-96f9-d6570b59df71:resource/ec2/spot-fleet-request/sfr-02fbb2d2-c36a-48e7-ae43-3a498bf24324:policyName/scale-down-staging

In order for this to behave properly, the implementation of the aws_appautoscaling_policy resource type needs to tell Terraform about this change as part of the plan phase.

The most straightforward thing for it to do would be to check whether resource_id is changing and, if so, mark the arn as unknown in the plan so that Terraform can defer evaluating it and expressions derived from it until the apply phase. That would cause the alarm actions list to itself be unknown until apply time, which will avoid tripping up Terraform's "diffs didn't match" safety check.

Since the fix here will be in the AWS provider, I'm going to ask our bot to migrate this issue over to the AWS provider's repository. Thanks again for reporting this and for the additional context around it!

[ghost]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#7963 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-aws#7963.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.