intermittent issue with aws_appautoscaling_target and aws_appautoscaling_policy creation

[brikis98]

Terraform Version

Terraform v0.7.7

Affected Resource(s)

• aws_appautoscaling_target
• aws_appautoscaling_policy

Terraform Configuration Files

resource "aws_appautoscaling_target" "target" {
  name = "MyService"
  resource_id = "service/MyCluster/MyService"
  role_arn = "${aws_iam_role.role.arn}"
  min_capacity = 1
  max_capacity = 10
}

resource "aws_appautoscaling_policy" "scale_out" {
  name = "MyServiceScaleOut"
  resource_id = "service/MyCluster/MyService"

  adjustment_type = "ChangeInCapacity"
  cooldown = 60
  metric_aggregation_type = "Average"

  step_adjustment {
    metric_interval_lower_bound = 0
    scaling_adjustment = 1
  }

  depends_on = ["aws_appautoscaling_target.target"]
}

Expected Behavior

The auto scaling policy and target are both created successfully.

Actual Behavior

Intermittently, you get an error like this:

Error putting scaling policy: ObjectNotFoundException: No scalable target registered for service namespace: ecs (... sorry, I forgot to copy/paste the rest of the error ...)

If you re-run terraform apply, the error usually goes away.

Steps to Reproduce

1. terraform apply

Important Factoids

If you re-run terraform apply, sometimes multiple times, it works. I suspect this is a classic issue with the fact that AWS is asynchronous and eventually consistent and Terraform is not properly waiting for the aws_appautoscaling_target resource to be created.

[yesteph]

Hello,

No problem with 0.7.13 and the following declarations:

resource "aws_appautoscaling_target" "scale_tgt" {
  name = "${var.env_prefix}-mlk-ms-brand-scale-tgt"
  service_namespace = "ecs"
  resource_id = "service/${var.cluster_name}/${aws_ecs_service.mlk_ms_brand.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  role_arn = "${var.ecs_autoscale_role}"
  min_capacity = 1
  max_capacity = "${lookup(var.autoscaling_max_size, var.env_prefix, "1")}"

  depends_on = ["aws_ecs_service.mlk_ms_brand"]
}

resource "aws_appautoscaling_policy" "scale_in" {
  name = "${aws_ecs_service.mlk_ms_brand.name}-scale-in"
  service_namespace = "ecs"
  resource_id = "service/${var.cluster_name}/${aws_ecs_service.mlk_ms_brand.name}"
  scalable_dimension = "ecs:service:DesiredCount"

  adjustment_type = "ChangeInCapacity"
  cooldown = 60
  metric_aggregation_type = "Average"

  step_adjustment {
    metric_interval_upper_bound = 0
    scaling_adjustment = -2
  }

  depends_on = ["aws_appautoscaling_target.scale_tgt"]
}

[brikis98]

It's an intermittent problem, so it won't always happen.

FWIW, as most timing issues with Terraform and AWS, it may make a difference where you're located. When I saw the error above, I was in the US, but deploying to the eu-central-1 data center, which would increase replication lag.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.