ECS Service erroring out due to eventually-consistent IAM role

[cordoval]

aws_autoscaling_group.ecs-cluster: Creation complete
Error applying plan:

1 error(s) occurred:

• aws_ecs_service.lead-validation: InvalidParameterException: Unable to assume role and validate the listeners configured on your load balancer. Please verify the role being passed has the proper permissions.
  status code: 400, request id: dca57d15-a4ef-11e5-9749-03d1427e2486

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

after terraform apply again, it passes

[radeksimko]

Your assumption is correct, this is caused by the eventually consistent nature of IAM.

We've been fighting with this all over the place, oddly it seems to be visible most in ECS... I'm not sure if it means that AWS treats ECS differently or whether people just manipulate with this IAM role here more often, so that the issue is more visible here. It may also be less visible in EC2 Instance Profiles purely because starting EC2 instance takes more time (I'm really just guessing).

Here is a sample from my debug log supporting the theory:

2015/12/22 13:55:25 [DEBUG] Creating ECS service: {
  ClientToken: "terraform-jorhmd4mrjbvbcikqyoune32zu",
  Cluster: "arn:aws:ecs:us-west-2:229213142987:cluster/terraformecstest11",
  DesiredCount: 1,
  LoadBalancers: [{
      ContainerName: "ghost",
      ContainerPort: 2368,
      LoadBalancerName: "tf-lb-orxnwpsknbgc5dhm6guvpq2rje"
    }],
  Role: "EcsService",
  ServiceName: "ghost",
  TaskDefinition: "arn:aws:ecs:us-west-2:229213142987:task-definition/ghost_service:56"
}
2015/12/22 13:55:25 [DEBUG] Waiting for state to become: success
2015/12/22 13:55:26 [DEBUG] Trying to create ECS service again: "Unable to assume role and validate the listeners configured on your load balancer.  Please verify the role being passed has the proper permissions."
2015/12/22 13:55:27 [DEBUG] ECS service created: arn:aws:ecs:us-west-2:229213142987:service/ghost
2015/12/22 13:55:27 [DEBUG] Updating ECS service arn:aws:ecs:us-west-2:229213142987:service/ghost
2015/12/22 13:55:27 [DEBUG] root: eval: *terraform.EvalWriteState
2015/12/22 13:55:27 [DEBUG] root: eval: *terraform.EvalApplyProvisioners
2015/12/22 13:55:27 [DEBUG] root: eval: *terraform.EvalIf
2015/12/22 13:55:27 [DEBUG] root: eval: *terraform.EvalWriteDiff
2015/12/22 13:55:27 [DEBUG] root: eval: *terraform.EvalIf
2015/12/22 13:55:27 [DEBUG] root: eval: *terraform.EvalWriteState
2015/12/22 13:55:27 [DEBUG] root: eval: *terraform.EvalApplyPost
2015/12/22 13:55:27 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* aws_ecs_service.ghost: InvalidParameterException: Unable to assume role and validate the listeners configured on your load balancer.  Please verify the role being passed has the proper permissions.
    status code: 400, request id: 1511a6fc-a8ab-11e5-bc54-1dd530d1af1e

As you can see ^ the error is coming from the Update call, which means that ECS API actually allows you to create a new ECS service at one point, purely because it talks to a part of IAM which already has the IAM policy, but the Update call hits a different part which doesn't have that yet, hence it fails.

I've described this problem in depth here:
#3928 (comment)

KMS is apparently affected by this too.

[russmac]

I have a similar issue with a different error, Although errors reported as mine are pointed here it seems.

2016/03/03 17:16:02 [DEBUG] terraform-provider-aws: 2016/03/03 17:16:02 [DEBUG] Trying to create ECS service again: "Unable to assume role and validate the listeners configured on your load balancer. Please verify the role being passed has the proper permissions."
2016/03/03 17:16:02 [DEBUG] terraform-provider-aws: 2016/03/03 17:16:02 [TRACE] Waiting 500ms before next try

[brikis98]

Same problem here. Also with ECS.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.