provider/aws: Retry RunInstance if IAM profile hasn't propagated

[catsby]

Newly created IAM roles can take a few seconds to propagate.
From Launching an Instance with an IAM Role Using the AWS CLI:

After you create an IAM role, it may take several seconds for the permissions to propagate. If your first attempt to launch an instance with a role fails, wait a few seconds before trying again. For more information, see Troubleshooting Working with Roles in the Using IAM guide.

In testing, this time is typically 5-10 seconds. In this PR, add a simple for loop that tries up to 5 times, sleeping 2 seconds in-between, if and only if we hit an error that matches that kind of failure.

Fixes #1885

[phinze]

RESILIENCE +5

LGTM! 👍

[JeanMertz]

👍

[osterman]

I think we're seeing a problem related to this. Every now and then we'll get an instance that comes online which has the proper IAM role assigned to it (by name), but it appears to be the ID of the old role. The consequence is that the instance in our case cannot AssumeRole to use AWS APIs. If we taint the aws_instance resource and apply, it usually works, so I don't think it's a misconfiguration on our end.

For the record, we use iam_instance_profile = "${aws_iam_instance_profile.cluster.name}" for each of our hosts, so we're expressing the dependencies to TF.

[phinze]

@osterman That's definitely odd! Sounds like it must be upstream misbehavior - the API accepts "name" here so I'm not sure if there's much else we can do on our side? Perhaps something to take up with AWS Support.

If you believe there are steps we should be taking on the Terraform side, feel free to file us a fresh issue!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.