helper: Add ContinuousTargetOccurence to work around inconsistency

[radeksimko]

This is to work around the known issue with inconsistent APIs, like IAM or KMS, namely #4375 #4306 #2869 #2499 #2136

Very similar issue has been raised recently in Vault too: hashicorp/vault#687
cc @jefferai

As you may expect such issues are difficult to reproduce unless you have a full copy of AWS environment.

Current solution(s)/work-around(s)

Just simply wait for the 1st "successful" state

The problem here is that 1st success != replication done, see below a snippet from my KMS testing:

KEY_ID=$(aws kms create-key --description "aws-eventually-consistent-kms-3" | jq -r .KeyMetadata.KeyId)
echo "Disabling $KEY_ID"
aws kms disable-key --key-id $KEY_ID
echo "$KEY_ID should be now disabled"
aws kms describe-key --key-id $KEY_ID | jq .KeyMetadata.Enabled
aws kms describe-key --key-id $KEY_ID | jq .KeyMetadata.Enabled
aws kms describe-key --key-id $KEY_ID | jq .KeyMetadata.Enabled
aws kms describe-key --key-id $KEY_ID | jq .KeyMetadata.Enabled
aws kms describe-key --key-id $KEY_ID | jq .KeyMetadata.Enabled
aws kms describe-key --key-id $KEY_ID | jq .KeyMetadata.Enabled

Disabling 128169fc-a5ac-481f-99f5-730c6980d924
128169fc-a5ac-481f-99f5-730c6980d924 should be now disabled
false
true
false
false
true
true

Retry dependent API calls

This for example means retrying ec2:StartInstance if the IAM Instance Profile has a policy which has not been replicated yet or retrying ecs:CreateService if the IAM Role has a policy which has not been replicated yet.

It seemed to be a pretty good solution, until I realised this approach fails exactly the same way as the approach above - see #4375 (comment) where ecs:CreateService succeeded and subsequent ecs:UpdateService failed.

[radeksimko]

This would help us to unblock the KMS PR (or at least it is my feeling that people will less likely complain about the eventual consistency effects of it).

@catsby @phinze What do you think of this concept? I know it may look a bit "hacky", but I was not able to come up with anything better. Vault apparently somehow worked around this by using STS (which is region specific), but that's not quite the option for us in Terraform I'm afraid as we're working with the resources directly, not solving auth problem.

[phinze]

Just finally got around to looking at this. Seems like a totally reasonable thing for us to add, and I like the unit tests!

LGTM

[radeksimko]

ok, I will resolve the conflicts, repush and wait for green light from Travis, then merge

[deitch]

It looks like this was merged in almost three years ago, but still seeing people reporting issues with IAM resource creation and eventual consistency.

Is there some (possibly unknown) config/flag one needs to use to indicate, "this is an eventually consistent resource, wait until you have x consistent successful gets before moving on"?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.