AWS Organizations throws ConcurrentModificationException when terraform applies concurrent modification with more than one aws_organizations_policy_attachment resource

[badaltitude]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.3
provider.aws v1.24.0
Currently blocked from upgrading past v0.11.3 due to proxy error

Affected Resource(s)

• aws_organizations_policy_attach

Terraform Configuration Files

terraform {
  required_version = ">= 0.11.1"

  backend "local" {
    path  = "scpattach.tfstate"
  }
}

provider "aws" {
  region  = "us-east-1"
  version = "1.24"
}

resource "aws_organizations_organization" "Root" {}

resource "aws_organizations_account" "SteveM" {
  name  = "SteveM"
  email = "stevem@xxxxxx.com"
}

resource "aws_organizations_policy" "Policy1" {
  name        = "Policy1"
  description = "Policy1"

  content = <<CONTENT
   {
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": [
            "artifact:*"
        ],
        "Resource": "*"
    }
}
CONTENT
}

resource "aws_organizations_policy" "Policy2" {
  name        = "Policy2"
  description = "Policy2"

  content = <<CONTENT
   {
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": [
            "events:*"
        ],
        "Resource": "*"
    }
}
CONTENT
}

resource "aws_organizations_policy" "Policy3" {
  name        = "Policy3"
  description = "Policy3"

  content = <<CONTENT
  {
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": [
            "trustedadvisor:*"
        ],
        "Resource": "*"
    }
}
CONTENT
}

resource "aws_organizations_policy_attachment" "SteveMAttachPolicy1" {
  policy_id = "${aws_organizations_policy.Policy1.id}"
  target_id = "${aws_organizations_account.SteveM.id}"
}

resource "aws_organizations_policy_attachment" "SteveMAttachPolicy2" {
  policy_id = "${aws_organizations_policy.Policy2.id}"
  target_id = "${aws_organizations_account.SteveM.id}"
}

resource "aws_organizations_policy_attachment" "SteveMAttachPolicy3" {
  policy_id = "${aws_organizations_policy.Policy3.id}"
  target_id = "${aws_organizations_account.SteveM.id}"
}

Expected Behavior

Account created. SCPs created. All SCP attached to account.

Actual Behavior

Account created. SCPs created. 1 x SCP attached to account.
2nd, 3rd, etc SCP attachments error with ConcurrentModificationException

E.g. with organization, account, SCPs and 1 out of 3 attachment in state already:

terraform apply --auto-approve
aws_organizations_organization.Root: Refreshing state... (ID: r-xxxx)
aws_organizations_account.SteveM: Refreshing state... (ID: 123412341234)
aws_organizations_policy_attachment.SteveMAttachPolicy1: Refreshing state... (ID: 123412341234:p-062qpzso)
aws_organizations_policy_attachment.SteveMAttachPolicy2: Creating...
  policy_id: "" => "p-3hplqnb2"
  target_id: "" => "123412341234"
aws_organizations_policy_attachment.SteveMAttachPolicy3: Creating...
  policy_id: "" => "p-mw667c88"
  target_id: "" => "123412341234"
aws_organizations_policy_attachment.SteveMAttachPolicy2: Creation complete after 4s (ID: 123412341234:p-3hplqnb2)

Error: Error applying plan:

1 error(s) occurred:

* aws_organizations_policy_attachment.SteveMAttachPolicy3: 1 error(s) occurred:

* aws_organizations_policy_attachment.SteveMAttachPolicy3: error creating Organizations Policy Attachment: ConcurrentModificationException: AWS Organizations can't complete your request because it conflicts with another attempt to modify the same entity. Try again later.
        status code: 400, request id: 34662f97-7f60-11e8-8d0d-bfc72f07fba5

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Steps to Reproduce

1. terraform apply

Important Factoids

Also happens

• if some resources are imported into state already
• >1 policy attachment is created in one apply
• >1 policy attachment is destroyed in one apply

Workarounds:

• run terraform apply again and again
• add depends_on to the aws_organizations_policy_attachment resources forces them to be created in serial, not concurrently.

resource "aws_organizations_policy_attachment" "SteveMAttachPolicy1" {
  policy_id  = "${aws_organizations_policy.Policy1.id}"
  target_id  = "${aws_organizations_account.SteveM.id}"
}

resource "aws_organizations_policy_attachment" "SteveMAttachPolicy2" {
  depends_on = ["aws_organizations_policy_attachment.SteveMAttachPolicy1"]
  policy_id  = "${aws_organizations_policy.Policy2.id}"
  target_id  = "${aws_organizations_account.SteveM.id}"
}

resource "aws_organizations_policy_attachment" "SteveMAttachPolicy3" {
  depends_on = ["aws_organizations_policy_attachment.SteveMAttachPolicy2"]
  policy_id  = "${aws_organizations_policy.Policy3.id}"
  target_id  = "${aws_organizations_account.SteveM.id}"
}

[atali]

Hello,

I have the same issue ( in my case I was creating 2 aws_cognito_identity_pool_roles_attachment separately

Error: Error applying plan:

1 error(s) occurred:

* aws_cognito_identity_pool_roles_attachment.xxxx_role_attachment: 1 error(s) occurred:

* aws_cognito_identity_pool_roles_attachment.xxxx_role_attachment: Error creating Cognito Identity Pool Roles Association: ConcurrentModificationException: Only one request to update resource type Identity Pool can be processed at a time
        status code: 400, request id: 86ffa926-8506-11e8-b6d9-43fe79929033

[andyakins]

Any update on this issue? Just ran into it today, when deleting SCPs from accounts/organizations...

Thanks.

[tdmalone]

Workaround:

terraform plan -out planfile
terraform apply -parallelism=1 planfile

Will be a bit slower than usual, but appears to avoid this particular error.

[bflad]

Fix submitted: #9195

[bflad]

The fix to have the Organizations service client within the Terraform AWS Provider automatically retry requests on this error has been merged and will release with version 2.19.0, likely later this week. 👍

[bflad]

This has been released in version 2.19.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!