[Bug]: InvalidParameterValueException on newly created IAM role with Backup Access Policy

[PeskyPotato]

Terraform Core Version

1.5.6

AWS Provider Version

5.13.1

Affected Resource(s)

• aws_backup_vault_policy
• aws_iam_role

Expected Behavior

Apply complete! after terraform apply.

Actual Behavior

Receive an InvalidParameterValueException while creating the Backup Vault Policy since it relays on the IAM role to be successfully created first as IAM is eventually consistent.

Relevant Error/Panic Output Snippet

│ Error: creating Backup Vault Policy (tf_backup_vault): InvalidParameterValueException: Provided principal is not valid. Please input a valid principal and retry the request.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "e81a3473-f7ce-4a56-82b3-4237cd97aa30"
│   },
│   Code_: "ERROR_3107",
│   Message_: "Provided principal is not valid. Please input a valid principal and retry the request."
│ }
│ 
│   with aws_backup_vault_policy.backup_vault_access_policy,
│   on main.tf line 65, in resource "aws_backup_vault_policy" "backup_vault_access_policy":
│   65: resource "aws_backup_vault_policy" "backup_vault_access_policy" {

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.13"
    }
  }

  required_version = ">= 1.5.0"
}

provider "aws" {
  region  = "eu-central-1"
}

resource "aws_iam_role" "backup_role" {
    name = "backup_role"
    
    assume_role_policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
          {
            Action = "sts:AssumeRole"
            Effect = "Allow"
            Sid    = ""
            Principal = {
              Service = "backup.amazonaws.com"
            }
          },
        ]
      })
}

resource "aws_iam_role_policy_attachment" "backup_policy_attachment" {
    role = aws_iam_role.backup_role.name
    policy_arn = "arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForBackup"
}

resource "aws_backup_vault" "backup_vault" {
    name = "tf_backup_vault"
}

data "aws_iam_policy_document" "example" {
    statement {
        effect = "Allow"

        principals {
          type        = "AWS"
          identifiers = [aws_iam_role.backup_role.arn]
        }
        actions = [
            "backup:DescribeBackupVault",
            "backup:DeleteBackupVault",
            "backup:PutBackupVaultAccessPolicy",
            "backup:DeleteBackupVaultAccessPolicy",
            "backup:GetBackupVaultAccessPolicy",
            "backup:StartBackupJob",
            "backup:GetBackupVaultNotifications",
            "backup:PutBackupVaultNotifications",
        ]
        resources = [aws_backup_vault.backup_vault.arn]
  }    
}

resource "aws_backup_vault_policy" "backup_vault_access_policy" {
    backup_vault_name = aws_backup_vault.backup_vault.name
    policy = data.aws_iam_policy_document.example.json
    depends_on = [ aws_iam_role.backup_role ]
}

Steps to Reproduce

terraform apply

Debug Output

$ terraform apply --auto-approve

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_iam_policy_document.example will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "example" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "backup:DeleteBackupVault",
              + "backup:DeleteBackupVaultAccessPolicy",
              + "backup:DescribeBackupVault",
              + "backup:GetBackupVaultAccessPolicy",
              + "backup:GetBackupVaultNotifications",
              + "backup:PutBackupVaultAccessPolicy",
              + "backup:PutBackupVaultNotifications",
              + "backup:StartBackupJob",
            ]
          + effect    = "Allow"
          + resources = [
              + (known after apply),
            ]

          + principals {
              + identifiers = [
                  + (known after apply),
                ]
              + type        = "AWS"
            }
        }
    }

  # aws_backup_vault.backup_vault will be created
  + resource "aws_backup_vault" "backup_vault" {
      + arn             = (known after apply)
      + force_destroy   = false
      + id              = (known after apply)
      + kms_key_arn     = (known after apply)
      + name            = "tf_backup_vault"
      + recovery_points = (known after apply)
      + tags_all        = (known after apply)
    }

  # aws_backup_vault_policy.backup_vault_access_policy will be created
  + resource "aws_backup_vault_policy" "backup_vault_access_policy" {
      + backup_vault_arn  = (known after apply)
      + backup_vault_name = "tf_backup_vault"
      + id                = (known after apply)
      + policy            = (known after apply)
    }

  # aws_iam_role.backup_role will be created
  + resource "aws_iam_role" "backup_role" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "backup_role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)
    }

  # aws_iam_role_policy_attachment.backup_policy_attachment will be created
  + resource "aws_iam_role_policy_attachment" "backup_policy_attachment" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForBackup"
      + role       = "backup_role"
    }

Plan: 4 to add, 0 to change, 0 to destroy.
aws_backup_vault.backup_vault: Creating...
aws_iam_role.backup_role: Creating...
aws_backup_vault.backup_vault: Creation complete after 0s [id=tf_backup_vault]
aws_iam_role.backup_role: Creation complete after 1s [id=backup_role]
aws_iam_role_policy_attachment.backup_policy_attachment: Creating...
data.aws_iam_policy_document.example: Reading...
data.aws_iam_policy_document.example: Read complete after 0s [id=3104379160]
aws_backup_vault_policy.backup_vault_access_policy: Creating...
aws_iam_role_policy_attachment.backup_policy_attachment: Creation complete after 0s [id=backup_role-20230824140655713300000001]
╷
│ Error: creating Backup Vault Policy (tf_backup_vault): InvalidParameterValueException: Provided principal is not valid. Please input a valid principal and retry the request.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "83943a0f-0ddc-4bef-887b-0dcf59595462"
│   },
│   Code_: "ERROR_3107",
│   Message_: "Provided principal is not valid. Please input a valid principal and retry the request."
│ }
│ 
│   with aws_backup_vault_policy.backup_vault_access_policy,
│   on main.tf line 65, in resource "aws_backup_vault_policy" "backup_vault_access_policy":
│   65: resource "aws_backup_vault_policy" "backup_vault_access_policy" {
│ 

Panic Output

N/A

Important Factoids

No response

References

• AWS IAM and eventual consistency
• Related MalformedPolicyDocument: Invalid principal in policy: "AWS" [Only when Principal is a ROLE.] #8905
• Related InvalidPolicy ("invalid principal") for aws_ses_identity_policy with freshly-created aws_iam_user — but only first time #22549

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[PeskyPotato]

Also observed with provider registry.terraform.io/hashicorp/aws v5.14.0.

[PeskyPotato]

Adding an artificial 20 second delay mitigates the issue for now. Less than 15 seconds and the issue still sometimes comes up, so I picked 20 seconds to be safe.

resource "time_sleep" "iam_delay" {
  depends_on = [aws_iam_role.backup_role]

  create_duration = "20s"
}

[github-actions]

This functionality has been released in v5.30.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.