[Bug]: CloudFormation Stack instances are being executed sequentially when using region_concurrency_type = "PARALLEL"

[andrejskuidins]

Terraform Core Version

1.3.4

AWS Provider Version

4.60.0

Affected Resource(s)

aws_cloudformation_stack_set
aws_cloudformation_stack_set_instance

Expected Behavior

In AWS console CloudFormation Stack instances are being executed in parallel

Actual Behavior

In AWS console CloudFormation Stack instances are being executed sequentially

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

Following code i being applied:

#KMS Key Replicas
resource "aws_cloudformation_stack_set" "kms_replicas" {
  administration_role_arn = aws_iam_role.AWSCloudFormationStackSetAdministrationRole.arn
  execution_role_name     = aws_iam_role.AWSCloudFormationStackSetExecutionRole.name
  name                    = "${var.name}-cf-stack-${var.tags["environment"]}-${data.aws_region.current.name}"
  capabilities            = var.capabilities
  permission_model        = var.permission_model
  operation_preferences {
    region_concurrency_type = "PARALLEL" #observed sequential propogation of stack set instances. Possible solution: new release of aws provider
    max_concurrent_count    = 1
  }
  template_body = templatefile("${path.module}/templates/replica.yml.tpl", {
    alias_name      = aws_kms_alias.kms_alias.name
    primary_key_arn = aws_kms_key.image_key.arn
    accounts        = var.accounts
    account_id      = data.aws_caller_identity.current.account_id
    roles           = var.kms_roles
    id              = aws_kms_key.image_key.id
  })
}

resource "aws_cloudformation_stack_set_instance" "kms_replicas" {
  for_each       = toset(local.replica_regions)
  region         = each.value
  stack_set_name = aws_cloudformation_stack_set.kms_replicas.name
  operation_preferences {
    region_concurrency_type = "PARALLEL" #observed sequential propogation of stack set instances. Possible solution: new release of aws provider
    max_concurrent_count    = 1
  }
  depends_on = [
    time_sleep.iam_propagation
  ]
}

#Since the IAM is a global service, it takes time to replicate its resources across the regions. 
#We need to have some roles for CloudFormation Stack Set
#It waits for 20 sec before creating the cloudformation stack set and stack set instances in order the roles to be available for use. 
#For further reading follow the link https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html#troubleshoot_general_eventual-consistency
resource "time_sleep" "iam_propagation" {
  create_duration = "20s"
  depends_on = [
    aws_iam_role.AWSCloudFormationStackSetAdministrationRole,
    aws_iam_role.AWSCloudFormationStackSetExecutionRole,
    aws_iam_role_policy.AWSCloudFormationStackSetExecutionRole_MinimumExecutionPolicy
  ]
}

Steps to Reproduce

terraform apply

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[aries1980]

I can confirm this is indeed the case with 4.65.0, although I am not convinced this is related to the AWS provider or Terraform itself. I can see the settings in the Terraform plan and turning on the debug log, I can see the settings are sent with the API request.

I have the same experience with failure_tolerance_percentage and max_concurrent_percentage, both are set to a non-zero number but they are both zero after terraform apply.

My snippet:

resource "aws_cloudformation_stack_set" "resource_explorer" {
  name             = "xzy"
  permission_model = "SERVICE_MANAGED"
  call_as          = "DELEGATED_ADMIN"

  auto_deployment {
    enabled                          = true
    retain_stacks_on_account_removal = false
  }

  template_body = file("${path.module}/template.yaml")

  operation_preferences {
    failure_tolerance_percentage = 99
    max_concurrent_percentage    = 50
    region_concurrency_type      = "PARALLEL"
  }
}

[evantlueck]

I did a lot of testing. I posted my findings here: #33170 (comment)