ECS Service can't update desired replicas when Blue Green deployment is enabled

[MehdiZonjy]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.26

• provider.aws v2.65.0
• provider.template v2.1.2

Affected Resource(s)

• resource aws_ecs_service

Terraform Configuration Files

• ECS with bluegreen deployment example in Terraform docs.
• or you can use this module which provisions ECS Service, CodeDeploy project and ALB (due to aws api eventual consistency you may have to retry on failure)

Debug Output

• plan logs

• apply logs

Expected Behavior

Should be able to update ECS Service desired_replicas even when deployment_controller.type is set to CODE_DEPLOY

Actual Behavior

• API call fails with the following message:

Error: Error updating ECS Service (arn:aws:ecs:us-east-1:708286315144:service/test/policies-service-v2): InvalidParameterException: Unable to update network parameters on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment.

Steps to Reproduce

• Create new ECS Service with deployment_controller.type = CODE_DEPLOY
• Change Service desired_count and attempt to apply changes again

Important Factoids

• Despite having no changes being made to network_configuration property, TF included it in the payload to update-service request

2020/06/08 04:05:32 [DEBUG] module.ecs.module.policies_service-v2.aws_ecs_service.service: applying the planned Update change
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4: 2020/06/08 04:05:32 [DEBUG] Updating ECS Service (arn:aws:ecs:us-east-1:708286315144:service/test/policies-service-v2): {
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:   Cluster: "arn:aws:ecs:us-east-1:708286315144:cluster/test",
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:   DesiredCount: 0,
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:   ForceNewDeployment: false,
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:   NetworkConfiguration: {
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:     AwsvpcConfiguration: {
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:       AssignPublicIp: "DISABLED",
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:       SecurityGroups: ["sg-02848434002f782f9"],
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:       Subnets: ["subnet-014f47760b389f8db","subnet-03cdd43d8265511be","subnet-0f67bb732709250c6"]
2020-06-08T04:05:32.935Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:     }
2020-06-08T04:05:32.936Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:   },
2020-06-08T04:05:32.936Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4:   Service: "arn:aws:ecs:us-east-1:708286315144:service/test/policies-service-v2"
2020-06-08T04:05:32.936Z [DEBUG] plugin.terraform-provider-aws_v2.65.0_x4: }

• TF Plan doesn't detect any change to network_configuration
• This problem isn't only triggered by changes to desired_count and can by reproduced by making changes to task_placement_constaints too

References

For services using the blue/green (CODE_DEPLOY ) deployment controller, only the desired count, deployment configuration, task placement constraints and strategies, and health check grace period can be updated using this API. If the network configuration, platform version, or task definition need to be updated, a new AWS CodeDeploy deployment should be created. For more information, see CreateDeployment in the AWS CodeDeploy API Reference .

• ecs update-service

[jooohn]

I encountered the same problem. Including network_configuration as ignore_changes doesn't work - network_configuration change doesn't appear in the diff view, but it is included in the API call request when apply.

Looking into the debug log, I found the order of the subnets in the update request is different from the value of the current state.

networkConfiguration of current state (ecs/DescribeServices)

{"awsvpcConfiguration":{"assignPublicIp":"ENABLED","securityGroups":["sg-09b7377097c0c97fd"],"subnets":["subnet-012c6b3edd7abd4d9","subnet-0f5c632fc7e9db0ab","subnet-041643c250580e1dc"]}}

networkConfiguration of update request (ecs/UpdateService)

{"awsvpcConfiguration":{"assignPublicIp":"ENABLED","securityGroups":["sg-09b7377097c0c97fd"],"subnets":["subnet-041643c250580e1dc","subnet-012c6b3edd7abd4d9","subnet-0f5c632fc7e9db0ab"]}}

The attribute type of subnets is TypeSet, so the two values should be the same.

https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_ecs_service.go#L238

However, it seems d.hasChange("network_configuration") returns true somehow since the request includes the networkConfiguration attribute.

https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_ecs_service.go#L1027

I'm not familiar with the codebase so I might look at unrelated place though.

[marcovergueira]

I'm having the same issue, trying to add tags to a service.

In the plan only appears the tags being changed.

During the apply I get this error: "InvalidParameterException: Unable to update network parameters on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment."

Including network_configuration as ignore_changes is not helping.

[wmaroy]

Having the same issues while trying to update the desired count of tasks, any updates on this?

[karan9nov]

Having the same issue when trying to update the platform version.

[hanswesterbeek]

Same here, when trying to modify something as harmless as the tags.

[Reli4ble]

Hi, same Problem here when i changed desired_count over Terraform:

Error: Error updating ECS Service InvalidParameterException: Unable to update network parameters on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment.

[Reli4ble]

Hi all,

what about updates on this issue ?

This is very important for us. We are running terraform-pipelines in production with ecs_service and CODE_DEPLOY deployment controller. Updates on the service with terraform are not possible over this way !

[berchev]

Hello Team,

Do you have any update/ETA on this issue?

We have TF Cloud customer (Team & Governance) affected by the same issue .
Thank you in advance!

Regards,
Georgi Berchev

[htquanq]

I tried to use CODE_DEPLOY with terraform but got either unable to update task definition or unable to update network configuration. I can't deploy anything. This is nightmare.

[joenoon]

FWIW, the way I was able to control desired_count is by adding an autoscaling group. I'm not using it to autoscale, since I have the min/max set to the same number and the cpu target_value 99. It just acts as a way to modify desired_count which seems to play well with terraform and the CODE_DEPLOY deployment_controller type:

resource "aws_iam_service_linked_role" "ecs_application_autoscaling" {
  aws_service_name = "ecs.application-autoscaling.amazonaws.com"
  description      = "Allows Application Auto Scaling to call ECS and CloudWatch on your behalf."
}

resource "aws_appautoscaling_target" "target" {
  # THIS CONTROLS THE DESIRED_COUNT
  max_capacity       = var.running_max
  min_capacity       = var.running_min
  resource_id        = "service/${aws_ecs_cluster.cluster.name}/${aws_ecs_service.app.name}"
  role_arn           = aws_iam_service_linked_role.ecs_application_autoscaling.arn
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

resource "aws_appautoscaling_policy" "cpu_tracking" {
  name               = "${aws_ecs_cluster.cluster.name}_${aws_ecs_service.app.name}_cpu_tracking"
  policy_type        = "TargetTrackingScaling"
  resource_id        = "service/${aws_ecs_cluster.cluster.name}/${aws_ecs_service.app.name}"
  scalable_dimension = aws_appautoscaling_target.target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.target.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }

    target_value       = 99
    scale_in_cooldown  = 300
    scale_out_cooldown = 60
  }
}

[dirk39]

I'm trying to replicate the issue with acceptance tests here. I hope to open a PR about this

[calimonk]

Could the PR made earlier this year, which references this issue, be reviewed? It's a fairly small change. We believe this to be a good way to address this, having resorted to using our own fork with that PR to address this for now and can confirm this resolves it for us.

For visibility @bflad @gdavison

hashicorp/terraform-plugin-sdk#711

[breathingdust]

Hi all 👋 Just letting you know that this is issue is featured on this quarters roadmap. If a PR exists to close the issue a maintainer will review and either make changes directly, or work with the original author to get the contribution merged. If you have written a PR to resolve the issue please ensure the "Allow edits from maintainers" box is checked. Thanks for your patience and we are looking forward to getting this merged soon!

[dirk39]

Hi @breathingdust, I'm trying to replicate the issue with a test, but it currently seems to work as expected. I'll try different changes to raise the error.

[dirk39]

Hi @breathingdust, I've written the test case that replicates the issue. When I try to update the desired_count, the function resourceServiceUpdate says I'm also changing the NetworkConfiguration. Like @calimonk says, the solution to this issue is the release of the version v2.9.0 of the terraform-sdk-plugin. So I can open the PR with the test case that replicates the bug and update the go.mod when the terraform-plugin-sdk is released.

Cheers

[ewbankkit]

@dirk39 v2.9.0 of terraform-plugin-sdk has now been released and integrated into the AWS Provider.
Are the commits you mention on https://github.com/dirk39/terraform-provider-aws/commits/r/ecs_service_update_with_bg_deployment valid for testing - i.e. can I (or you) open a PR with those changes?
Thanks.

[dirk39]

@ewbankkit let me update the branch and add further tests to cover all the fields allowed by the APIs

[ewbankkit]

#22034 validates that this has been fixed via terraform-plugin-sdk@v2.9.0 which was released in v3.68.0 of the Terraform AWS Provider.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.