[ECS] Add the ability to delete an ASG capacity provider.

[coultn]

Currently, ECS Capacity Providers are immutable. Once created, they cannot be deleted. The change proposed here is to support deleting capacity providers.

[oleg-z]

Is there any way to new associate capacity provider with ASG which is associated with another capacity provider? I'm getting next error if I try to create new capacity provider for the same ASG:

botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the CreateCapacityProvider operation: The specified Auto Scaling group ARN is already being used by another capacity provider. Specify a unique Auto Scaling group ARN and try again.

It looks like currently you have to create new pair of capacity provider and asg everytime you want to adjust capacity provider settings.

[gmagella-ca]

+1 on this behaviour. Once you create a capacity provider and assign to ASG, they're married (til death do us part)... You can "deactivate" and remove the "link" between them, but you can't re-establish this again.

[chaitan94]

In my case, I am not even able to "deactivate" my capacity provider. When I look from Chrome's dev tools, aws seems to be returning 500 for this request. I thought it could be a temporary issue, but it's been over a week and the behavior is still the same. Is there any bug or is this expected?

For more details, I created the capacity provider with both Managed scaling and Managed termination protection disabled. Also I created it in ap-south-1 region if that helps.

[DigitallyBorn]

I'm stuck in this same scenario. This feels less like a feature request and more like a bug.

[pauldraper]

I've also run into this bug.

After deactivating, there appears to be no visible association with the autoscaling group, yet the autoscaling group cannot be used for any other capacity provider, even after removing the ASG tag.

Better get it right the first time !!!

[louisbuchbinder]

I'm excited to start using capacity providers, but this is clearly a bug. Please let us know where this issue fits on the roadmap.

[coultn]

Thanks everyone for your feedback. This is not a bug - it is functioning as designed. However, we do understand that the current design has some limitations, which is why we are working on the ability to delete capacity providers (this issue) as well as update the configuration of an existing capacity provider.

[0xGosu]

I also run into the same issue when trying out this Capacity Provider feature, once I deactivate the capacity provider, I can't create a new one link to the ASG any more. Please help fix this asap.

[wliao008]

+1 stuck on this this as well, kinda disappointing and frustrating experience.

[sdot257]

I'm trying to understand something. If this isn't a "bug," then what is the "proper" way of using this? I feel like I'm stuck on this and in order to move forward I have to recreate my ASG.

[JaswanthReddy]

I stuck into the problem of cluster not getting destroyed due to the bug/design of aws_ecs_capacity_provider not getting deleted.I have to bypass and directly delete ASG in order to delete the cluster.Clearly this stops from creating clusters from terraform or likes

[michael-robbins]

This clearly isn't working as provided, not being able to do one of:

• Delete a capacity provider
• Delete the link between a capacity provider and its associated ASG
• Reassign an ASG to another capacity provider

Is clearly a very wrong assumption and shows you have not thought out/designed the API with enough foresight.

+1 to getting the design and implementation of capacity providers built and deployed correctly

[rametta]

This is a problem for me too. Causing a lot of issues with my terraform scripts and making it impossible to actually use ECS on EC2 in an automated way.

[begetan]

I came here from the AWS support. The workaround is to create new capacity provider with the different name. Hope the team will rise this request, because Capacity Provider is really needed feature.

[kgyovai]

@begetan - It's not only the capacity provider that must be recreated. Since the capacity provider and the ASG are inextricably linked, a new ASG must be created for each new capacity provider.

@rametta - As a workaround for this when using Terraform, the name of the capacity provider can be dependent upon the name of the ASG. Then, if you need to make an update to the capacity provider, the ASG name can be updated. This feels super janky in practice but it's been working for me for the time being.

resource "aws_autoscaling_group" "ecs_autoscaling_group" {
  name_prefix               = join("-", [var.app_name, var.environment, "recreated"])
  max_size                  = 4
  min_size                  = 0
  health_check_grace_period = 180
  health_check_type         = "ELB"
  desired_capacity          = 0
  force_delete              = false
  protect_from_scale_in     = true
  launch_configuration      = aws_launch_configuration.ecs_launch_config.id
  vpc_zone_identifier       = data.terraform_remote_state.vpc.outputs.vpc_private_subnet_ids

  lifecycle {
    ignore_changes = [desired_capacity]
    create_before_destroy = true
  }
}

resource "aws_ecs_capacity_provider" "ecs_capacity_provider" {
  /* The immutable dependency of an ECS capacity provider on a specific ASG is a major pain point.
   * Since this relationship is enforced as 1-to-1, the name of the capacity provider should reflect
   * the name of the ASG.
   * See  https://github.com/aws/containers-roadmap/issues/632
   * Also https://github.com/aws/containers-roadmap/issues/633
   */
  name = join("-", [var.app_name, var.environment, "asg", aws_autoscaling_group.ecs_autoscaling_group.name])

  auto_scaling_group_provider {
    auto_scaling_group_arn         = aws_autoscaling_group.ecs_autoscaling_group.arn
    managed_termination_protection = "ENABLED"

    managed_scaling {
      maximum_scaling_step_size = 2
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      // Although poorly documented by AWS, target_capacity is a percentage (%).
      target_capacity           = 100
    }
  }

  tags = {
    App       = var.app_name
    Environment = var.environment
  }

}

[brettwgreen]

Good lord... I even deleted the entire cluster the thing was associated to, so it's not in any way even accessible through the console. And still it lingers and prevents me from making a new correctly configured capacity provider after I re-created the cluster. What a nightmare. It's kinda messed up my entire account.

Update: And cool... I deleted the scaling group (I'm creating a whole setup for new company, so nuking and starting over is not entirely insane) and now I've got a capacity provider pointing at a scaling group that no longer exists. I had my fingers crossed that removing the scaling group might clear things up... nope.

[piotrb]

Yep this is a bit insane .. especially with the inability to "Update" a capacity provider .. even its basic attributes .. so if you create a capacity provider with managed_termination_protection off initially and want to use it now .. you're kind of screwed ..

AWS guys .. can you give us a better idea what's going on here and a better ETA than "we're working on it" .. this just seems straight up broken ..

[dejonghe]

Thanks everyone for your feedback. This is not a bug - it is functioning as designed.

K, so this design flaw? Sorry to have to pile on but this is rough.

[jdb1441]

I agree, this is crazy. Having to delete an entire ASG is a ridiculous requirement to simply tweak a capacity provider parameter. This it a complete non-starter for any established environment where SLAs need to be adhered to.

[osiegmar]

This is not a bug - it is functioning as designed.

Somehow that reminds me of this: https://dilbert.com/strip/2010-03-14

Sorry, couldn't resist.

It would be very helpful if the current API could be improved so that tools like terraform could do their job as intended.

[eddgrant]

Hi @coultn , thanks for confirming that this is being worked on. We're keen to migrate over to using capacity providers, however are rather hesitant until this issue has been resolved, due to the operational headaches that come with the current status quo. Are you able to provide any further details as to when we might expect this work to hit GA? Thanks.

[eddgrant]

Hi @kgyovai , Thanks for posting your #632 (comment) workaround. I'm trying to replicate the approach you have taken , however I still get the capacity provider already exists error when I try to apply an update a capacity provider attribute.

I noticed that you mentioned:

if you need to make an update to the capacity provider, the ASG name can be updated

Are you doing something specific to trigger an update of the ASG name? I'm wondering if I need to change an attribute of the ASG at the same time as I change the attribute of the capacity provider?

[kgyovai]

@eddgrant - In the approach that I described above, the name of the capacity provider is dependent on the name of the ASG
name = join("-", [var.app_name, var.environment, "asg", aws_autoscaling_group.ecs_autoscaling_group.name])

Since the capacity provider cannot be updated and an ASG cannot be associated w/ multiple capacity providers, any change to the capacity provider requires creating a new capacity provider (which is subject of the current issue) and a new ASG. In order to ensure that a one-to-one relationship exists between the ASG and capacity provider, I have been making a simple increment a name prefix of the ASG anytime I need to change the capacity provider.

For example, if the name of my ASG was:
name_prefix = join("-", [var.app_name, var.environment, "1"])
It would then become:
name_prefix = join("-", [var.app_name, var.environment, "2"])

Note that since I'm using Terraform, I went with a name_prefix so that the name also includes a timestamp of when the ASG was created. This isn't a requirement for the approach that I'm describing but it does help to simplify the "book keeping" of numerous ASG -> capacity provider pairs.

The bad thing about this approach (believe me, I realize how many hoops are being jumped through by doing this as opposed to just have an update option for the capacity provider), is that a simple change to the ASG name prefix results in a cascading dependency of resource recreation.

aws_autoscaling_attachment.autoscaling_attachment must be replaced
aws_autoscaling_group.ecs_autoscaling_group must be replaced
aws_ecs_capacity_provider.ecs_capacity_provider must be replaced
aws_ecs_service.ecs_service must be replaced (because of the dependency on the capacity provider strategy)

So all of that just to create a new copy of the capacity provider.

The other "gotcha" to be aware of when using Terraform for this approach is resource lifecycle management. As shown below, specifying create_before_destroy on the ASG is required in order to ensure that the new ASG->capacity-provider pair are provisioned prior to deleting the old ASG.

  lifecycle {
    ignore_changes = [desired_capacity]
    create_before_destroy = true
  }

[coultn]

As always, we deeply value your feedback and strong desire to see this shipped. As the product manager who created the capacity provider and cluster auto scaling features, and who works closely with the team building it, I absolutely understand your frustration that the delete capacity provider API is not yet available. However, I can share that it is indeed coming soon, and we will provide updates here the moment we have more information to share.

[briancurt]

Some comments in this issue are just needlessly rude. We all want this fixed and we already know they are working on it. If anything, take this as a lesson to thorougly test new features in other environments before applying them in production.

[jwhipp]

I know this isn't great but for people who deactivated a scaling provider and just want to get them back, you can as long as you choose the exact same name and auto-scaling group. I didn't know I couldn't make new ones and hit this issue. This at least allowed me to get them back even know they are not named the way I want and I have to live with it for now. :)

[jdb1441]

Hey @coultn , thank you for the update. We are very excited for this to be released.

Will the ability to tweak a Capacity Provider Target Capacity be included with this release? Given the many variables that play into the about of hosts desired it can take a bit of tweaking to get the free capacity at the level that we want it.

[nathando]

Please let us know the status of the update as well. If we deactivate a capacity provider, the link between it and the ASG was still there. We cannot create new Capacity Provider with the same ASG which is very odd, especially there is no page to edit the Capacity Provider as well.

[sean-kang]

I just wanted to update the target capacity in a capacity provider. Now I have to replace the entire auto scaling group. This is simply stupid.

[erichydrick]

Any ETA on a release date? This is blocking needed infrastructure work for us.

[KMahankali]

Any ETA on a release data yet?

[guillaumesmo]

Version 2.695.0
feature: ECS: This release adds support for deleting capacity providers.

is this what we were waiting for?

[ziggythehamster]

Looks like it dropped today:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/document_history.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/asg-capacity-providers.html#asg-capacity-providers-delete-capacity-provider

I'll go check if @terraform-providers has a ticket. I wonder also if the feature of updating constraints/strategies is out of beta. https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecs/update-service.html indicates it is still beta, but the ECS docs disagree.

[ziggythehamster]

Ah, I misread - you can delete, but not update. Still better than before, assuming you can reuse the same name. Then a destroy/create becomes possible as an update.

[dcelasun]

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/asg-capacity-providers.html#asg-capacity-providers-delete-capacity-provider

Without CloudFormation support, of course.

[pauldraper]

Without CloudFormation support, of course.

But next day support in Terraform (on a fork). hashicorp/terraform-provider-aws#13740

[anoopkapoor]

Delete capacity providers functionality is now available through cli/sdk/api/console in all regions!
https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-ecs-capacity-providers-support-delete-functionality/

[chrizonline]

So all of that just to create a new copy of the capacity provider.

The other "gotcha" to be aware of when using Terraform for this approach is resource lifecycle management. As shown below, specifying create_before_destroy on the ASG is required in order to ensure that the new ASG->capacity-provider pair are provisioned prior to deleting the old ASG.

  lifecycle {
    ignore_changes = [desired_capacity]
    create_before_destroy = true
  }

Hi @eddgrant, does this mean that there will be plenty of ghost capacity provider left in aws?

[chrizonline]

Delete capacity providers functionality is now available through cli/sdk/api/console in all regions!
https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-ecs-capacity-providers-support-delete-functionality/

Hi @anoopkapoor, I tried deleting with aws cli but im getting the following error.

> aws ecs delete-capacity-provider --capacity-provider arn:aws:ecs:ap-southeast-1:...:capacity-provider/myCapacityProvider
> Invalid choice: 'delete-capacity-provider', maybe you meant:

  * create-capacity-provider
  * describe-capacity-providers

aws cli version

aws --version
aws-cli/2.0.4 Python/3.7.4 Darwin/19.5.0 botocore/2.0.0dev8

[anoopkapoor]

Delete capacity providers functionality is now available through cli/sdk/api/console in all regions!
https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-ecs-capacity-providers-support-delete-functionality/

Hi @anoopkapoor, I tried deleting with aws cli but im getting the following error.

> aws ecs delete-capacity-provider --capacity-provider arn:aws:ecs:ap-southeast-1:...:capacity-provider/myCapacityProvider
> Invalid choice: 'delete-capacity-provider', maybe you meant:

  * create-capacity-provider
  * describe-capacity-providers

aws cli version

aws --version
aws-cli/2.0.4 Python/3.7.4 Darwin/19.5.0 botocore/2.0.0dev8

Hi! Your CLIv2 version is from March. If you update that to the latest version, it will work just fine.

[chrizonline]

Delete capacity providers functionality is now available through cli/sdk/api/console in all regions!
https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-ecs-capacity-providers-support-delete-functionality/

Hi @anoopkapoor, I tried deleting with aws cli but im getting the following error.

> aws ecs delete-capacity-provider --capacity-provider arn:aws:ecs:ap-southeast-1:...:capacity-provider/myCapacityProvider
> Invalid choice: 'delete-capacity-provider', maybe you meant:

  * create-capacity-provider
  * describe-capacity-providers

aws cli version

aws --version
aws-cli/2.0.4 Python/3.7.4 Darwin/19.5.0 botocore/2.0.0dev8

Hi! Your CLIv2 version is from March. If you update that to the latest version, it will work just fine.

Hi @anoopkapoor, thank you it works!!!!

[gblikas]

@anoopkapoor Do you know if there is a corresponding issue registered with Terraform? It still seems to me that it's the case that even in terraform version

terraform --version                      
Terraform v0.14.2
+ provider registry.terraform.io/hashicorp/aws v3.20.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

this issue still persists. I tried searching through the issues on their github issues, but wasn't able to find anything.

[rmccarthy-ellevation]

Can we get a force delete so that deleted caps don't just hang around. It's annoying when testing or building a pipeline to have to change the name everytime.