aws_subnet destroy times out

[random930]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.2

• provider registry.terraform.io/hashicorp/aws v3.7.0

Affected Resource(s)

• aws_subnet

Terraform Configuration Files

I'm including the plan here instead of the HCL. The issue is during DESTROY and not creation of resources.

Terraform will perform the following actions:

  # aws_ecs_service.service will be created
  + resource "aws_ecs_service" "service" {
      ...
    }

  # aws_efs_mount_target.chunks_external1 will be created
  + resource "aws_efs_mount_target" "chunks_external1" {
      ...
    }

  # aws_efs_mount_target.chunks_external2 will be created
  + resource "aws_efs_mount_target" "chunks_external2" {
      ...
    }

  # aws_elasticsearch_domain.esdomain will be updated in-place
  ~ resource "aws_elasticsearch_domain" "esdomain" {
        ...
    }

  # aws_lb.main will be created
  + resource "aws_lb" "main" {
      ...
    }

  # aws_lb_listener.main will be created
  + resource "aws_lb_listener" "main" {
      ...
    }

  # aws_lb_target_group.main will be created
  + resource "aws_lb_target_group" "main" {
      ...
    }

  # aws_network_acl.external will be updated in-place
  ~ resource "aws_network_acl" "external" {
        ...
    }

  # aws_route_table_association.external1_subnet_to_internet will be created
  + resource "aws_route_table_association" "external1_subnet_to_internet" {
      ...
    }

  # aws_route_table_association.external2_subnet_to_internet will be created
  + resource "aws_route_table_association" "external2_subnet_to_internet" {
      ...
    }

  **# aws_subnet.external will be destroyed
  - resource "aws_subnet" "external" {
      - arn                             = "arn:aws:ec2:us-east-2:4730...055:subnet/subnet-0985f28cca9cdb006" -> null
      - assign_ipv6_address_on_creation = false -> null
      - availability_zone               = "us-east-2a" -> null
      - availability_zone_id            = "use2-az1" -> null
      - cidr_block                      = "172.27.65.0/24" -> null
      - id                              = "subnet-0985f28cca9cdb006" -> null
      - map_public_ip_on_launch         = false -> null
      - owner_id                        = "473....86055" -> null
      - tags                            = {} -> null
      - vpc_id                          = "vpc-0aa7f45fdcd31dfab" -> null
    }**

  # aws_subnet.external1 will be created
  + resource "aws_subnet" "external1" {
      + arn                             = (known after apply)
      + assign_ipv6_address_on_creation = false
      + availability_zone               = "us-east-2a"
      + availability_zone_id            = (known after apply)
      + cidr_block                      = "172.27.65.0/24"
      + id                              = (known after apply)
      + ipv6_cidr_block_association_id  = (known after apply)
      + map_public_ip_on_launch         = false
      + owner_id                        = (known after apply)
      + vpc_id                          = "vpc-0aa7f45fdcd31dfab"
    }

  # aws_subnet.external2 will be created
  + resource "aws_subnet" "external2" {
      + arn                             = (known after apply)
      + assign_ipv6_address_on_creation = false
      + availability_zone               = "us-east-2b"
      + availability_zone_id            = (known after apply)
      + cidr_block                      = "172.27.65.0/24"
      + id                              = (known after apply)
      + ipv6_cidr_block_association_id  = (known after apply)
      + map_public_ip_on_launch         = false
      + owner_id                        = (known after apply)
      + vpc_id                          = "vpc-0aa7f45fdcd31dfab"
    }

Plan: 10 to add, 2 to change, 1 to destroy.

Debug Output

The relevant part shows that TF is ignoring the error returned from AWS:

2020-09-18T16:08:33.693-0700 [DEBUG] plugin.terraform-provider-aws_v3.7.0_x5: -----------------------------------------------------
2020-09-18T16:08:33.693-0700 [DEBUG] plugin.terraform-provider-aws_v3.7.0_x5: 2020/09/18 16:08:33 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2020-09-18T16:08:33.693-0700 [DEBUG] plugin.terraform-provider-aws_v3.7.0_x5: <Response><Errors><Error><Code>DependencyViolation</Code><Message>The subnet 'subnet-0985f28cca9cdb006' has dependencies and cannot be deleted.</Message></Error></Errors><RequestID>2179779f-18e0-4a17-b3ff-5c6882d6f906</RequestID></Response>
2020-09-18T16:08:33.693-0700 [DEBUG] plugin.terraform-provider-aws_v3.7.0_x5: 2020/09/18 16:08:33 [DEBUG] [aws-sdk-go] DEBUG: Validate Response ec2/DeleteSubnet failed, attempt 0/25, error DependencyViolation: The subnet 'subnet-0985f28cca9cdb006' has dependencies and cannot be deleted.
2020-09-18T16:08:33.693-0700 [DEBUG] plugin.terraform-provider-aws_v3.7.0_x5: 	status code: 400, request id: 2179779f-18e0-4a17-b3ff-5c6882d6f906
2020-09-18T16:08:33.693-0700 [DEBUG] plugin.terraform-provider-aws_v3.7.0_x5: 2020/09/18 16:08:33 [TRACE] Waiting 10s before next try
aws_subnet.external: Still destroying... [id=subnet-0985f28cca9cdb006, 20s elapsed]

Expected Behavior

Either Terraform should show me the error. Or, better yet, determine what is the dependency and help me destroy it.

Actual Behavior

Times out after 20 minutes.

[random930]

After a bit more digging, I found that it's the elasticsearch domain that has an ENI in the subnet being destroy. That's getting in the way of the destroy. If you notice in my original post, there's a change being done to the elasticsearch domain. Here's more details on it:

  ~ resource "aws_elasticsearch_domain" "esdomain" {
        ...

      ~ vpc_options {
            availability_zones = [
                "us-east-2a",
            ]
            security_group_ids = [
                "sg-0ff1213eec08abc81",
            ]
          ~ subnet_ids         = [
              - "subnet-0985f28cca9cdb006",
            ] -> (known after apply)
            vpc_id             = "vpc-0aa7f45fdcd31dfab"
        }
    }

So, TF should first make the subnet change on the ES before trying to destroy the subnet. To do that, it needs to create the two new subnets (because I'm moving the ES domain to them). However, that's not possible, because one of the two new subnets has the same CIDR block as the old subnet. So there's somewhat of a circular issue here.

[random930]

My way around this was to change the CIDR blocks of the new subnets.

Run apply with -target for the two new subnets (so they get created).
Run apply with -target on the esdomain (so it gets moved to the new subnet).

Then, destroying the subnet still took 9 minutes:

aws_subnet.external: Still destroying... [id=subnet-0985f28cca9cdb006, 9m20s elapsed]

Why? Because Elasticsearch was stuck trying to move the domain between subnets. The way to get it out of that state was to upgrade the domain to the most recent version. (the update process forced Elasticsearch to detach the network interface from the old subnet... freeing it to be deleted)

Ah what a day :)

[gdavison]

Hi @random930. Unfortunately, Terraform doesn't support those types of dependencies between resources. There is some thought being put into how Terraform could handle this. For example hashicorp/terraform#16065, hashicorp/terraform#8099.

Since this applies to the Terraform tool as a whole, I'm going to close this ticket. Their repository can be found at https://github.com/hashicorp/terraform

[random930]

What about the ability to get the actual error message, instead of waiting for a timeout? (the original post in this ticket)

…

On Mon, Sep 21, 2020 at 2:49 PM Graham Davison ***@***.***> wrote: Hi @random930 <https://github.com/random930>. Unfortunately, Terraform doesn't support those types of dependencies between resources. There is some thought being put into how Terraform could handle this. For example hashicorp/terraform#16065 <hashicorp/terraform#16065>, hashicorp/terraform#8099 <hashicorp/terraform#8099>. Since this applies to the Terraform tool as a whole, I'm going to close this ticket. Their repository can be found at https://github.com/hashicorp/terraform — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15230 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOL3ATJWB643AI3Y6QKP77DSG7C6DANCNFSM4RSUIJ2Q> .

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!