Dependency violation when deleting a Subnet: ENI still attached to ELB

[hashibot]

This issue was originally opened by @rokka-n as hashicorp/terraform#6318. It was migrated here as part of the provider split. The original body of the issue is below.

---

Hi,

I'm currently seeing terraform tries to destroy a subnet that was used in ELB and fails. I think its because ENI interface that belongs to the subnet is still attached to ELB. The subnet itself is removed from ELB correctly via ASG routine.

Terraform Version

v0.6.14

Actual Behavior

When I remove a subnet from vars, terraform correctly removes it from ASG/ELB. But it hangs at the subnet destroying step.

Workaround is to manually detach ENI(s) that belong to subnet and delete it.

aws_autoscaling_group.wp_asg: Refreshing state... (ID: wp-staging-qmlts6tb7fb3va6r3xe5glsie4-asg)
aws_subnet.blog_subnets.2: Refreshing state... (ID: subnet-c21543ff)
aws_subnet.blog_subnets.2: Destroying...

aws_subnet.blog_subnets.2: Destruction complete

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Steps to Reproduce

Create an ELB with ASG that uses few subnets. Remove 1 subnet.
Watch ASG/ELB gets updated properly, but terraform hangs on subnet termination. If I try to delete subnet manually in aws interface - it complaines about attached ENI.

resource "aws_elb" "wp_elb" {
name = foo
subnets = ["${split(",", join(",", aws_subnet.app_subnets.*.id))}"]
}

Could be related to hashicorp/terraform#934

[radeksimko]

Hi @rokka-n
just to clarify - this is not a dependency problem, but rather eventual consistency or generally a problem caused by undocumented actions that happen behind the scenes in AWS. Some services manage their own ENIs but do so asynchronously with delays significant enough to cause problems for any tool managing the rest of the infrastructure.

#1036 was merged a while back to address very similar problem - detaching ENIs on ELB deletion. It probably wouldn't be too difficult to crib something from that PR and use it in this context. I might take a look at it at some point.

[rokka-n]

Thanks for looking into this, I'd like to close this issues since I haven't seen this happened for some time.

Probably just weird edge case.

[awilkins]

Not just a weird edge case, this happens reliably for us with any plan that includes a delete/recreate of a subnet attached to an ELB.

I would read it as a bug determining the dependency graph because the subnet identifier is referenced directly in the ELB concerned. We even tried explicitly declaring a depends_on relationship from the ELB to the subnet - it still tries to delete the subnet without first deleting the ELB or detaching the subnet from it. Is there some kind of special case in the code that treats ELB -> subnet dependencies as "soft"?

Further comment :

If you run into this problem and manually trash the ELB during execution (while the "destroying subnet" messages are still scrolling past)... the plan successfully resumes, but then errors out when it tries to unregister the subnet from the ELB.

So the provider is aware that the ELB has a dependency on the subnet but is attempting to delete the subnet before this dependency is removed (either by unregistering the subnet from the ELB or by just destroying the ELB and recreating it after the subnet has been recreated).

[radeksimko]

I managed to reliably reproduce this using the following config:
https://gist.github.com/radeksimko/4132453a6cf6edc3ccb61c8bbe31e076

As @awilkins mentioned already this is triggered by deletion of a subnet which is associated with an LB.

Sadly though there doesn't seem to be anything we can realistically do from Terraform's perspective. Even if we got the ordering of operations right and managed to disassociate the subnet from LB before attempting to delete it, Amazon wouldn't let us detach these ENIs as they hold them until the LB is fully destroyed. You will get the following error when attempting to do so:

An error occurred (AuthFailure) when calling the DetachNetworkInterface operation: You do not have permission to access the specified resource.

I am not sure if this is by design. Perhaps we could get some explanation from the LB/EC2 teams?

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.