Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

destroy failing in AWS #1203

Closed
jaxxstorm opened this issue Mar 14, 2015 · 7 comments
Closed

destroy failing in AWS #1203

jaxxstorm opened this issue Mar 14, 2015 · 7 comments
Labels
bug provider/aws waiting-response An issue/pull request is waiting for a response from the community

Comments

@jaxxstorm
Copy link

Looks like the latest build still having some issues with destroys:

Do you really want to destroy?
  Terraform will delete all your managed infrastructure.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

module.ap_vpc.aws_vpc.vpc: Refreshing state... (ID: vpc-f0ae0895)
module.ap_subnets.aws_subnet.dmz_subnet: Refreshing state... (ID: subnet-22bb2947)
module.ap_subnets.aws_subnet.console01_subnet: Refreshing state... (ID: subnet-2dbb2948)
module.ap_subnets.aws_subnet.web01_subnet: Refreshing state... (ID: subnet-26bb2943)
module.ap_subnets.aws_subnet.storage01_subnet: Refreshing state... (ID: subnet-21bb2944)
module.ap_subnets.aws_subnet.db01_subnet: Refreshing state... (ID: subnet-20bb2945)
module.ap_subnets.aws_subnet.infra_subnet: Refreshing state... (ID: subnet-23bb2946)
module.ap_subnets.aws_subnet.app01_subnet: Refreshing state... (ID: subnet-2cbb2949)
module.ap_securitygroups.aws_security_group.nat: Refreshing state... (ID: sg-75d6af10)
module.ap_securitygroups.aws_security_group.infra: Refreshing state... (ID: sg-6ad6af0f)
module.ap_securitygroups.aws_security_group.console01: Refreshing state... (ID: sg-74d6af11)
module.ap_securitygroups.aws_security_group.storage01: Refreshing state... (ID: sg-6bd6af0e)
module.ap_securitygroups.aws_security_group.db01: Refreshing state... (ID: sg-69d6af0c)
module.ap_securitygroups.aws_security_group.app01: Refreshing state... (ID: sg-77d6af12)
module.ap_securitygroups.aws_security_group.web01: Refreshing state... (ID: sg-68d6af0d)
module.ap_kms.aws_instance.kms: Refreshing state... (ID: i-db8fe73f)
module.ap_bigip.aws_instance.bigip: Refreshing state... (ID: i-ad8fe749)
module.ap_smtp.aws_instance.smtp: Refreshing state... (ID: i-ae8fe74a)
module.ap_nexus.aws_instance.nexus: Refreshing state... (ID: i-a98fe74d)
module.ap_dnsslave.aws_instance.dnsslave: Refreshing state... (ID: i-a88fe74c)
module.ap_build.aws_instance.build: Refreshing state... (ID: i-cd8fe729)
module.ap_puppet.aws_instance.puppet: Refreshing state... (ID: i-af8fe74b)
module.ap_nat.aws_instance.nat: Refreshing state... (ID: i-da8fe73e)
module.ap_nat.aws_internet_gateway.gw: Refreshing state... (ID: igw-a128e9c4)
module.ap_nat.aws_route_table.internet: Refreshing state... (ID: rtb-0cbb0c69)
module.ap_nat.aws_route_table_association.internet: Refreshing state... (ID: rtbassoc-1e8b307b)
module.ap_nat.aws_eip.nat: Refreshing state... (ID: eipalloc-55e91430)
module.ap_opsrouting.aws_route_table.ops: Refreshing state... (ID: rtb-0bbb0c6e)
module.ap_opsrouting.aws_main_route_table_association.ops: Refreshing state... (ID: rtbassoc-ed8b3088)
aws_vpc.vpc: Destroying...
aws_vpc.vpc: Error: 1 error(s) occurred:

* Error deleting VPC: The vpc 'vpc-f0ae0895' has dependencies and cannot be deleted.
aws_security_group.infra: Destroying...
aws_security_group.storage01: Destroying...
aws_security_group.app01: Destroying...
aws_security_group.web01: Destroying...
aws_security_group.db01: Destroying...
aws_security_group.nat: Destroying...
aws_security_group.console01: Destroying...
aws_security_group.app01: Destruction complete
aws_security_group.web01: Destruction complete
aws_security_group.db01: Destruction complete
aws_security_group.storage01: Destruction complete
aws_security_group.infra: Error: 1 error(s) occurred:

* resource sg-6ad6af0f has a dependent object
aws_security_group.nat: Error: 1 error(s) occurred:

* resource sg-75d6af10 has a dependent object
aws_security_group.console01: Error: 1 error(s) occurred:

* resource sg-74d6af11 has a dependent object
aws_main_route_table_association.ops: Destroying...
aws_main_route_table_association.ops: Destruction complete
aws_route_table.ops: Destroying...
aws_route_table.ops: Destruction complete
Error applying plan:

2 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* Error deleting VPC: The vpc 'vpc-f0ae0895' has dependencies and cannot be deleted.
* 3 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* resource sg-6ad6af0f has a dependent object
* 1 error(s) occurred:

* 1 error(s) occurred:

* resource sg-75d6af10 has a dependent object
* 1 error(s) occurred:

* 1 error(s) occurred:

* resource sg-74d6af11 has a dependent object

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

it seems to me that the issue here lies around not deleting EC2 instances inside a security group or subnet BEFORE deleting those security groups. Perhaps there needs to be dependencies on that worked in?
@jaxxstorm
Copy link
Author

I manually removed the EC2 instances in the VPC, and the plan executed differently, but still failed:

module.ap_vpc.aws_vpc.vpc: Refreshing state... (ID: vpc-f0ae0895)
module.ap_subnets.aws_subnet.console01_subnet: Refreshing state... (ID: subnet-2dbb2948)
module.ap_subnets.aws_subnet.storage01_subnet: Refreshing state... (ID: subnet-21bb2944)
module.ap_subnets.aws_subnet.db01_subnet: Refreshing state... (ID: subnet-20bb2945)
module.ap_subnets.aws_subnet.infra_subnet: Refreshing state... (ID: subnet-23bb2946)
module.ap_subnets.aws_subnet.dmz_subnet: Refreshing state... (ID: subnet-22bb2947)
module.ap_subnets.aws_subnet.app01_subnet: Refreshing state... (ID: subnet-2cbb2949)
module.ap_subnets.aws_subnet.web01_subnet: Refreshing state... (ID: subnet-26bb2943)
module.ap_securitygroups.aws_security_group.infra: Refreshing state... (ID: sg-6ad6af0f)
module.ap_securitygroups.aws_security_group.nat: Refreshing state... (ID: sg-75d6af10)
module.ap_securitygroups.aws_security_group.console01: Refreshing state... (ID: sg-74d6af11)
module.ap_puppet.aws_instance.puppet: Refreshing state... (ID: i-af8fe74b)
module.ap_dnsslave.aws_instance.dnsslave: Refreshing state... (ID: i-a88fe74c)
module.ap_build.aws_instance.build: Refreshing state... (ID: i-cd8fe729)
module.ap_smtp.aws_instance.smtp: Refreshing state... (ID: i-ae8fe74a)
module.ap_kms.aws_instance.kms: Refreshing state... (ID: i-db8fe73f)
module.ap_bigip.aws_instance.bigip: Refreshing state... (ID: i-ad8fe749)
module.ap_nexus.aws_instance.nexus: Refreshing state... (ID: i-a98fe74d)
module.ap_nat.aws_instance.nat: Refreshing state... (ID: i-da8fe73e)
module.ap_nat.aws_internet_gateway.gw: Refreshing state... (ID: igw-a128e9c4)
module.ap_nat.aws_route_table.internet: Refreshing state... (ID: rtb-0cbb0c69)
module.ap_nat.aws_eip.nat: Refreshing state... (ID: eipalloc-55e91430)
module.ap_nat.aws_route_table_association.internet: Refreshing state... (ID: rtbassoc-1e8b307b)
aws_vpc.vpc: Destroying...
aws_vpc.vpc: Error: 1 error(s) occurred:

* Error deleting VPC: The vpc 'vpc-f0ae0895' has dependencies and cannot be deleted.
aws_security_group.nat: Destroying...
aws_security_group.infra: Destroying...
aws_security_group.console01: Destroying...
aws_security_group.nat: Destruction complete
aws_security_group.console01: Destruction complete
aws_security_group.infra: Destruction complete
aws_route_table_association.internet: Destroying...
aws_route_table_association.internet: Destruction complete
aws_route_table.internet: Destroying...
aws_route_table.internet: Destruction complete
aws_internet_gateway.gw: Destroying...
aws_internet_gateway.gw: Destruction complete
Error applying plan:

1 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* Error deleting VPC: The vpc 'vpc-f0ae0895' has dependencies and cannot be deleted.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

@jaxxstorm jaxxstorm changed the title destroy hanging destroy failing Mar 14, 2015
@jaxxstorm jaxxstorm changed the title destroy failing destroy failing in AWS Mar 14, 2015
@mitchellh
Copy link
Contributor

This looks like the same issue as #276. That issue might be too old but there is definitely one open. This is due to the eventual consistency of amazon's API, the "destroy" was successful but Terraform is too fast and moves on before it actually processes it internally. We have to just do a poll on the GET until it 404s. (We've done this for some other resources, a sadness)

@jaxxstorm
Copy link
Author

@mitchellh I personally don't think it is, but you know better than me. In #276 it seems to be destroying the instances first, but here it tries to destroy the security groups - the AWS instances don't get destroyed at all during the output

@catsby might be able to comment as I reckon this might be related to the porting to aws-sdk-go ? It definitely worked before on 0.3.7 with the same plan.

@radeksimko
Copy link
Member

The AWS API is not really being helpful in these cases as we don't know what dependencies is it waiting for, but it might be as well related to #497 (although it has a different error message).

In case of VPC it can really be almost any resource which has IP address assigned from that VPC subnet and/or any subnet relationship - EC2, ELB, ASG, RDS, ...

@catsby
Copy link
Contributor

catsby commented Mar 16, 2015

Hey @jaxxstorm do you have a minimal config that reproduces this? If you do, that would be really helpful. Be sure to strip any sensitive / secrets.

There was similar issues in #1189 that was resolved in part by using depends_on to explicitly set the order of deleting. I'd like to see if that helps here.

@catsby catsby added bug waiting-response An issue/pull request is waiting for a response from the community provider/aws labels Mar 16, 2015
@jaxxstorm
Copy link
Author

Hi @catsby - I don't think I can provide the config because we use several self written modules to create our VPC config. The problem arises when using these modules. We have a module to create a NAT instance, then separate modules to create our routing tables and individual instances separately. I can probably provide it in private, but not publicly.

I agree that using depends_on would fix this, but this doesn't yet work across modules so it's not an option here unfortunately.

Should we reopen this bug, do you think?

@catsby catsby mentioned this issue Mar 17, 2015
Closed
@ghost
Copy link

ghost commented May 4, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators May 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug provider/aws waiting-response An issue/pull request is waiting for a response from the community
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mitchellh @catsby @radeksimko @jaxxstorm