Skip to content

Commit

Permalink
Fix issue where VPCs weren't deleted (hashicorp#628)
Browse files Browse the repository at this point in the history
There's a race condition were even though the NAT gateway was just
deleted, AWS won't delete the VPC because it still sees the NAT gateway
as a dependency. This fix retries the VPC delete up to 10 times (waiting
1s between retries).
  • Loading branch information
lkysow authored Aug 20, 2021
1 parent b8e13ec commit 1283392
Showing 1 changed file with 20 additions and 6 deletions.
26 changes: 20 additions & 6 deletions charts/consul/hack/aws-acceptance-test-cleanup/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -276,14 +276,28 @@ func realMain(ctx context.Context) error {
fmt.Printf("ELB: Destroyed [id=%s]\n", *elbDescrip.LoadBalancerName)
}

// Delete VPC.
// Delete VPC. Sometimes there's a race condition where AWS thinks
// the VPC still has dependencies but they've already been deleted so
// we may need to retry a couple times.
fmt.Printf("VPC: Destroying... [id=%s]\n", *vpcID)
_, err = ec2Client.DeleteVpc(&ec2.DeleteVpcInput{
VpcId: vpcID,
})
if err != nil {
return err
// Retry up to 10 times.
retryCount := 0
for ; retryCount < 10; retryCount++ {
_, err = ec2Client.DeleteVpc(&ec2.DeleteVpcInput{
VpcId: vpcID,
})
if err == nil {
break
}
fmt.Printf("VPC: Destroy error... [id=%s,err=%q,retry=%d]\n", *vpcID, err, retryCount)
time.Sleep(1 * time.Second)
}
if retryCount == 10 {
return errors.New("reached max retry count deleting VPC")
}

// Now that the destroy request went through we still need to wait for
// the deletion to complete.
if err := destroyBackoff(ctx, "VPC", *vpcID, func() error {
currVPCs, err := ec2Client.DescribeVpcsWithContext(ctx, &ec2.DescribeVpcsInput{
VpcIds: []*string{vpcID},
Expand Down

0 comments on commit 1283392

Please sign in to comment.