Nuke CloudFormation stacks first

[leozhad]

I would like to suggest that aws-nuke should nuke any existing CloudFormation stacks before anything else, when running on an account. This way, they will terminate any resource launched by those stacks, and cut down on wait times that would occur on if you terminate other resources first.

[svenwltr]

I am not sure how this actually would improve the wait times. At least not in a way that would justify the increasing code complexity.

Currently aws-nuke does an initial scan. While doing this it does not remove anything, so we would not speed up anything there. After this is sends a delete request to each found resource, but does not actually wait for the resource to be gone. Actually, this depends on the resource and if the remove request is sync or async, but AFAIK the most ones are async. Therefore the maximum amount of time we could save, is the time of the first run.

Are there actually so many resources in your scenario, that the first run takes so long? Am I missing something else?

Any other opinions @rebuy-de/prp-aws-nuke?

[leozhad]

I am not sure it would make a drastic difference -- but it seems logical to terminate CloudFormation stacks first, since this would in turn terminate any resources they launched cleanly.

[srij91]

Only issue I came across is, if awsnuke deletes the resources first even before the CFN stack is deleted, the CFN stack deletion fails saying it is unable to find the resource to delete and gets stuck in Deletion failed status. This is keeping awsnuke to go in loop and retry deleting CFN stack over and over. The moment I manually deleted CFN stack nuke job completed.

[svenwltr]

but it seems logical to terminate CloudFormation stacks first

Logical yes, but I am concerned about the increased complexity. Also the code base is currently very messy and adding more stuff would make it worse.

Only issue I came across is, if awsnuke deletes the resources first even before the CFN stack is deleted, the CFN stack deletion fails saying it is unable to find the resource to delete and gets stuck in Deletion failed status. This is keeping awsnuke to go in loop and retry deleting CFN stack over and over. The moment I manually deleted CFN stack nuke job completed.

That sounds like a bug to me. If you are able to delete it manually, aws-nuke should also be able to delete it. We should take a look into this.

[leflamm]

I usually delete CFN stacks first, too.

Reason being: I already ran into the situation that the role that was used to create the CFN stack was nuked before the stack. IIUC the role used to create a stack is used to destroy it as well.

If you have CodePipelines that create/update CFN stacks they will have a certain role for doing that. If that role gets nuked before the stacks get nuked - those stacks are not deleteable by aws-nuke anymore.

Cheers, Christian.

(At least this has been the case a couple of weeks ago)

[svenwltr]

After some internal discussion, there is the proposal to delete CF stacks first in a separate run (ie before actually scanning all other resources). What are your thoughts on this?

[leozhad]

I like this idea

…

Sent from my iPhone

On Aug 10, 2018, at 5:25 AM, Sven Walter ***@***.***> wrote: After some internal discussion, there is the proposal to delete CF stacks first in a separate run (ie before actually scanning all other resources). What are your thoughts on this? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[stephanlindauer]

PTAL: #242

[root360-AndreasUlm]

Hi,

when deleting CFN stacks first you have to consider that an CFN stack might not be deleted successfully when a controlled resource depends on something that was changed manually.

An example for this is:

1. Create a VPC with routing tables using CFN stack.
2. Create a VPC Peering and add a route to the routing table.
   With this setup deletion of the CFN stack fails until the peering and route are deleted manually.

The implementation should be fine when it deletes the CFN stacks with ignoring depending resources that cannot be deleted and after that deletes those resources using the concerning routine.

Does you implementation already consider this?

[stephanlindauer]

The solution proposed in #242 is that just the CFS is deleted and the resources that were created by it will be left intact by the CFS resource implementation.

CFN stack might not be deleted successfully

Is something that would not happen this way.

The resources that are retained from the CFS should be deleted by their own resource implementation (like any other resource in aws-nuke).

[stephanlindauer]

@leozhad please close the issue if #242 fixes your concerns

[leozhad]

Works for me!

[adamcousins]

hi @leozhad @svenwltr
I'm interested in a similar thing where Cloudformation stacks must be deleted first before attempting to delete resources individually.
Looks like you have found a way to achieve this (trying to follow the conversation above)

Does aws-nuke behave like this by default, or do i need to add certain configuration to achieve this?

[svenwltr]

Hello @adamcousins,

aws-nuke does not do this, but it should not matter.

Why do you want to delete the CF Stacks before anything other?

[adamcousins]

@svenwltr
similar reasons to the above.

aws-nuke deleted a resource before CFN had the chance to (Elastic IP in my case). The CFN stack now will not delete as it cant find the EIP resource its trying to delete. Cloudformation delete workload then expect the user to 'ignore' any resource errors to continue the delete action.

Is it possible using filters to instruct aws-nuke to 'target' these types of resources first CFN stacks, then anything else?

[svenwltr]

The deletion of the CloudFormation stack should not fail in this case.

The issues should be solved with #424.