Uninstalling ArgoCD tears down all "managed" namespaces?

[rsteppac]

Hello all,

I accidentally uninstalled our HA ArgoCD installation via our internal kustomize build pointing to manifests/ha/cluster-install?ref=v1.7.9 (kustomize build ... | kubectl delete -f -). What I thought would be a 10 minute mishap (--> argocd-util restore) turned into a significant incident as ArgoCD apparently turned around and deleted all namespaces across all clusters that were part of the app-of-app setup on that ArgoCD instance.

As this is hardly a desirable default behavior I am pretty sure we got up our setup wrong to cause this to happen. But I was not able to find documentation about uninstalling ArgoCD. And none of the tickets (I found) relating to uninstall report a similar behavior. If anyone could point me in the right direction where to look to help us understand the cascading deletion behavior, that would be greatly appreciated.

ArgoCD 1.7.9
K8S 1.17.5

Thanks in advance!
Ralf

[jannfis]

Hi @rsteppac, unless knowing details of your Kustomize build, I can just guess what was happening:

• Your Kustomize build did not only contain the Argo CD manifests, but additionally either a) also a Namespace resource for the argocd namespace or your application namespaces or b) the Application resources for your apps
• In case of a), when you were running kubectl delete the argocd namespace got deleted, tearing down all resources including your Application resources within that namespace, or in the worst case, all application namespaces as well
• In case of b), the Application resources got deleted too

My next guess would be that your Application resources have the deletion finalizer set, as described in https://argoproj.github.io/argo-cd/user-guide/app_deletion/#about-the-deletion-finalizer. This will trigger a cascading delete of the applications. Using the finalizer within the resources is a double-edged sword: It will keep your cluster tidy when using Argo CD in a mostly declarative way (app-of-apps), but also can lead to mayhem on mishaps such as yours.

[rsteppac]

@jannfis , thank you for your your help!

Our kustomize build does not create the namespaces or application resources. It deals with istio specifics, RBAC settings, LDAP/dex integration and such.

But you are right: We indeed set the finalizer on our application resources. I don't understand the mechanics, though, that lead to the finalizer to be acted on. As far as I can tell I did not delete the Application resources, nor their namespaces. Instead I deleted the service that would act upon the finalizer.
Do CRD instances (the apps in this case) get deleted if the CRD definition gets deleted?

Thanks again!
Ralf