You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you delete an application whose cluster no longer exists, the application delete will fail (not great). But, worse, after 3-4 minutes, Argo CD enters an infinite deletion reconciliation loop, flooding the k8s server with requests (see logs below).
There are two problems here:
I don't think we should prevent Applications from being deleted when the targeted cluster doesn't exist anymore. The cluster being deleted seems like a pretty clear signal of user intent.
We probably shouldn't be entering an infinite deletion reconciliation loop 😄 .
This was originally reproduced with the ApplicationSet controller (and is causing issues with cluster generator), but here are a quick set of steps to reproduce that don't require the controller.
Reproduced with Argo CD master branch (as of this writing), but was also reproducible with v1.8.x.
Steps to reproduce:
Create a cluster named 'cluster1' (for example)
Create an application against that cluster
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata:
name: cluster1-guestbooknamespace: argocdspec:
destination:
name: cluster1 # must match cluster name above, of coursenamespace: guestbookproject: defaultsource:
path: guestbookrepoURL: https://github.com/argoproj/argocd-example-apps.gittargetRevision: HEAD
(you don't need to sync it)
Ensure that the application appears within Argo CD.
Begin tailing the logs of the application controller, eg kubectl logs -f pod/argocd-application-controller-0 -n argocd
Delete that cluster (either delete the cluster within the web UI, or delete the cluster secret)
Delete the application within the web UI or CLI (foreground deletion, though background has the same behaviour)
Within the application controller logs, you will see:
time="2021-03-19T11:49:22Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:49:22Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server=
Which isn't great (IMHO we should allow deletion if the cluster doesn't exist), but keep going...
IMPORTANT: wait ~3-4 minutes, and watch the application controller logs.
After about 3-4 minutes, you will see Argo CD get stuck in what appears to be a deletion reconciliation loop. It gets into a state where it keeps trying to process that Application over and over, and since the cluster doesn't exist, it will never succeed.
IMHO it seems like the correct behaviour is for Argo CD to allow invalid Applications (Applications that point to clusters w/o a corresponding secret) to be deleted. The deletion finalizer and/or reconciler shouldn't prevent this operation.
Raw logs:
# Initial deletion error
time="2021-03-19T11:49:22Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:49:22Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
time="2021-03-19T11:49:22Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:49:22Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
# A minute later, still OK
time="2021-03-19T11:50:25Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:50:25Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
# We enter the infinite loop
time="2021-03-19T11:53:25Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Updated sync status: OutOfSync -> Unknown" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=ResourceUpdated type=Normal
time="2021-03-19T11:53:25Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
time="2021-03-19T11:53:25Z" level=info msg="Updated health status: Missing -> Unknown" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=ResourceUpdated type=Normal
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=69
time="2021-03-19T11:53:25Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=21
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=19
time="2021-03-19T11:53:25Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=19
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=25
time="2021-03-19T11:53:25Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=24
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=16
time="2021-03-19T11:53:25Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=21
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Deleting resources" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Update successful" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Reconciliation completed" application=cluster1-guestbook dest-name=cluster6 dest-namespace=guestbook dest-server= fields.level=2 time_ms=20
time="2021-03-19T11:53:25Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-03-19 11:47:54 +0000 UTC, expiry: 3m0s), level (2)" application=cluster1-guestbook
time="2021-03-19T11:53:25Z" level=info msg="Unable to delete application resources: unable to find destination server: there are no clusters with this name: cluster6" application=cluster1-guestbook dest-namespace=guestbook dest-server= reason=StatusRefreshed type=Warning
(... this keeps going ...)
The text was updated successfully, but these errors were encountered:
jgwest
changed the title
Unable to delete Application if its cluster is deleted, Argo CD enters infinite app deletion reconciliation loop
Unable to delete an Application if its target cluster is deleted, Argo CD enters infinite app deletion reconciliation loop
Mar 19, 2021
If you delete an application whose cluster no longer exists, the application delete will fail (not great). But, worse, after 3-4 minutes, Argo CD enters an infinite deletion reconciliation loop, flooding the k8s server with requests (see logs below).
There are two problems here:
This was originally reproduced with the ApplicationSet controller (and is causing issues with cluster generator), but here are a quick set of steps to reproduce that don't require the controller.
Reproduced with Argo CD master branch (as of this writing), but was also reproducible with v1.8.x.
Steps to reproduce:
(you don't need to sync it)
Ensure that the application appears within Argo CD.
Begin tailing the logs of the application controller, eg
kubectl logs -f pod/argocd-application-controller-0 -n argocd
Delete that cluster (either delete the cluster within the web UI, or delete the cluster secret)
Delete the application within the web UI or CLI (foreground deletion, though background has the same behaviour)
Within the application controller logs, you will see:
Which isn't great (IMHO we should allow deletion if the cluster doesn't exist), but keep going...
After about 3-4 minutes, you will see Argo CD get stuck in what appears to be a deletion reconciliation loop. It gets into a state where it keeps trying to process that Application over and over, and since the cluster doesn't exist, it will never succeed.
IMHO it seems like the correct behaviour is for Argo CD to allow invalid Applications (Applications that point to clusters w/o a corresponding secret) to be deleted. The deletion finalizer and/or reconciler shouldn't prevent this operation.
Raw logs:
The text was updated successfully, but these errors were encountered: