Argo CD loses track of managed live resource when switching tracking method

[jannfis]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When switching tracking method from label to annotation (or vice versa) in a running Argo CD system, Argo CD will lose track of managed live resources, resulting in a situation where Argo CD will not pick up changes to live resources and also fails to prune them.

To Reproduce

• Create the guestbook Application:

  $ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-namespace guestbook --dest-server https://kubernetes.default.svc --sync-option CreateNamespace=true
  application 'guestbook' created

• Sync application

  $ argocd app sync guestbook
  TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS    HEALTH        HOOK  MESSAGE
  2022-06-24T10:23:26+00:00            Service   guestbook          guestbook-ui  OutOfSync  Missing              
  2022-06-24T10:23:26+00:00   apps  Deployment   guestbook          guestbook-ui  OutOfSync  Missing              
  2022-06-24T10:23:29+00:00          Namespace                         guestbook   Running   Synced              namespace/guestbook created
  2022-06-24T10:23:29+00:00          Namespace                         guestbook  Succeeded   Synced              namespace/guestbook created
  2022-06-24T10:23:29+00:00            Service   guestbook          guestbook-ui  OutOfSync  Missing              service/guestbook-ui created
  2022-06-24T10:23:29+00:00   apps  Deployment   guestbook          guestbook-ui  OutOfSync  Missing              deployment.apps/guestbook-ui created
  
  Name:               guestbook
  Project:            default
  Server:             https://kubernetes.default.svc
  Namespace:          guestbook
  URL:                http://127.0.0.1:8080/applications/guestbook
  Repo:               https://github.com/argoproj/argocd-example-apps.git
  Target:             
  Path:               guestbook
  SyncWindow:         Sync Allowed
  Sync Policy:        <none>
  Sync Status:        Synced to  (53e28ff)
  Health Status:      Progressing
  
  Operation:          Sync
  Sync Revision:      53e28ff20cc530b9ada2173fbbd64d48338583ba
  Phase:              Succeeded
  Start:              2022-06-24 10:23:26 +0000 UTC
  Finished:           2022-06-24 10:23:29 +0000 UTC
  Duration:           3s
  Message:            successfully synced (all tasks run)
  
  GROUP  KIND        NAMESPACE  NAME          STATUS     HEALTH       HOOK  MESSAGE
         Namespace              guestbook     Succeeded  Synced             namespace/guestbook created
         Service     guestbook  guestbook-ui  Synced     Healthy            service/guestbook-ui created
  apps   Deployment  guestbook  guestbook-ui  Synced     Progressing        deployment.apps/guestbook-ui created
  

• Check that application is healthy and in sync

  $ argocd app list
  NAME       CLUSTER                         NAMESPACE  PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                 PATH       TARGET
  guestbook  https://kubernetes.default.svc  guestbook  default  Synced  Healthy  <none>      <none>      https://github.com/argoproj/argocd-example-apps.git  guestbook  

• Change a managed live resource in the cluster and check that Argo CD is aware of it

  $ kubectl patch -n guestbook deployments.apps guestbook-ui --patch='{"spec":{"replicas":2}}'
  deployment.apps/guestbook-ui patched
  
  $ argocd app list
  NAME       CLUSTER                         NAMESPACE  PROJECT  STATUS     HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                 PATH       TARGET
  guestbook  https://kubernetes.default.svc  guestbook  default  OutOfSync  Healthy  <none>      <none>      https://github.com/argoproj/argocd-example-apps.git  guestbook  
  

• Revert change by syncing the app

  $ argocd app sync
  TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS    HEALTH        HOOK  MESSAGE
  2022-06-24T10:36:12+00:00            Service   guestbook          guestbook-ui    Synced   Healthy              
  2022-06-24T10:36:12+00:00   apps  Deployment   guestbook          guestbook-ui  OutOfSync  Healthy              
  2022-06-24T10:36:12+00:00            Service   guestbook          guestbook-ui    Synced   Healthy              service/guestbook-ui unchanged
  2022-06-24T10:36:12+00:00   apps  Deployment   guestbook          guestbook-ui  OutOfSync  Healthy              deployment.apps/guestbook-ui configured
  
  Name:               guestbook
  Project:            default
  Server:             https://kubernetes.default.svc
  Namespace:          guestbook
  URL:                http://127.0.0.1:8080/applications/guestbook
  Repo:               https://github.com/argoproj/argocd-example-apps.git
  Target:             
  Path:               guestbook
  SyncWindow:         Sync Allowed
  Sync Policy:        <none>
  Sync Status:        Synced to  (53e28ff)
  Health Status:      Healthy
  
  Operation:          Sync
  Sync Revision:      53e28ff20cc530b9ada2173fbbd64d48338583ba
  Phase:              Succeeded
  Start:              2022-06-24 10:36:12 +0000 UTC
  Finished:           2022-06-24 10:36:12 +0000 UTC
  Duration:           0s
  Message:            successfully synced (all tasks run)
  
  GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
         Service     guestbook  guestbook-ui  Synced  Healthy        service/guestbook-ui unchanged
  apps   Deployment  guestbook  guestbook-ui  Synced  Healthy        deployment.apps/guestbook-ui configured

• Change tracking method to annotation

  $ kubectl patch configmaps argocd-cm --patch='{"data": {"application.resourceTrackingMethod": "annotation"}}'
  configmap/argocd-cm patched

• Sync application to apply annotation to live resources and remove previous label

  $ argocd app sync guestbook
  TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
  2022-06-24T10:39:29+00:00   apps  Deployment   guestbook          guestbook-ui    Synced  Healthy              
  2022-06-24T10:39:29+00:00            Service   guestbook          guestbook-ui    Synced  Healthy              
  
  Name:               guestbook
  Project:            default
  Server:             https://kubernetes.default.svc
  Namespace:          guestbook
  URL:                http://127.0.0.1:8080/applications/guestbook
  Repo:               https://github.com/argoproj/argocd-example-apps.git
  Target:             
  Path:               guestbook
  SyncWindow:         Sync Allowed
  Sync Policy:        <none>
  Sync Status:        Synced to  (53e28ff)
  Health Status:      Healthy
  
  Operation:          Sync
  Sync Revision:      53e28ff20cc530b9ada2173fbbd64d48338583ba
  Phase:              Succeeded
  Start:              2022-06-24 10:39:29 +0000 UTC
  Finished:           2022-06-24 10:39:30 +0000 UTC
  Duration:           1s
  Message:            successfully synced (all tasks run)
  
  GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
         Service     guestbook  guestbook-ui  Synced  Healthy        service/guestbook-ui configured
  apps   Deployment  guestbook  guestbook-ui  Synced  Healthy        deployment.apps/guestbook-ui configured

• Check annotation has been applied, and label has been removed

  $ kubectl get -n guestbook deployments.apps guestbook-ui -o jsonpath='{.metadata.annotations.argocd\.argoproj\.io\/tracking-id}'
  guestbook:apps/Deployment:guestbook/guestbook-ui
  $ kubectl get -n guestbook deployments.apps guestbook-ui -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/instance}'
  # <empty>

• Patch managed live resource again

  $ kubectl patch -n guestbook deployments.apps guestbook-ui --patch='{"spec":{"replicas":4}}'

• Observe that Argo CD has not picked up the change (i.e. Application is still in Synced state)

  $ argocd app list
  NAME       CLUSTER                         NAMESPACE  PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                 PATH       TARGET
  guestbook  https://kubernetes.default.svc  guestbook  default  Synced  Healthy  <none>      <none>      https://github.com/argoproj/argocd-example-apps.git  guestbook  

• Delete the app with cascade, and observe that resources will not get removed:

  $ argocd app delete guestbook --cascade
  Are you sure you want to delete 'guestbook' and all its resources? [y/n]
  y
  application 'guestbook' deleted
  $ kubectl get -n guestbook all
  NAME                               READY   STATUS    RESTARTS   AGE
  pod/guestbook-ui-6b689986f-dqgvm   1/1     Running   0          13m
  pod/guestbook-ui-6b689986f-hsdlt   1/1     Running   0          2m32s
  
  NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
  service/guestbook-ui   ClusterIP   10.43.104.8   <none>        80/TCP    13m
  
  NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/guestbook-ui   2/2     2            2           13m
  
  NAME                                     DESIRED   CURRENT   READY   AGE
  replicaset.apps/guestbook-ui-6b689986f   2         2         2       13m

Expected behavior

Argo CD being able to switch resource tracking methods while running.

Screenshots

Version

Latest code in master branch as of 2022-06-24 13:00:00 CEST (commit 9d4c940)

Logs

Paste any relevant application logs here.

[jannfis]

I have found the culprit, working on a fix.