ArgoCD Application Controller causes Sync Issues when run with more than 1 replica

[bruce-y]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When running the ArgoCD application controller we observe that Sync jobs occasionally have trouble hitting completion. The UI seems to display the Sync jobs in progress perpetually. However, when we try to terminate such a job, it returns that there are no sync operations in progress. Additionally, the logs from the application controller seem to imply that the sync operation was already completed successfully.

To Reproduce

Scale up the Application controller (we used 3 replicas) and attempt a sync job. We've noticed that this occurs more frequently on ArgoCD Applications with many resources.

Expected behavior

The Sync job progress to completion appropriately, even with multiple Application Controller Replicas

Version

argocd: v1.8.1+c2547dc
  BuildDate: 2020-12-10T03:00:31Z
  GitCommit: c2547dca95437fdbb4d1e984b0592e6b9110d37f
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: darwin/amd64

Logs

argocd-application-controller-0 argocd-application-controller time="2020-12-14T22:14:40Z" level=info msg="updated 'test-app' operation (phase: Succeeded)"

...


argocd-application-controller-0 argocd-application-controller time="2020-12-14T22:14:41Z" level=info msg="Update successful" application=test-app

We found the above log 30 seconds into the sync but we observed the UI reporting the application sync still in progress for minutes afterwards.

[jannfis]

Hi @bruce-au, have you actually scaled up according to https://argoproj.github.io/argo-cd/operator-manual/high_availability/#argocd-application-controller (i.e. setting the ARGOCD_CONTROLLER_REPLICAS to an appropriate value as well)?