ArgoCD and Linkerd in conflict? Unexpected restarts

[gtoledot]

Hi everyone! I'm facing an interesting challenge in my GitOps setup with ArgoCD and Linkerd, and I'd like to hear your opinions and experiences.

Context:

I have two independent Kubernetes clusters: one for pre-production and one for production.
I use ArgoCD to synchronize my deployments from a Git repository with a single master branch, where each environment has its own folder.
Linkerd is injecting a proxy into my Pods to provide service mesh functionalities.
My CI/CD flow involves Jenkins building Docker images and updating the deployment manifests in the Git repository.
ArgoCD monitors the repository and synchronizes changes to the corresponding clusters.
Problem:

Even though changes are made only in the pre-production folder, ArgoCD sometimes triggers synchronizations and Pod restarts in production, even if there are no direct changes to the production manifests.
I have observed that there is a rollme annotation in the deployment manifests, whose value changes or is different between what I have versioned and what is deployed. ArgoCD seems to be detecting these changes in the annotation as a reason to synchronize, even if there are no other relevant changes.
Question:

Is the best solution to disable ArgoCD's autosync option to avoid these unexpected restarts in production?
Are there other solutions or configurations that allow me to keep ArgoCD's automatic synchronization but prevent changes in pre-production from affecting production due to Linkerd's proxy injection?

argocd version: v2.9.3

[ChristianCiach]

I have observed that there is a rollme annotation in the deployment manifests, whose value changes or is different between what I have versioned and what is deployed.

This seems to be your issue. This would "just" work if these annotations were only in the cluster (and not in git), because ArgoCD only cares about the yaml attributes that are present in git (unless self-heal is enabled).

Since your "rollme" annotation differs between your versioned and live state, ArgoCD triggers a sync.

You can probably solve your issue by using ignoreDifferences: https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/

For example, to globally (for all Applications) ignore the annotation for all Deployments, you may put this into your argocd-cm ConfigMap:

data:
  resource.customizations.ignoreDifferences.apps_Deployment: |
    jsonPointers: 
      - /spec/template/metadata/annotations/rollme

You may or may not want to combine this with the RespectIgnoreDifferences sync option: https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#respect-ignore-difference-configs

(The rollme annotation is described by the official Helm documentation. There it is used to force a restart of the Deployment on every update. This doesn't really work well with GitOps. If you don't use RespectIgnoreDifferences, your Deployments will be restarted on every sync, because ArgoCD would still update the rollme annotation. This may or may not be what you want. But configuring ignoreDifferences as shown above should be enough to avoid the unwanted auto syncs).

[gtoledot]

Thank you very much for your valuable help and the proposed solutions! I really appreciate the time you've taken.

The information provided has been fundamental in better understanding the root cause of the unexpected restarts in production. The information about the rollme annotation has given me a much deeper understanding of the problem.

The solution you propose, using ignoreDifferences and RespectIgnoreDifferences, seems to be the right approach to solve my problem without having to disable ArgoCD's automatic synchronization, which is crucial for maintaining my GitOps workflow.

Thank you again for your invaluable contribution!