Multiple ArgoCD deployments attempt to reconcile the same ApplicationSet despite ARGOCD_APPLICATIONSET_CONTROLLER_NAMESPACES being set to mutually exclusive namespaces

[eadred]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I have two ArgoCD deployments in the same cluster configured as cluster-wide instances. To ensure that they manage mutually exclusive sets of Applications/ApplicationSets I have set the ARGOCD_APPLICATIONSET_CONTROLLER_NAMESPACES and ARGOCD_APPLICATION_NAMESPACES environment variables in the relevant controllers (as described here and here), such that each instance should monitor apps/appsets in mutually exclusive sets of namespaces.

I am deploying an ApplicationSet into a namespace monitored by the first ArgoCD instance whose generated Applications use an AppProject only known to that instance.

What I am finding is that the ApplicationSet's status flip-flops between being Healthy and being Degraded, showing the error message: "error getting project dh-applications-dev: AppProject.argoproj.io "dh-applications-dev" not found". I can see the same error message in the container logs of the appset controller of the instance that shouldn't be managing this appset

To Reproduce

• Deploy a pair of ArgoCD instances in separate namespaces.
• For the first instance set applicationsetcontroller.namespaces and application.namespaces (in the Helm values under configs.params) both to *-dev,dev-argocd-seed-manifests.
• For the second instance set those settings to *-prd,prd-argocd-seed-manifests,*-ppr,ppr-argocd-seed-manifests.
• Define an AppProject in the first instance's namespace called dh-applications-dev.
• Define an ApplicationSet which uses the dh-applications-dev project and deploy it in a namespace which should be covered by the first ArgoCD instance, eg someteam-applications-dev.
• Check the status of the ApplicationSet periodically and note that it flips between the Healthy and Degraded states.

Expected behavior

The ApplicationSet in a namespace should be ignored by the appset controller which is not monitoring that namespace.

Version

v2.12.6+4dab5bd

My guess as to what is wrong

I notice that the ignoreNotAllowedNamespaces predicate (here) which is used by the controller only applies the namespace filter to create events.

Could it be that when the first ArgoCD instance, after reacting to the create event, updates the status thus triggering an update event, which now both instances will react to? The second instance would then update the status to indicate the error state (because it doesn't know about the AppProject). That triggers a subsequent update event, the first instance reacts to this and puts the status back to healthy, and so on.

Should the ignoreNotAllowedNamespaces predicate not apply the filter to all types of event, eg using the NewPredicateFuncs method?

[eadred]

Actually a way to reproduce another unexpected effect which would have the same underlying cause is this:

• Deploy a cluster-wide ArgoCD, but with ARGOCD_APPLICATIONSET_CONTROLLER_NAMESPACES set to a limited number of namespaces.
• Deploy an ApplicationSet into a namespace not included in ARGOCD_APPLICATIONSET_CONTROLLER_NAMESPACES.
• kubectl get the ApplicationSet - there will be no status property, as expected because ArgoCD should not be managing this resource.
• Update the ApplicationSet in some way (eg add a label to it) and redeploy it.
• Now when you run kubectl get you will see a status property filled in because ArgoCD has (incorrectly) responded to the change and attempted to reconcile the system state with the ApplicationSet. If the ApplicationSet is correctly configured ArgoCD may even have deployed Application resources into the un-monitored namespace (which it may or may not also react to depending on how you have also set application.namespaces).

[andrii-korotkov-verkada]

Are you using separate tracking mechanism for each ArgoCD?

[eadred]

If you mean by setting the installation id then yes. However, I think that's a solution to a different problem, namely identically named Applications getting confused about which resources they respectively own because the tracking labels/annotations on those resources are insufficiently unambiguous.