Performance of ApplicationSet Controller

[hcelaloner]

Summary

We are heavily using ApplicationSet in our company. At the time of writing, we have 1296 ApplicationSet, 5155 Application created in the ArgoCD cluster. Most of the time, making a change on ApplicationSet and generation of a new application takes approximately 30 minutes. Because of that making a new deployment via ApplicationSet initially takes 30 minutes. Can we analyze and improve this one (btw, glad to work on the subject if you can guide us)?

Proposal

If the controller is managing too many ApplicationSet then can we implement a sharding option similar to the ArgoCD application controller?

We saw that the maximum number of concurrent Reconciles which can be run defaults to 1 in the ApplicationSet controller. Can we speed up processing ApplicationSets by increasing this number? Would it cause any concurrency-related issues for ApplicationSet? If not, would it be possible to make this option configurable? Somehow similar to "--operation-processors" or "--status-processors" options in ArgoCD application controller.

[iam-veeramalla]

Hi @jgwest, I thought your experience with applicationsets might help here :). Are you aware of any workaround or future solution(something in roadmap) that might help @hcelaloner ?

[hcelaloner]

Hi, any updates on this?

[hcelaloner]

In https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/, for argocd-application-controller it is stated that

If the controller is managing too many clusters and uses too much memory then you can shard clusters across multiple controller replicas. To enable sharding increase the number of replicas in argocd-application-controller StatefulSet and repeat number of replicas in ARGOCD_CONTROLLER_REPLICAS environment variable. The strategic merge patch below demonstrates changes required to configure two controller replicas.

so argocd-application-controller uses a sharding technique to support managing many clusters and many applications.

If a single applicationset controller is not able to process ~1.8k applicationsets, maybe we can implement a similar sharding approach. In #9568, we tried to bring the same approach to applicationset controller. It randomly shards applicationsets to each controller instance based on the hash of their ids in k8s.

[hcelaloner]

Also, https://github.com/argoproj/argo-cd/releases/tag/v2.4.0, mentions OpenTelemetry Tracing Integration. Are there any plans for other components like applicationset controller. In our company, we are also trying to use OpenTelemetry in our applications, it may provide richer telemetry data to investigate performance bottlenecks in cases like this. If there are such plans, we can discuss and I would like to give it try for applicationset controller (contribute as much as I can)

[crenshaw-dev]

@hcelaloner looks like opentelemetry was not added for the appset controller, but I'd welcome a PR to add it and would be happy to review!