ApplicationSet Controller Scaling?

[dee-kryvenko]

I tried to search https://github.com/search?q=repo%3Aargoproj%2Fargo-cd+applicationset+scaling&type=issues and I am aware of https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/, but I cannot find anything regarding ApplicationSet Controller scaling. I have what I consider a fairly small setup, I've seen much bigger around here, I have about ~20 k8s clusters and only about ~250 ApplicationSets with a total of about ~5k applications. Most of them are span up by ApplicationSets although not all.

It scales pretty well for the most part except for the ApplicationSet controller, which looks like I can't scale horizontally at the moment? It takes it about 5-10 minutes to do a full cycle and reconcile changes to ApplicationSets onto the Applications it manages.

Is there anything I am missing?

[rumstead]

The ApplicationSet controller doesn't do any sharding so running multiple controllers won't make the reconciliation any faster.

What generators/applicationsets do you see the controller taking most of its time? Have you tried increasing the number of workers?

You can set the value applicationsetcontroller.concurrent.reconciliations.max: "25" in argocd-cmd-params-cm configmap.

EDIT: PR to add an example to the docs - #19166

[dee-kryvenko]

Thanks @rumstead, I was not aware this option existed. That PR should help a lot with discoverability, but to be honest I think the best place to mention it in the https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/. Although this is not HA topic but at least I did refer to this doc as a general scalability guide. I am trying 50 now and I will let you know if I notice any difference in a week or so.

[rumstead]

Agreed, there isn't anything for the applicationset currently in the HA docs so I shied away.

[voron]

I'm watching the similar performance - 1k apps in 50 seconds. reconciliations.max doesn't affect performance for me, as there are 0 effective reconciliations in my case. I've tried to spread 1k apps into multiple appsets (from 1 to 16), but it changes nothing, as the source repo is the same and all 16 appsets refresh from a webhook at the same time.

I've noticed rest_client_requests_total metric is maxed out on 40.0 for appset controller.
One of possible reproduce cases is to use requeueAfterSeconds: 1 in appset spec with git generator, multiple appsets with 100+ apps each. I see that controller_runtime_active_workers stays equal to the number of appsets refreshing, it confirms reconciliations.max is active.
Moreover, when multiple appsets are refreshing at the same time, it seems to share some limit during a refresh

• single appset - 20 apps per second
• 2 appsets - 10 apps per second per each appset, effective speed is the same

I suppose a single app requires 2 k8s api requests to refresh from appset controller side

Is there some limit rate from k8s controller-runtime ? I don't see any similar limits from application controller ( and we can scale application controller horizontally to push a performance even further).
Here is a chart for rest_client_requests_total in my tests with 2 appsets and requeueAfterSeconds: 1

[voron]

It may be fixed by #18623

[voron]

Yes, two env vars and the patch from #18623 fix related appset performance issues in our case.

global:
  env:
    - name: ARGOCD_K8S_CLIENT_QPS
      value: "200"
    - name: ARGOCD_K8S_CLIENT_BURST
      value: "400"

[amorozkin]

In my case with monоrepo approach I have to explicitly list paths for git file generators in ApplicationSet spec pointing to limited set of CLUSTERS/NAMESPACES depending on ApplicationSet like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
  generators:
  - git:
      files:
      - path: CLUSTER_NAME_01/NAMESPACE_NAME_01/*/argocd-app.yaml
      - path: CLUSTER_NAME_01/NAMESPACE_NAME_02/*/argocd-app.yaml
      - path: CLUSTER_NAME_01/NAMESPACE_NAME_03/*/argocd-app.yaml
      - path: CLUSTER_NAME_04/NAMESPACE_NAME_01/*/argocd-app.yaml
      - path: CLUSTER_NAME_11/NAMESPACE_NAME_01/*/argocd-app.yaml
      - path: CLUSTER_NAME_11/NAMESPACE_NAME_02/*/argocd-app.yaml
      - path: CLUSTER_NAME_23/NAMESPACE_NAME_01/*/argocd-app.yaml
....
      - path: CLUSTER_NAME_XX/NAMESPACE_NAME_XX/*/argocd-app.yaml
      repoURL: https://some-git-url-here/some-argocdrepo.git
      revision: HEAD
  template:
....

Using exclude with wildcard */*/*/argocd-app.yaml is an option - but it has other side-effects.

With 90 appsets and 1600 paths in total (about 17 paths per one appset) it takes up to 15-20 minutes to catch changes in repo and create new application for example.

Applicationset controller doing git ls .. for each path one by one:

level=info msg=Trace args="[git ls-files --full-name -z -- CLUSTER_NAME_XX/NAMESPACE_NAME_XX/*/argocd-app.yaml]" dir=/tmp/https__... operation_name="exec git" time_ms=3.101526

Will applicationsetcontroller.concurrent.reconciliations.max help in this case?
If NOT - what another optiоns could be?

Now using version 2.6 - considering switching to 2.13 ..

[rumstead]

2.6 doesn't have the levers to control applicationsetcontroller.concurrent.reconciliations.max it also doesn't use the repo server for git caching. You should upgrade to the latest version and see how it improves the existing configuration's performance.