Sharding application-controller by Application or a Project

[msobkowiak-olx]

Hey there !

Wanted to ask on possibilities to distribute load across multiple application-controllers, our setups usually consist of one ArgoCD instance per cluster, thus is not very beneficial for us to use default sharding option described here:
https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/

I would like to ask if there is any plans to actually change this to shard per application name (using static or consistent (with crash support) hashing, just like on memcached clients or in loadbalancers), which should have the following benefits:

• best possible granularity of the load (with monorepos still being sort of an issue however it is still an issue even now)
• best resource ultilisation (achieving best possible "even" distribution, with exception for large monorepos)
• best possible scaling scenarios (supporting ArgoCD with one or multiple clusters)

Similar can be applied to a project. The hashing can simply take project or app name, calculate the hash and send it to proper slot for processing in the deployed statefulset. This should be possible because of the constraints ArgoCD puts on the Applications and Projects (unique naming, guaranteeing that each entity has unique key for hashing function, the distribution happens onto application-controller pods).

Please note I am aware of the queue tuning on operation and status processors, however we would like to avoid creating too large pods not fitting our workload nodes in the clusters hosting ArgoCD.

Let me know if it makes sense.

[awx-fuyuanchu]

Yeah. It sounds like a more general pain point for most users. One cluster could spawn hundreds of Applications and sharding the Applications in one cluster can help a lot for performance, efficiency, and scalability.

[saprette]

Hello, that is something that we would benefit from.

[andrii-korotkov-verkada]

I suggest to open an enhancement proposal for this. I don't know if this is easily solvable, since resources can move between applications and applications can move between projects.

[andrii-korotkov-verkada]

I think the problem is two things can maintain a different view of the same object in Kubernetes, so there might be conflicting changes.

[msobkowiak-olx]

Isn't this an issue even now on single shard ? I mean you can technically deploy two different applications with different name and exactly the same manifests (we used to hit collisions sometimes during migrations from helm).

[andrii-korotkov-verkada]

There's a cluster lock acquired by a number of places.

[msobkowiak-olx]

Thanks, I mean it would be crucial to understand what exactly this lock synchronizes in this case. Implementing a distributed lock would not be that easy (technically the locking for cross-shard implementation could be externalised to redis or so - that is actually not that hard but prone to process crashes without expiration implementation).

[andrii-korotkov-verkada]

I can bring this up during the contributors meeting.

[andrii-korotkov-verkada]

See #6125 for a similar thing.