-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Very slow performance of newest argocd versions - plugin + monorepo #17775
Comments
I don't think we have enough information to really brainstorm solutions yet. The spiked repo-server CPU usage is a good hint. Do you know if any Argo CD component is hitting its CPU limits at any time, i.e. being throttled? |
I appreciate I owe you a proper analysis. My apologies - I still haven't had a chance to set up prometheus etc. to scrape the metrics / traces / set up pprofiler etc. To just answer you previous question - there i no throttling being observed, the nodes are fairly big with plenty of RAM and CPU headroom (under normal circumstances). In the meantime, I wanted to share the following, in case useful and perhaps symptomatic of other issues listed above. When navigating to Application -> Details -> Params tab (which, in theory, should only show two variables as inputs), The following three log lines can be observed (note the 35+ seconds duration before the kill): repo-server (container: repo-server)
repo-server (container: tanka)
repo-server (container: tanka)
In case useful:
All applications are generated from For the record - running It feels suspicious that: (a) sending across 27MB would take over 30 seconds and time out; (b) it is necessary to resolve the manifests to just show the Params (often used e.g. to quickly check the (Our main problem is the overall sync-up of 30-50 Applications on deployment, which can take 30min+ just to update image tags, but... perhaps the above issue, which is also quite troublesome, is related). |
Regarding sparse/shallow checkout, let's consolidate that part of the conversation here: #11198 |
I think performance will be improved by #18053 (to be released in v2.13), where we can skip MatchRepo if we set pluginName explicitly. |
Checklist:
argocd version
.Describe the bug
Unfortunately, upgrading to the newest versions of ArgoCD (with the cmp migrated to a sidecar) resulted in an order of magnitude degradation of refresh/sync/deployment speeds.
Setup
Monorepo with c. 50 Applications, all defined under a single sub-dir, written in jsonnet + tanka. An Application in our realm is a microservice (deployment + a few helper manifests), nothing extraordinary. We run repo-server with
--plugin-tar-exclude
set to.git/*
.Bug / Observations
app set
(to specifyimage_tag
, a plugin variable) and thenapp sync
for all Applications. It takes c. 3-5 minutes per a single Application. In contrast, running a diff and applying manually (tk apply . --ext-str=image_tag=MY_TAG
) takes a fraction of this time (c. 5 seconds per Application)Parameters
page on a single Application Details view (UI). This is quite interesting, as showing theDiff
tab is (usually) almost instantaneous.Normal operation:
During a manually-triggered update (
app set
) + sync:Potential Solutions
We see some hope (and look forward to) to the following potential solutions (naturally, it is difficult to gauge a priori to what extent any of them would resolve the issues observed):
--plugin-tar-include
(i.e. include-only manifests dir) - probably of limited benefit, since we are already passing--plugin-tar-exclude
, which excludes the bulk of the repo.To Reproduce
Store manifests of 50-100 Applications in a mono-repo, together with other code (Go, TS, etc.). Use tanka to apply them, configured via an argocd sidecar plugin (example plugin configuration here).
Expected behavior
A simple update of minimal changes should be relatively fast (taking a bit longer than a manual application, but not 10-50x longer).
Version
argocd-server: v2.10.6+d504d2b
The text was updated successfully, but these errors were encountered: