fix(repo-server): excess git requests, short-circuit GenerateManifests ref only (Issue #14725) #1

nromriell · 2023-11-21T09:25:29Z

Second part of a split of the argoproj#16309 and follow-up to argoproj#16410. Continues the fixes required to remove the excess git requests when using multi-source and ref sources.

Issue:
When GenerateManifest is called with a ref only source manifest generation is still run, and since there are no manifests this is always considered a manifest cache miss causing an excess number of fetch requests to the git server

Changes:
Short-circuit calls to GenerateManifest when it is a ref only source, however as part of the short circuit still resolve the references if they are not already in cache, this can help prevent additional calls to ls-remote when GenerateManifest is called with a related manifest generating source

Impact:
This PR should greatly reduce the number of fetch calls made to git for multi-source applications with a ref only source. Some minor improvements to ls-remote should be observed as well since we skip unnecessary git util calls.

There is one additional item that will be included in a follow-up PR to finish up the reduction of ls-remote calls

Baseline v2.9.0 with 200 multi-source applications with a ref only source:

With the changes in the first PR under the same conditions:

With the changes from the first PR and this PR under the same conditions:

Checklist:

…rgoproj#16418) Bumps [github.com/go-jose/go-jose/v3](https://github.com/go-jose/go-jose) from 3.0.0 to 3.0.1. - [Release notes](https://github.com/go-jose/go-jose/releases) - [Changelog](https://github.com/go-jose/go-jose/blob/v3/CHANGELOG.md) - [Commits](go-jose/go-jose@v3.0.0...v3.0.1) --- updated-dependencies: - dependency-name: github.com/go-jose/go-jose/v3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

This actually wasn't that bad. It was just a few bumps in the right order. I ran tests and built everything and it appears to work. Signed-off-by: Dan Lorenc <dlorenc@chainguard.dev>

Add Salad Technologies Signed-off-by: Kyle Dodson <kyle@salad.com>

* fix: fixed cli admin dashboard cmd Signed-off-by: Soumya Ghosh Dastidar <gdsoumya@gmail.com> * feat: update docs Signed-off-by: Soumya Ghosh Dastidar <gdsoumya@gmail.com> --------- Signed-off-by: Soumya Ghosh Dastidar <gdsoumya@gmail.com>

…rgoproj#15573) * feat(opentelemetry): ✨ support for secured OTLP endpoint and headers Signed-off-by: Prashant Shahi <me@prashantshahi.dev> * docs(opentelemetry): 📝 include new otlp headers in docs Signed-off-by: Prashant Shahi <me@prashantshahi.dev> * docs(opentelemetry): 📝 update readme docs as per integration tests Signed-off-by: Prashant Shahi <me@prashantshahi.dev> * docs(opentelemetry): 📝 update readme docs as per integration tests Signed-off-by: Prashant Shahi <me@prashantshahi.dev> * chore: resolve indentation issues Signed-off-by: Prashant Shahi <me@prashantshahi.dev> * chore: fix indentation issues Signed-off-by: Prashant Shahi <me@prashantshahi.dev> * chore: include OTLP options in deployment manifests Signed-off-by: Prashant Shahi <me@prashantshahi.dev> * fix: update manifests to resolve failing CI Signed-off-by: Prashant Shahi <me@prashantshahi.dev> --------- Signed-off-by: Prashant Shahi <me@prashantshahi.dev>

Signed-off-by: Andrew Block <andy.block@gmail.com>

…ons/Settings/Documentation (argoproj#12016) Signed-off-by: Shi, Stone <jiadong.shi@fmr.com>

* adding in spec to logappevent params ARGO-1017: Signed-off-by: Nandini <nandinisingh759@gmail.com> * updating logappevent to include spec Signed-off-by: Nandini <nandinisingh759@gmail.com> * updated logevent to take marshal logfields into json Signed-off-by: Nandini <nandinisingh759@gmail.com> * removing spec from parameters just grabbing from app.spec Signed-off-by: Nandini <nandinisingh759@gmail.com> * removing app.spec from test Signed-off-by: Nandini <nandinisingh759@gmail.com> --------- Signed-off-by: Nandini <nandinisingh759@gmail.com> Co-authored-by: Nandini Singh <nandini_singh@intuit.com>

* fix: upgrade notifications-engine Signed-off-by: Gilad Salmon <gilad.salmon@gmail.com> * update notification-engine version Signed-off-by: pashakostohrys <pavel@codefresh.io> * go mod tidy Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * use correct go version in codeql Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * silly Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * make notifications-docs Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> --------- Signed-off-by: Gilad Salmon <gilad.salmon@gmail.com> Signed-off-by: pashakostohrys <pavel@codefresh.io> Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> Co-authored-by: pasha-codefresh <pavel@codefresh.io> Co-authored-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

…argoproj#16060) (argoproj#16421)

* fix: Address diff cache miss issues Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com> * validate mergo.Merge errors Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com> * Address review comments Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com> * Allow setting log level at the controller Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com> * remove unnecessary log setup Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com> --------- Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com>

Signed-off-by: Filip Rafaj <filip.rafaj@prusa3d.cz> Co-authored-by: Filip Rafaj <filip.rafaj@prusa3d.cz>

…es (argoproj#16128) * cluster.go, projectwindow.go Signed-off-by: Surajyadav <harrypotter1108@gmail.com> * updated-examples Signed-off-by: Surajyadav <harrypotter1108@gmail.com> * format-corrected Signed-off-by: Surajyadav <harrypotter1108@gmail.com> * spell-mistake Signed-off-by: Surajyadav <harrypotter1108@gmail.com> * format-correction Signed-off-by: Surajyadav <harrypotter1108@gmail.com> * retrigger Signed-off-by: Surajyadav <harrypotter1108@gmail.com> * retrigger-2 Signed-off-by: Surajyadav <harrypotter1108@gmail.com> * retirgger-3 Signed-off-by: Surajyadav <harrypotter1108@gmail.com> * update Signed-off-by: Surajyadav <harrypotter1108@gmail.com> --------- Signed-off-by: Surajyadav <harrypotter1108@gmail.com> Co-authored-by: Ishita Sequeira <46771830+ishitasequeira@users.noreply.github.com>

* bug: add parent ref node info on resource list Signed-off-by: ashutosh16 <11219262+ashutosh16@users.noreply.github.com> * bug: add parent ref node info on resource list Signed-off-by: ashutosh16 <11219262+ashutosh16@users.noreply.github.com> * bug: add parent ref node info on resource list Signed-off-by: ashutosh16 <11219262+ashutosh16@users.noreply.github.com> * bug: add parent ref node info on resource list Signed-off-by: ashutosh16 <11219262+ashutosh16@users.noreply.github.com> * bug: add parent ref node info on resource list Signed-off-by: ashutosh16 <11219262+ashutosh16@users.noreply.github.com> * bug: add parent ref node info on resource list Signed-off-by: ashutosh16 <11219262+ashutosh16@users.noreply.github.com> --------- Signed-off-by: ashutosh16 <11219262+ashutosh16@users.noreply.github.com>

Signed-off-by: Ashin Sabu <ashin.sabu@harness.io>

…unManifestGenAsync not using cache (Issue argoproj#14725) (argoproj#16410) * fix(repo-server): excess git requests part 1, resolveReferencedSources and runManifestGenAsync Signed-off-by: nromriell <nateromriell@gmail.com> * fix: remove unnecessary settings instantiation Signed-off-by: nromriell <nateromriell@gmail.com> --------- Signed-off-by: nromriell <nateromriell@gmail.com>

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* self service notification Signed-off-by: May Zhang <may_zhang@intuit.com> * revert back the changes for redis-ha Signed-off-by: May Zhang <may_zhang@intuit.com> * revert back the changes for redis-ha Signed-off-by: May Zhang <may_zhang@intuit.com> * update notification engine Signed-off-by: May Zhang <may_zhang@intuit.com> * re-trigger build Signed-off-by: May Zhang <may_zhang@intuit.com> * self service notification Signed-off-by: May Zhang <may_zhang@intuit.com> * revert back the changes for redis-ha Signed-off-by: May Zhang <may_zhang@intuit.com> * revert back the changes for redis-ha Signed-off-by: May Zhang <may_zhang@intuit.com> * update notification engine Signed-off-by: May Zhang <may_zhang@intuit.com> * re-trigger build Signed-off-by: May Zhang <may_zhang@intuit.com> * fix conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * fix conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * fix conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * fix conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * update notification enginer version Signed-off-by: May Zhang <may_zhang@intuit.com> * update notification enginer version Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * add back checkAppNotInAdditionalNamespaces Signed-off-by: May Zhang <may_zhang@intuit.com> * add cm and secret to clusterRole Signed-off-by: May Zhang <may_zhang@intuit.com> * if applicationNamespaces is not used, then use namespaced appClient Signed-off-by: May Zhang <may_zhang@intuit.com> * fix merge conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * fix doc and test based on review Signed-off-by: May Zhang <may_zhang@intuit.com> * self service notification Signed-off-by: May Zhang <may_zhang@intuit.com> * revert back the changes for redis-ha Signed-off-by: May Zhang <may_zhang@intuit.com> * revert back the changes for redis-ha Signed-off-by: May Zhang <may_zhang@intuit.com> * update notification engine Signed-off-by: May Zhang <may_zhang@intuit.com> * re-trigger build Signed-off-by: May Zhang <may_zhang@intuit.com> * fix conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * self service notification Signed-off-by: May Zhang <may_zhang@intuit.com> * revert back the changes for redis-ha Signed-off-by: May Zhang <may_zhang@intuit.com> * revert back the changes for redis-ha Signed-off-by: May Zhang <may_zhang@intuit.com> * update notification engine Signed-off-by: May Zhang <may_zhang@intuit.com> * re-trigger build Signed-off-by: May Zhang <may_zhang@intuit.com> * fix conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * fix conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * fix conflict Signed-off-by: May Zhang <may_zhang@intuit.com> * update notification enginer version Signed-off-by: May Zhang <may_zhang@intuit.com> * update notification enginer version Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * fixing go tidy Signed-off-by: May Zhang <may_zhang@intuit.com> * add back checkAppNotInAdditionalNamespaces Signed-off-by: May Zhang <may_zhang@intuit.com> * add cm and secret to clusterRole Signed-off-by: May Zhang <may_zhang@intuit.com> * if applicationNamespaces is not used, then use namespaced appClient Signed-off-by: May Zhang <may_zhang@intuit.com> * fix doc and test based on review Signed-off-by: May Zhang <may_zhang@intuit.com> * disable defining and using secrets within notification templates for self-service Signed-off-by: May Zhang <may_zhang@intuit.com> * tweaks Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * fix docs formatting Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * more docs and Procfile update for local run convenience Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> --------- Signed-off-by: May Zhang <may_zhang@intuit.com> Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> Co-authored-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* fix: Tooltips point in wrong direction#11935 Signed-off-by: Teng, Jessie <yilin.teng@fmr.com> * fix: Tooltips point in wrong direction#11935 Signed-off-by: Teng <yilin.teng@fmr.com> --------- Signed-off-by: Teng, Jessie <yilin.teng@fmr.com> Signed-off-by: Teng <yilin.teng@fmr.com> Co-authored-by: Teng, Jessie <a715260@fmr.com>

* fix(11164): Advanced templating using patchTemplate Signed-off-by: gmuselli <geoffrey.muselli@gmail.com> * small changes Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> --------- Signed-off-by: gmuselli <geoffrey.muselli@gmail.com> Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> Co-authored-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

…sInAnyNamespaceEnabled flag is set (argoproj#16249) Signed-off-by: Eilers, Jonas <133217951+jdvgh@users.noreply.github.com>

…rgoproj#16062) (argoproj#16241) * fix(appset): store sha from webhook to get latest change during reconcile (argoproj#16062) Signed-off-by: dhruvang1 <dhruvang1@users.noreply.github.com> * fix(appset): Don't use revision cache when reconciling after webhook(argoproj#16062) Signed-off-by: dhruvang1 <dhruvang1@users.noreply.github.com> --------- Signed-off-by: dhruvang1 <dhruvang1@users.noreply.github.com>

Signed-off-by: gmuselli <geoffrey.muselli@gmail.com>

…ssue argoproj#16523) (argoproj#16520) * Update cert-manager.opcertificate health.lua Signed-off-by: Chris Murray <chris@distrail.io> * adding test case for cert issuing Signed-off-by: Chris Murray <chris@distrail.io> * fixing typo Signed-off-by: Chris Murray <chris@distrail.io> --------- Signed-off-by: Chris Murray <chris@distrail.io>

…t platforms (argoproj#16065) Signed-off-by: Ondrej Sika <ondrej@ondrejsika.com>

…rgoproj#15902)

Signed-off-by: Josh Soref <jsoref@gmail.com> Co-authored-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

Wrongly placed horizontal line (`----`) was formatting code-block as a header. Fixed it with a necessary line break Signed-off-by: Elouan Keryell-Even <elouan.keryell@gmail.com>

Signed-off-by: Phil Nichol <35630607+philnichol@users.noreply.github.com>

…6560) Signed-off-by: Jesse Suen <jesse@akuity.io>

…proj#16364) Signed-off-by: Rotem Tamir <rotemtamir@gmail.com>

… allow app's deletion (argoproj#12172) (argoproj#16506) * fix(appset): remove unnecessary condition Signed-off-by: mikutas <23391543+mikutas@users.noreply.github.com> * docs: update explanation about policy Signed-off-by: mikutas <23391543+mikutas@users.noreply.github.com> --------- Signed-off-by: mikutas <23391543+mikutas@users.noreply.github.com>

Signed-off-by: Jay Shah <jay.shah@konghq.com>

…oj#16581) * chore: upgrade kubernetes dependencies from 0.26.4 to 0.26.11 Fixes some vulnerabilities trivy is reporting on (not necessarily vulnerabe, trivy tends to have a lot of false positives when it comes to golang projects): * CVE-2023-3676 * CVE-2023-3955 * CVE-2023-5528 * CVE-2023-2431 * CVE-2023-2727 * CVE-2023-2728 Signed-off-by: Zoltán Reegn <zoltan.reegn@gmail.com> * go mod tidy Signed-off-by: Zoltán Reegn <zoltan.reegn@gmail.com> * Add go mod tidy to kubernetes updater script Signed-off-by: Zoltán Reegn <zoltan.reegn@gmail.com> --------- Signed-off-by: Zoltán Reegn <zoltan.reegn@gmail.com>

Signed-off-by: Elouan Keryell-Even <elouan.keryell@gmail.com>

…anifests ref only Signed-off-by: nromriell <nateromriell@gmail.com>

Signed-off-by: nromriell <nateromriell@gmail.com>

ishitasequeira · 2024-02-06T02:13:14Z

@nromriell Do you mind rebasing the PR with argo-cd/master to be able to get the PR reviewed?

nromriell mentioned this pull request Nov 21, 2023

fix(repo-server): excess git request when using multi-source and ref sources (Issue #14725) argoproj/argo-cd#16309

Closed

11 tasks

nromriell force-pushed the lsremote-part1 branch from d47a64c to c118ff4 Compare November 21, 2023 17:33

dependabot bot and others added 18 commits November 21, 2023 19:34

chore: Bump otel to 1.21.0 (argoproj#16420)

78460c4

This actually wasn't that bad. It was just a few bumps in the right order. I ran tests and built everything and it appears to work. Signed-off-by: Dan Lorenc <dlorenc@chainguard.dev>

chore: Update USERS.md (argoproj#16425)

1600c03

Add Salad Technologies Signed-off-by: Kyle Dodson <kyle@salad.com>

Renamed/corrected OCI proposal filename (argoproj#16452)

0cb9980

Signed-off-by: Andrew Block <andy.block@gmail.com>

fix(ui): User Info blob is too far to the right relative to Applicati…

b34e587

…ons/Settings/Documentation (argoproj#12016) Signed-off-by: Shi, Stone <jiadong.shi@fmr.com>

fix(cli): pass redis compression to cluster stats and shards commands (…

23f2767

…argoproj#16060) (argoproj#16421)

fix PerconaXtraDBCluster health (argoproj#16434)

c602302

Signed-off-by: Filip Rafaj <filip.rafaj@prusa3d.cz> Co-authored-by: Filip Rafaj <filip.rafaj@prusa3d.cz>

Fix extra space in application tree pod group (argoproj#16358)

673d661

Signed-off-by: Ashin Sabu <ashin.sabu@harness.io>

chore(deps): bump cosign-installer from 3.1.2 to 3.2.0 (argoproj#16495)

27e927b

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

nromriell force-pushed the lsremote-part2 branch 7 times, most recently from 1597b4c to c32957c Compare December 1, 2023 03:36

JessieTeng89 and others added 3 commits December 1, 2023 10:07

feat(ui): Add sourceNamespaces in Projects UI and only show it if App…

017b08a

…sInAnyNamespaceEnabled flag is set (argoproj#16249) Signed-off-by: Eilers, Jonas <133217951+jdvgh@users.noreply.github.com>

dhruvang1 and others added 17 commits December 3, 2023 17:54

fix(doc): Fix documentation templatePatch (argoproj#16522)

30b92b2

Signed-off-by: gmuselli <geoffrey.muselli@gmail.com>

fix: Use math.MaxInt (instead of math.MaxInt64) to fix builds on 32bi…

99c2859

…t platforms (argoproj#16065) Signed-off-by: Ondrej Sika <ondrej@ondrejsika.com>

Added missing 'alias:' prefix for repository name as described here: (a…

7484f1d

…rgoproj#15902)

fix(ui): use background delete to match k8s terminology (argoproj#15579)

5c51dcb

Signed-off-by: Josh Soref <jsoref@gmail.com> Co-authored-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

docs: Fix format issue in rbac.md (argoproj#16521)

07a2e64

Wrongly placed horizontal line (`----`) was formatting code-block as a header. Fixed it with a necessary line break Signed-off-by: Elouan Keryell-Even <elouan.keryell@gmail.com>

docs: Fix minor typo in Declarative Setup (argoproj#16550)

86f79ec

Signed-off-by: Phil Nichol <35630607+philnichol@users.noreply.github.com>

chore: update PR template to suggest cherry-pick releases (argoproj#1…

710777e

…6560) Signed-off-by: Jesse Suen <jesse@akuity.io>

resource_customizations/db.atlasgo.io: atlas operator resources (argo…

35f1ee7

…proj#16364) Signed-off-by: Rotem Tamir <rotemtamir@gmail.com>

docs: Add Kong Inc. as a user (argoproj#16582)

f67dcac

Signed-off-by: Jay Shah <jay.shah@konghq.com>

docs: fix broken link in secret-management.md (argoproj#16588)

10bb8b0

Signed-off-by: Elouan Keryell-Even <elouan.keryell@gmail.com>

fix(repo-server): excess git requests part 2, short-circuit GenerateM…

c46f5aa

…anifests ref only Signed-off-by: nromriell <nateromriell@gmail.com>

chore(logging): pr feedback, add debug log to short circuit

4e84097

Signed-off-by: nromriell <nateromriell@gmail.com>

chore: pr feedback, add ref to debug statement

6e08acf

Signed-off-by: nromriell <nateromriell@gmail.com>

nromriell force-pushed the lsremote-part2 branch from 31f0ef7 to 6e08acf Compare December 12, 2023 02:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(repo-server): excess git requests, short-circuit GenerateManifests ref only (Issue #14725) #1

fix(repo-server): excess git requests, short-circuit GenerateManifests ref only (Issue #14725) #1

nromriell commented Nov 21, 2023

ishitasequeira commented Feb 6, 2024

fix(repo-server): excess git requests, short-circuit GenerateManifests ref only (Issue #14725) #1

Are you sure you want to change the base?

fix(repo-server): excess git requests, short-circuit GenerateManifests ref only (Issue #14725) #1

Conversation

nromriell commented Nov 21, 2023

ishitasequeira commented Feb 6, 2024