feat: Multiple sources for applications

[ishitasequeira]

Note on DCO:

If the DCO action in the integration test fails, one or more of your commits are not signed off. Please click on the Details link next to the DCO action for instructions on how to resolve this.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).

Closes #677
Closes #2789

Description

This change enables users to provide multiple resources for the Application. The change aims to be fully backwards compatible.

This PR implements proposal #8322.

Change details

Adding Multiple Sources

The new field sources would allow users to input list of ApplicationSources. We would ignore the source field we can find even a single entry under sources field.

Below example shows how the yaml would look like for source and sources field. We would ignore the source field and apply the resources mentioned under sources field.




spec:
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    path: guestbook
    targetRevision: HEAD
  sources:
    - chart: elasticsearch
      helm:
        valueFiles:
          - values.yaml
      repoURL: https://helm.elastic.co
      targetRevision: 7.6.0
    - repoURL: https://github.com/argoproj/argocd-example-apps.git
      path: guestbook
      targetRevision: HEAD

Multiple Value files for Helm Source

For making files accessible to other sources, add a new ref field to the source. For example, in below ApplicationSpec, we have added ref: myRepo to the my-repo repository and have used reference $myRepo to the elasticSearch repository.

spec:
  sources:
    - repoURL: https://github.com/my-org/my-repo  # path is missing so no manifests are generated
      targetRevision: master
      ref: myRepo                                 # repo is available via symlink "myRepo"
    - repoURL: https://github.com/helm/charts
      targetRevision: master
      path: incubator/elasticsearch               # path "incubator/elasticsearch" is used to generate manifests
      helm:
        valueFiles:
          - $myRepo/values.yaml                   # values.yaml is located in source with reference name $myRepo

How to test manually

• run locally, use make start or make start-local
• Create an application with sources field. Below is one example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
  labels:
    argocd.argoproj.io/refresh: hard
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  sources:
    - chart: elasticsearch
      helm:
        valueFiles:
          - values.yaml
      repoURL: https://helm.elastic.co
      targetRevision: 7.6.0
    - repoURL: https://github.com/argoproj/argocd-example-apps.git
      path: guestbook
      targetRevision: HEAD
    - chart: elasticsearch
      helm:
        valueFiles:
          - values.yaml
      repoURL: https://helm.elastic.co
      targetRevision: 7.7.0

• Try different things - app update, rollback, remove sources and add a single source under source field, value files for helm sources, etc.

[codecov]

Codecov Report

Attention: Patch coverage is 60.24590% with 388 lines in your changes missing coverage. Please review.

Project coverage is 47.19%. Comparing base (a2d756e) to head (bbb9240).
Report is 2232 commits behind head on master.

Files with missing lines	Patch %	Lines
controller/state.go	63.22%	43 Missing and 14 partials ⚠️
util/argo/argo.go	74.43%	41 Missing and 16 partials ⚠️
reposerver/repository/repository.go	70.22%	40 Missing and 13 partials ⚠️
cmd/util/app.go	9.80%	45 Missing and 1 partial ⚠️
server/application/application.go	24.56%	34 Missing and 9 partials ⚠️
cmd/argocd/commands/app.go	19.56%	36 Missing and 1 partial ⚠️
controller/appcontroller.go	62.76%	29 Missing and 6 partials ⚠️
util/webhook/webhook.go	48.57%	17 Missing and 1 partial ⚠️
controller/sync.go	63.63%	7 Missing and 9 partials ⚠️
reposerver/cache/cache.go	71.05%	10 Missing and 1 partial ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10432      +/-   ##
==========================================
+ Coverage   47.01%   47.19%   +0.17%     
==========================================
  Files         243      243              
  Lines       40587    41088     +501     
==========================================
+ Hits        19084    19393     +309     
- Misses      19583    19734     +151     
- Partials     1920     1961      +41     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ArshamTeymouri]

Thanks for the PR! I'm looking forward to v2.5.0.

Just a question about plugin, how will these remote sources work in case of using plugins? In our case, we use the helm-secrets plugin to fetch secrets directly from Vault by an argocd custom plugin. I wondered how I can integrate that with remote values files from git without doing so much manual scripting.

[amohamedhey]

Thanks for the PR!

[ishitasequeira]

Thanks for the PR! I'm looking forward to v2.5.0.

Just a question about plugin, how will these remote sources work in case of using plugins? In our case, we use the helm-secrets plugin to fetch secrets directly from Vault by an argocd custom plugin. I wondered how I can integrate that with remote values files from git without doing so much manual scripting.

Hi @amohamedhey, we are not going to support CMP with initial release of multiple sources in v2.5, but the feature would follow up as an extension to this feature in future releases.

[jannfis]

First round of superficial style check. PTAL.

I'm gonna test the PR locally.

[tback]

It looks as if this addresses #677, yet I didn't find the issue mentioned here.

[crenshaw-dev]

@tback updated the PR description, thanks!

[jannfis]

I still have issues testing this code locally. What is the proper way to use the new sources field?

Is it complementary to source, or is a source xor sources decision?

  spec:
    destination:
      namespace: testcmp
      server: https://kubernetes.default.svc
    project: default
    sources:
    - path: guestbook
      repoURL: https://github.com/argoproj/argocd-example-apps
      targetRevision: HEAD
    syncPolicy:
      syncOptions:
      - AutoCreateNamespace=true
  status:
    conditions:
    - lastTransitionTime: "2022-09-14T09:21:19Z"
      message: 'rpc error: code = Internal desc = Failed to fetch default: `git fetch
        origin --tags --force` failed exit status 128: fatal: no path specified; see
        ''git help pull'' for valid url syntax'
      type: ComparisonError
    - lastTransitionTime: "2022-09-14T09:25:10Z"
      message: spec.source.repoURL and spec.source.path either spec.source.chart are
        required
      type: InvalidSpecError
    health:
      status: Unknown

[amohamedhey]

@ishitasequeira How can I get the generated image and test?

[crenshaw-dev]

@amohamedhey clone Ishita's fork, check out this branch, and run make image TAG=multisource. It'll save the image tagged argocd:multisource.

[crenshaw-dev]

I'm getting an index out of range panic (index 0) here: https://github.com/ishitasequeira/argo-cd/blob/ce5a3b0b16cffe3af3dd3f79ea14fda4d07aeef5/controller/state.go#L168

[crenshaw-dev]

I wonder how we should expect .argocd-source to behave for a multi-source app.

[crenshaw-dev]

@ishitasequeira can you add docs, particularly for the external-values-file use case? I think that'll also help reviewers.

We should document what happens when two sources contain the same resource. I think Argo CD will apply both resources in sequence, with the latter being the resource that is finally applied and shown in the UI. We should document that because, if the first source contains a malicious manifest which is overridden by a later source, only the latter source will be visible in the UI. So, for example, if source 1 contains a Deployment manifest with malicious code and source 2 contains the same Deployment without malicious code, then I believe Argo CD will apply the malicious Deployment and then very quickly apply the non-malicious Deployment. None of that will really be apparent in the UI.

[crenshaw-dev]

What happens if I have another source from this same repoURL but with a different path and/or targetRevision?

[romuduck]

Same question with using externalValueFiles/repoURL on top:

• does the repoURL uniqueness check applies only for sources/repoURL?
• what if for one source i have multiple externalValueFiles pointing to the same repoURL, but different versions, different files?
• what if i have 2 sources pointing to the same repoURL but using different valueFiles or externalValueFiles?

Do we keep a kind of (repo, chart/path) couple for duplicate checks?

Edit:
Just remembered the "externalValueFiles" was reworked with using reference to a sources/repoURL.
But I guess the question remains.

[KrasimirAntonov]

Seeing this PR move forward, I think this proposal could be interesting.
Clone the different sources into the same {uuid} directory to access them with the repository name
Example:
If I have two git sources Test A (https://github.com/test/test-a.git) and Test B(https://github.com/test/test-b.git), I'd like the repo-server to clone them into /tmp/_argocd-repo/{uuid} like this:

/tmp/_argocd-repo/{uuid}/test-a
/tmp/_argocd-repo/{uuid}/test-b

Also, it would be interesting if repo-server first clone the repositories in the order they are defined in the sources field of the application.yaml file and then performs the apply or build tasks.

[crenshaw-dev]

@KrasimirAntonov I think it could definitely be done that day. It would also reduce lock contention, at the cost of disk space/time. What additional use case(es) would this solve?

[KrasimirAntonov]

Thanks for your quick response Michael!
In multi-repository approach, it resolves the deployment dependencies of the environments. Imagine I have two deployment environments (dev and pro) and these environments have common settings like license. With this function I can create a common environment/repository (let's call it: base) and inside it I can put the common configuration for all environments and change only once the license for all the environments. Significantly reduces time

[zenitraM]

I've got another related question (happy to create another issue if desired for this!) - are there any plans or ideas on how to expose multiple repos to custom CMPs?

The first obvious case I have in mind would be to use this with argocd-vault-plugin but also possibly with other custom CMPs. It would be nice if we were able to pass as a parameter the path to a file in another repo by reference somehow (to be able to replicate what the built-in Helm support does)

[crenshaw-dev]

@KrasimirAntonov sure, makes sense... I wonder if that's wandering too far into the territory of Argo CD becoming its own Config Management tool. But I'm curious. Would you be up for opening an issue for further discussion?

[crenshaw-dev]

@zenitraM good thought! That definitely deserves a new issue, since a lot of folks use Helm in their CMPs. Would you mind opening an issue?

[ShirMonDT]

Hey guys,
Been waiting for this PR!
Came across a problem with my use-case, I want to reference the same source with different versions, but I'm getting a comparison error;

Is there a reason for not being able to use the same repo with different refs?

[fad3t]

Hi,

That's a cool feature, thanks for this!
I'm trying to use multiple sources but I keep getting error messages such as:

• missing required field "source" in io.argoproj.v1alpha1.Application.spec]
• unknown field "sources" in io.argoproj.v1alpha1.Application.spec

I'm using ArgoCD argocd:v2.6.0-rc1.

Am I missing something?
Thx,
Fred

[jannfis]

Am I missing something?

@fad3t I think you may want to update your installation manifests, especially the CRDs, as well.

[fad3t]

Oh right, indeed :)

[danhawkins]

Love this feature, thanks so much

[omri-shilton]

any update on the UI side of this feature?

[crenshaw-dev]

@omri-shilton afaik this is the only ongoing frontend-related work: https://github.com/argoproj/argo-cd/pull/14124/files

Full support is "planned," but just in the sense that it obviously needs to be done - afaik there's no ongoing, concerted effort to write the necessary code.

[xgt001]

Thank you for this wonderful feature. Works dandy!