ArgoCD CMP sub resource/repositiory credentials

[ghost]

Describe the bug

CMP plugins are the proposed way to extend ArgoCD chart rendering, and according to the documentation/proposal should push plugins forward to become a first-class citizen.
That may hold up for use cases where the argocd-repo-server can fetch all dependencies (maybe there are none though) and push the results down to the CMP to render.

But as soon as you must fetch additional resources from Git, OCI, Helm repos, there are no credentials available out of the box to fulfill this task, even though all needed credentials already exist in the ArgoCD repository configs.

This missing credential push down functionallity is blocking all use cases that need to fetch additional resources like.
e.g.:

Kustomize with kustomizations pointing to sub-Git repositories.
Helm umbrella charts where sub charts need to get loaded via dependency updates from OCI or other Helm registries.
...

There is a bunch of open issues all framing the same situation as

• Unable to use OCI Registry with sub helm charts #12436
• Sidecar CMP plugin cannot use private dependencies #10265
• Argocd-2.4.17 CMP plugin fails to "kustomize build ." on remote repo. Getting Error: accumulating resources due to git credentials #11318

Just to list some of the more prominent ones.

This current behavior leads to many workarounds that require mapping arbitrary credentials into the CMP sidecar, and is blocking CMP adoption for all use cases where the argocd-repo-sever can't provide everything in one go. (when talking about non public dependencies, which is more or less default in an enterprise setup)

To Reproduce

• Create a simple CMP plugin that just does a Helm template, like https://github.com/argoproj/argo-cd/tree/master/examples/plugins/helm
• Create a repository in ArgoCD for the private Git repo homing the umbrella chart

apiVersion: v2
name: umbrella-chart
type: application
version: 0.9.0
dependencies:
  - name: sample-chart
    version: 0.1.0
    repository: oci://<WhateverSubChart>

• Place the subchart in an OCI registry
• Create a repository in ArgoCD for the private OCI registry homing the sub chart
• Create an application using the plugin pointing to the Helm umbrella chart

Expected behavior

Using plugins should behave similar in functionality as using native Helm & Kustomize support in ArgoCD
That does not mean, that no additional effort is expected, like linking repositories of interest (within the application? project?) for credential pushdown candidates, or whatever solution would do the trick to not maintain the same credentials on different layers.

Version

quay.io/argoproj/argocd:v2.8.4

Logs

time="2023-10-24T09:35:59Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-10-24T09:35:59Z" grpc.time_ms=4.83 span.kind=server system=grpc
time="2023-10-24T09:35:59Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-10-24T09:35:59Z" grpc.time_ms=4.831 span.kind=server system=grpc
time="2023-10-24T09:35:59Z" level=info msg="Generating manifests with no request-level timeout"
time="2023-10-24T09:35:59Z" level=info msg="sh -c \"helm dependency build || true\"" dir=/tmp/_cmp_server/49b83398-ff26-459b-9090-104ccd9608b9/umbrella-chart execID=b0044
time="2023-10-24T09:35:59Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=GetParametersAnnouncement grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-10-24T09:35:59Z" grpc.time_ms=5.377 span.kind=server system=grpc
time="2023-10-24T09:36:00Z" level=debug msg="Saving 1 charts\nDownloading sample-chart from repo oci:/<removed url>/helm\nSave error occurred:  could not download oci://<removed url>/helm/sample-chart: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden\n" duration=1.047357626s execID=b0044
time="2023-10-24T09:36:00Z" level=info msg="sh /var/run/argocd/helm-plugin/generate.sh" dir=/tmp/_cmp_server/49b83398-ff26-459b-9090-104ccd9608b9/umbrella-chart execID=6cf60
time="2023-10-24T09:36:00Z" level=debug duration=359.689583ms execID=6cf60
time="2023-10-24T09:36:00Z" level=error msg="`sh /var/run/argocd/helm-plugin/generate.sh` failed exit status 123: Error: An error occurred while checking for chart dependencies. You may need to run `helm dependency build` to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: sample-chart" execID=6cf60
time="2023-10-24T09:36:00Z" level=error msg="finished streaming call with code Unknown" error="error generating manifests: `sh /var/run/argocd/helm-plugin/generate.sh` failed exit status 123: Error: An error occurred while checking for chart dependencies. You may need to run `helm dependency build` to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: sample-chart" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-10-24T09:35:59Z" grpc.time_ms=1412.54 span.kind=server system=grpc

[blakepettersson]

Could potentially be solved with #15107

[ghost]

@blakepettersson, thx for linking!

If it is a more generic approach, not focusing on Git credentials only it may can be a solution (thinking of the OCI part which may needs another way of providing the creds)

[jmcshane]

I appreciate the more generic approach that @FeltlP is taking to this problem, there's a challenge in identifying a valid credentials set that would be applicable to push to the sidecar. I'd love to discuss this as I've got the context from the work in #15107, feel free to reach out on the CNCF slack

[jmcshane]

One of the interesting things I just found is the difference between git.Creds (interface) and helm.Creds (struct) both in the nasty util package. There's also db.RepoCredsDB.

I wonder if it makes sense to have a higher level Creds interface that could be implemented by all secret-type: repository credentials that we could then use to pass across to CMP

[thesuperzapper]

I think most of the discussion around this issue is happening on #8820

[Xkynar]

@crenshaw-dev #8820 seems to have implemented a way to provide git credentials to CMPs, but I don't think it works for helm/oci credentials right?