Repository credentials templates and Helm dependency #11717

Gianluca755 · 2022-12-15T11:55:21Z

Checklist:

I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I've included steps to reproduce the bug.
I've pasted the output of argocd version.

Describe the bug
Given a private Helm chart repository, an application Helm chart with a dependency (subchart) to the private repository.
The repository credentials templates don't work while the normal credentials work correctly.

Expected behavior
The repository credentials should also work for the Helm dependencies (recursively).

Edit
On second thought I think it's the problem is that I'm using repository credential templates without creating the repository resources in argocd. Is this the wrong approach? It works if there are no Helm dependencies.

Version

{
    "Version": "v2.5.2+148d8da",
    "BuildDate": "2022-11-07T16:42:47Z",
    "GitCommit": "148d8da7a996f6c9f4d102fdd8e688c2ff3fd8c7",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.8",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.5.7 2022-08-02T16:35:54Z",
    "HelmVersion": "v3.10.1+g9f88ccb",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.18.0"
}

The text was updated successfully, but these errors were encountered:

blakepettersson · 2022-12-27T10:10:35Z

Are the Helm dependencies OCI charts? If so, a fix (#11327) was added since 2.5.5.

Gianluca755 · 2023-01-04T12:16:53Z

Are the Helm dependencies OCI charts? If so, a fix (#11327) was added since 2.5.5.

Thanks @blakepettersson for the reply but no, the Helm dependency uses HTTP.
Moreover, I don't know if it is a bug or a feature, the key point is that it is unclear (to me) if, in case of repository credentials templates we need to also create the repository resource in Argo CD.

blakepettersson · 2023-01-05T11:45:03Z

Moreover, I don't know if it is a bug or a feature, the key point is that it is unclear (to me) if, in case of repository credentials templates we need to also create the repository resource in Argo CD.

No, that shouldn't be necessary, that should work. It's possible that you might be bumping into the same issue I bumped into in #10218.

jacek-jablonski · 2023-01-07T14:23:27Z

I'm also affected by this issue. I can use OCI repository directly without any issues. But if OCI chart is declared as a subchart, it unfortunately doesn't work.

rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://europe-west4-docker.pkg.dev/xxx/helm/app: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

jacek-jablonski · 2023-01-10T10:42:23Z

So, I managed to take a look and debug this issue and finally got it working.

Here is my secret with credentials template:

data:
  enableOCI: "true"
  type: "helm"
  url: "europe-west4-docker.pkg.dev"
  username: "xxx"
  pasword: "xxx"

Here is my helm chart definition with dependency in OCI repository:

apiVersion: v2
name: docs

type: application
version: 0.1.0

dependencies:
  - name: app
    version: "^0.1.19"
    repository: oci://europe-west4-docker.pkg.dev/myproject/helm

However, what was most surprising to me was that helm repo credentials are treated the same as git repo credentials. So they must be placed as an allowed source in the project where your application is located (screenshot attached). Without this, helm repo credentials will be cut in this function and will never be passed to repository-controller and access to OCI repository will never be configured before generating manifests.

This is very counterintuitive, because in the project configuration, the Source repositories field only mentions git repositories, not helm registries.

@blakepettersson please let me know if it is a bug or it is by design. If it is a bug, I can open a PR.

Gianluca755 · 2023-01-10T12:44:17Z

@jacek-jablonski thanks for taking the time to write the post, but I'm having this problem with http and adding it in the source repository doesn't work.
Duplicate of: #7104
Edit: For making it work I had to: create the repo credential templates, create the repo "placeholder" in Argo CD, then add the source repository URL in the project

jacek-jablonski · 2023-01-10T12:49:00Z

I'm also affected by #7104, but I haven't investigated the reasons for this yet.

jascsch · 2023-01-13T15:33:43Z

We have the same issue and are not able to fix this.
We created the repo and repocreds via argo cli which looks like this:

argocd repocreds add registry.app.corpintra.net --enable-oci --type helm --username 'robot$les+robo' --password xxx argocd repo add registry.app.corpintra.net --enable-oci --type helm --name registry --username 'robot$les+robo' --password xxx --project default

We use a sub helm chart and with a Chart.yaml
`apiVersion: v2
name: les-service

type: application
version: 1.0.0

dependencies:

name: les-service
version: "1.0.0"
repository: oci://registry.app.corpintra.net/les`

The error message is always the same when we try to deploy the app:

Unable to create application: application spec for ass is invalid: InvalidSpecError: Unable to generate manifests in env: rpc error: code = Unknown desc = helm dependency build failed exit status 1: Error: could not download oci://registry.app.corpintra.net/les/les-service: failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://s3-edc.emea.svc.corpintra.net/edcs3dhccaasprodb/docker/registry/v2/blobs/sha256/24/2431b3f41cb1c457ca3828492ab626f446096b6205e297694a913e8e534416c2/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=7cbc312f5c926f7fc1ad%2F20230113%2Feddhcbuckets%2Fs3%2Faws4_request&X-Amz-Date=20230113T153231Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=e9c0e391e10a05307d4a27286ccec4c46553f5fba7fb4751a355fefbdec7c549": Forbidden

Any ideas how this can be fixed?

blakepettersson · 2023-02-02T11:52:57Z

@blakepettersson please let me know if it is a bug or it is by design. If it is a bug, I can open a PR.

Hey @jacek-jablonski this one totally slipped through the cracks. This is by design, but it is not very intuitive. I have a WIP PR which should hopefully make this more visible when this happens, see #12255.

lhriley · 2023-04-17T20:09:01Z

xposting this workaround: #12436 (comment)

plaisted · 2023-08-01T17:38:17Z

Just a heads up to anyone who may be having this issue as well, there was a breaking behavior change in argocd, don't know exact versions.

Previously a chart referencing a oci helm depedency like oci://<account>.dkr.ecr.us-east-1.amazonaws.com/<sub-path> worked fine with a secret having a URL like <account>.dkr.ecr.us-east-1.amazonaws.com. This no longer works in v2.7.8 (don't know exactly when it broke, we had this setup working for a year previously). Now the secret is required to match the repository url (minus the oci://) so needs to be <account>.dkr.ecr.us-east-1.amazonaws.com/<sub-path> in this example.

apiVersion: v2
name: helm-name
type: application
version: 0.0.0
dependencies:
  - name: char-name
    version: "chart-varion"
    repository: 'oci://<account>.dkr.ecr.us-east-1.amazonaws.com/<sub-path>'

apiVersion: v1
kind: Secret
metadata:
  name: $secretName
  namespace: $ns
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  url: "<account>.dkr.ecr.us-east-1.amazonaws.com" # NO LONGER WORKS FOR DEPENDENCIES!! Used to work for direct chart reference AND dependencies
  url: "<account>.dkr.ecr.us-east-1.amazonaws.com/<sub-path>" # Required now for OCI chart dependencies
  name: ecr
  type: helm
  enableOCI: "true"
  username: AWS
  password: "<password>

lhaussknecht · 2023-08-14T16:00:37Z

Thank's @plaisted.
Unfortunately this is a absolute mess. We would need to scan all our charts and compile a list of sub-paths for dependencies and create repository secrets out of it.

It worked in v2.7.4 and before for a very long time.

blakepettersson · 2023-08-14T16:06:55Z

Isn't this the issue that was fixed in #14699 (and hence in >= 2.7.10)?

lhaussknecht · 2023-08-14T16:11:10Z

Just tried it with the latest helm chart 5.43.3 and v2.8.0+804d4b8 of argo-cd and it is not working with parts of the repository.

plaisted · 2023-08-14T17:10:19Z

We updated from v2.7.4 to v2.7.8 and saw the issue. We have automation that syncs our ECR creds to cluster so wasn't a huge effort to add the sub-repos (note you don't need one for each chart, just each prefix if you are using them).

We did deploy v2.8.0 in one cluster and didn't see issues so think it may be resolved but will dig in a little more to verify since @lhaussknecht is saying it is not working.

lhaussknecht · 2023-09-28T10:45:19Z

Just tried again with v2.8.4 and it is still a problem. I also tried to use sub-paths in our secrets and we still get "unexpected status from HEAD request" with 401 Unauthorized.

Guitarkalle · 2024-01-11T14:28:37Z

Also have the same issue on v2.8.4, it seems a bit random though, but mostly not working

awx-fuyuanchu · 2024-04-02T16:19:48Z

Hi guys, I have a similar issue using the credential template on v2.4. But I found a workaround for my case.

Only the applications in the AppProject that contain specific sourceRepos can't work with the credential template. If I set the sourceRepos to *, then the credential template works for the applications. Maybe you can have a try to see if it works.

Gianluca755 · 2024-07-12T10:45:06Z

I finally found some energy to experiment with a more recent Argo CD version: v2.11.4 in Helm chart 7.3.4.
I could have made some mistakes due to the credentials' cache, but I think that I click hard refresh for each app.

If anyone wants the credentials let me know.

Assets:

Unified credential template without specifying the project

apiVersion: v1
kind: Secret
metadata:
  name: gitlab-access-token
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  url: 'https://gitlab.com'
  username: test
  password: glpat-

Separated credential templates without specifying the project

apiVersion: v1
kind: Secret
metadata:
  name: gitlab-access-token-1
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  url: 'https://gitlab.com'
  username: test
  password: glpat-
  type: git
---
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-access-token-2
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  url: 'https://gitlab.com'
  username: test
  password: glpat-
  type: helm

ArgoProject that allows all repositories

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: open-project
spec:
  destinations:
  - name: '*'
    namespace: '*'
    server: '*'
  sourceRepos:
  - '*'

ArgoProject with allowlist

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: closed-project
spec:
  destinations:
  - name: '*'
    namespace: '*'
    server: '*'
  sourceRepos:
  - 'https://gitlab.com/argocd-helm-11717/test'
  - 'https://gitlab.com/api/v4/projects/59757174/packages/helm/stable'

Helm chart with subchart (in https://gitlab.com/argocd-helm-11717/test)

apiVersion: v2
name: my-argo
description: My amazing decorated chart
type: application
appVersion: "1.0.0"
version: 0.1.0
dependencies:
  - name: argo-cd
    version: 7.3.4
    repository: https://gitlab.com/api/v4/projects/59757174/packages/helm/stable

'open' app with open project

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: open
spec:
  destination:
    name: ''
    namespace: default
    server: 'https://kubernetes.default.svc'
  source:
    path: .
    repoURL: 'https://gitlab.com/argocd-helm-11717/test.git'
    targetRevision: main
    helm:
      valueFiles:
        - values.yaml
  project: open-project

'closed' app with closed project

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: closed
spec:
  destination:
    name: ''
    namespace: default
    server: 'https://kubernetes.default.svc'
  source:
    path: .
    repoURL: 'https://gitlab.com/argocd-helm-11717/test.git'
    targetRevision: main
    helm:
      valueFiles:
        - values.yaml
  project: closed-project

fix-credentials

apiVersion: v1
kind: Secret
metadata:
  name: fix-credentials
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  name: fix-credentials
  type: helm
  url: https://gitlab.com/api/v4/projects/59757174/packages/helm/stable

Some resulting combinations:

unified credential template and open app
Error: looks like "https://gitlab.com/api/v4/projects/59757174/packages/helm/stable" is not a valid chart repository or cannot be reached: failed to fetch https://gitlab.com/api/v4/projects/59757174/packages/helm/stable/index.yaml : 401 Unauthorized
unified credential template and closed app
Error: looks like "https://gitlab.com/api/v4/projects/59757174/packages/helm/stable" is not a valid chart repository or cannot be reached: failed to fetch https://gitlab.com/api/v4/projects/59757174/packages/helm/stable/index.yaml : 401 Unauthorized
separated credential templates and open app
Error: looks like "https://gitlab.com/api/v4/projects/59757174/packages/helm/stable" is not a valid chart repository or cannot be reached: failed to fetch https://gitlab.com/api/v4/projects/59757174/packages/helm/stable/index.yaml : 401 Unauthorized
separated credentials templates and closed app
Error: looks like "https://gitlab.com/api/v4/projects/59757174/packages/helm/stable" is not a valid chart repository or cannot be reached: failed to fetch https://gitlab.com/api/v4/projects/59757174/packages/helm/stable/index.yaml : 401 Unauthorized
unified credential template, open app and fix-credentials
It works.
unified credential template, closed app and fix-credentials
It works.
separated credential templates, open app and fix-credentials
It works.
separated credential templates, closed app and fix-credentials
It works.
unified credentials template with specified project (open-project) and open app
Error: looks like "https://gitlab.com/api/v4/projects/59757174/packages/helm/stable" is not a valid chart repository or cannot be reached: failed to fetch https://gitlab.com/api/v4/projects/59757174/packages/helm/stable/index.yaml : 401 Unauthorized
unified credentials template with specified project (closed-project) and closed app
Error: looks like "https://gitlab.com/api/v4/projects/59757174/packages/helm/stable" is not a valid chart repository or cannot be reached: failed to fetch https://gitlab.com/api/v4/projects/59757174/packages/helm/stable/index.yaml : 401 Unauthorized

@blakepettersson does this clarify the issue? Or is it an undocumented "feature"?

Version

{
    "Version": "v2.11.4+e1284e1",
    "BuildDate": "2024-07-02T19:17:02Z",
    "GitCommit": "e1284e19e03c9abab2ea55314b14b1e0381c4045",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.4+g81c902a",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}

Gianluca755 added the bug Something isn't working label Dec 15, 2022

MioOgbeni mentioned this issue Sep 4, 2023

Unable to use OCI Registry with sub helm charts #12436

Closed

lhaussknecht mentioned this issue Oct 9, 2023

Pulling helm charts from GCP Artifact Registry fails after upgrading Argo to v2.8.2 #15344

Closed

3 tasks

alexmt added component:helm type:bug labels Jul 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository credentials templates and Helm dependency #11717

Repository credentials templates and Helm dependency #11717

Gianluca755 commented Dec 15, 2022 •

edited

Loading

blakepettersson commented Dec 27, 2022

Gianluca755 commented Jan 4, 2023 •

edited

Loading

blakepettersson commented Jan 5, 2023

jacek-jablonski commented Jan 7, 2023

jacek-jablonski commented Jan 10, 2023 •

edited

Loading

Gianluca755 commented Jan 10, 2023 •

edited

Loading

jacek-jablonski commented Jan 10, 2023

jascsch commented Jan 13, 2023

blakepettersson commented Feb 2, 2023

lhriley commented Apr 17, 2023

plaisted commented Aug 1, 2023

lhaussknecht commented Aug 14, 2023

blakepettersson commented Aug 14, 2023

lhaussknecht commented Aug 14, 2023

plaisted commented Aug 14, 2023

lhaussknecht commented Sep 28, 2023

Guitarkalle commented Jan 11, 2024 •

edited

Loading

awx-fuyuanchu commented Apr 2, 2024

Gianluca755 commented Jul 12, 2024

Repository credentials templates and Helm dependency #11717

Repository credentials templates and Helm dependency #11717

Comments

Gianluca755 commented Dec 15, 2022 • edited Loading

blakepettersson commented Dec 27, 2022

Gianluca755 commented Jan 4, 2023 • edited Loading

blakepettersson commented Jan 5, 2023

jacek-jablonski commented Jan 7, 2023

jacek-jablonski commented Jan 10, 2023 • edited Loading

Gianluca755 commented Jan 10, 2023 • edited Loading

jacek-jablonski commented Jan 10, 2023

jascsch commented Jan 13, 2023

blakepettersson commented Feb 2, 2023

lhriley commented Apr 17, 2023

plaisted commented Aug 1, 2023

lhaussknecht commented Aug 14, 2023

blakepettersson commented Aug 14, 2023

lhaussknecht commented Aug 14, 2023

plaisted commented Aug 14, 2023

lhaussknecht commented Sep 28, 2023

Guitarkalle commented Jan 11, 2024 • edited Loading

awx-fuyuanchu commented Apr 2, 2024

Gianluca755 commented Jul 12, 2024

Version

Gianluca755 commented Dec 15, 2022 •

edited

Loading

Gianluca755 commented Jan 4, 2023 •

edited

Loading

jacek-jablonski commented Jan 10, 2023 •

edited

Loading

Gianluca755 commented Jan 10, 2023 •

edited

Loading

Guitarkalle commented Jan 11, 2024 •

edited

Loading