2.12.x throws could not read Username for 'https://gitlab.com' from the UI

[c-buisson]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Since 2.12.x, the UI throws this error message:

Unable to load data: error acquiring repo lock: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: fatal: could not read Username for 'https://gitlab.com/': terminal prompts disabled

To Reproduce

Upgrade to 2.12.x from 2.11.x with connectivity to Gitlab repositories.
Using Helm chart version 7.4.4

Expected behavior

UI able to refresh Applications from Gitlab.

Screenshots

argocd: v2.11.2+25f7504
  BuildDate: 2024-05-23T13:58:43Z
  GitCommit: 25f7504ecc198e7d7fdc055fdb83ae50eee5edd0
  GitTreeState: clean
  GoVersion: go1.21.10
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.12.1+26b2039
  BuildDate: 2024-08-16T16:42:13Z
  GitCommit: 26b2039a55b9bdf807a70d344af8ade5171d3d39
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.4.2 2024-05-22T15:19:38Z
  Helm Version: v3.15.2+g1a500d5
  Kubectl Version: v0.29.6
  Jsonnet Version: v0.20.0

Logs

`git fetch origin --tags --force --prune` failed exit status 128: fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled

[argo1984]

Hi together,

i have the same issue with 2.12.X and it seems that the login to the repository works (it is green) but the Projects and Applications are not able to access the credentials.

[blakepettersson]

@c-buisson and @argo1984 can you give the specs of the applications which are failing, and the (redacted) specs of the repo credentials that are expected to work with these applications? I suspect shenanigans from #18388.

[crenshaw-dev]

+1 to Blake's question, but to anyone who's experiencing this issue. :-)

[mozemke]

hi there,

we're facing the same problems and I read through #19587 IMO we should not be affected with our setup but are still getting the listed error.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: project-1-appset
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://gitlab.xxxxx/project-1/deployment.git
        revision: dev
        directories:
        - path: '*/overlays/*'
        - path: '*/overlays/prod*'
          exclude: true
        - path: '*/overlays/xx*'
          exclude: true
        values:
          appName: '{{index .path.segments 0}}-{{.path.basenameNormalized}}'
  template:
    metadata:
      name: '{{.values.appName}}'
      labels:
        env: '{{.path.basenameNormalized}}'
    spec:
      project: < project-1 >
      source:
        repoURL: https://gitlab.xxxxx/project-1/deployment.git
        targetRevision: dev
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'xx-{{.values.appName}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
        managedNamespaceMetadata:
          labels:
            app.kubernetes.io/instance: '{{.values.appName}}'
          annotations: 
            field.cattle.io/projectId: xx
            argocd.argoproj.io/tracking-id: xx
---
# add creds for appset in admin project
apiVersion: v1
kind: Secret
metadata:
  name: xxx-admin-repo-creds
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://gitlab.xxxxx/project-1/deployment.git
  username: xxxx
  password: xxxx
  project: < admin-project >
---
# add creds for project-1 apps
apiVersion: v1
kind: Secret
metadata:
  name: xxx-xxx-repo-creds
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://gitlab.xxxxx/project-1/deployment.git
  username: xxxx
  password: xxxx
  project: < project-1 >

the appset of project-1 is deployed to the admin-project, that's why we are having the same secret two times, but with different project scope, so that the admin project is able to generate the apps. If now the new way to match repo creds to apps is using project + url - I don't see why our setup is failing :(

The apps themselves have no connection problems.

error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo https://gitlab.xxxxx/project-1/deployment.git with revision e32c9a160962137771b976eff8c82c0c28bad41a: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: fatal: could not read Username for 'https://gitlab.xxxx': terminal prompts disabled

We are facing this problem with all our Appsets, above is just one example.

[blakepettersson]

I suspect it's the same thing actually :( An applicationset doesn't have a project; it's admin-scoped. I believe the resolution is exactly the same as for #19587 - having a repo-cred that doesn't have a project set.

[mozemke]

yep that did indeed fix it. I now understand what you mean :D

We actually deploy our appsets, with one bootstrapping app, that one has set a project. But since AppSets do not have the key, it all makes sense now! :)

The downside now is, that the repo creds can now be used by all projects.

thank you! :)

[blakepettersson]

The downside now is, that the repo creds can now be used by all projects.

Yes, that is true; that was however the same behavior in < 2.12. A workaround for that would be to disallow that source to be used in your AppProjects:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: < project-1 >
spec:
  sourceRepos:
    - '!https://gitlab.xxxxx/project-1/deployment.git'

[mozemke]

that was however the same behavior in < 2.12.

I actually never knew that before I read through the issue/pr to enhance the situation :D
We'll implement your workaround! :)

thanks! :)

[ChristianCiach]

I am confused. I am seeing this error after upgrading from v2.11.7 to v.2.12.2, but after reading this and the linked issue, I don't think I should be affected, because my Secret doesn't have a project set:

apiVersion: v1
kind: Secret
metadata:
  labels:
    argocd.argoproj.io/secret-type: repository
  name: repocred-gitoops
  namespace: argocd
type: Opaque
data:
  password: base64(somePassword)
  url: base64(https://internal-url/some-group/gitoops.git)
  username: base64(gitlab+deploy-token-62)

---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: bootstrap:argoproj.io/AppProject:argocd/egs-external
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  labels:
    argocd.argoproj.io/instance: bootstrap
  name: egs-external
  namespace: argocd
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  description: EGS External Cluster
  destinations:
  - name: egs-external
    namespace: '*'
  sourceRepos:
  - '*'

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/tracking-id: bootstrap:argoproj.io/Application:argocd/netshoot
  labels:
    argocd.argoproj.io/instance: bootstrap
  name: netshoot
  namespace: argocd
spec:
  destination:
    name: egs-external
    namespace: default
  project: egs-external
  source:
    path: netshoot
    repoURL: https://internal-url/some-group/gitoops.git
    targetRevision: HEAD

I've tried to debug this for hours, but I am none the wiser.

[blakepettersson]

@ChristianCiach from what I can see you should indeed be unaffected - does rolling back to 2.11.x solve your issue?

[ChristianCiach]

@blakepettersson Yes, downgrading immediately fixes all apps, and upgrading immediately triggers the issue again. Weird..

[blakepettersson]

@ChristianCiach are you sure that there are no other repo-creds making use of the same repo url?

[ChristianCiach]

@blakepettersson This is a small test cluster. There is a single other repo-cred (not technically, they are both argocd.argoproj.io/secret-type: repository, not repocreds), but it is referencing another project on the same gitlab server.

[blakepettersson]

That other argocd.argoproj.io/secret-type: repository secret - is it referring to the same repo url?

[ChristianCiach]

It is not. It is used for my bootstrap application (which belongs to project default) to set up the other AppProjects and Applications.

Unsurprisingly, deleting this other secret breaks the bootstrap app, but doesn't fix all the other apps.

[blakepettersson]

@ChristianCiach are you around on the CNCF slack?

[ChristianCiach]

@blakepettersson Not yet, but I could change that :) I was just about to leave for vacation for the next three weeks, but this is too interesting to let go. Give me a few minutes to figure out how to join the slack channel.

Btw, the crazy thing is that I can sync the applications just fine! Argo CD picks up the changes from the repo without issues. It's just the UI that shows this error message all the time...

[ChristianCiach]

Weird... The error message disappears from the UI as soon as I sync the application, even if there are no changes. I need to do this once for each application, even though they are all synced from the same git repository. But the error message is gone for good after syncing each application once. (We don't use auto-sync in this cluster)

[ChristianCiach]

This may or may not have something to do with the fact that we moved the gitlab project from one group to another group (on the same server) last week. But we did this way before upgrading Argo CD to v2.12, including updating all Applications and Secrets to the new repo url. But maybe the old repo url was still cached somewhere and only triggered this issue with v2.12...?

Anyway, I seem to be the only one affected by this, so this probably has something to do with dirty caches and the recently moved git repo, so we probably shouldn't think too hard about this. I probably should have started with this information, so I'm sorry about that!

[moonkaktus]

So private (project-scoped) repositories now can not be used in other projects, similar to clusters. It seems right, but for clusters we have permitOnlyProjectScopedClusters in appProject, so one can disable or enable such behavior per project. Shouldn't such setting exist for repos now too? Take my case:

• I want to set permitOnlyProjectScopedClusters: true for user projects and forbid deployments in ArgoCD cluster
• I want users to use applicationSets and applications defined in their repos with yaml

Apps and appsets can only reside in ArgoCD cluster (can be in separate namespace, but still) so I do:

• project for user with permitOnlyProjectScopedClusters: true, appset/app repo bound to this project (by user)
• another supplementary project with permitOnlyProjectScopedClusters: false, which has above repo in sourceRepos and where master app for this repo is created. This project can not be used by user, only by admin (automated)

With 2.11 this workflow is possible, but with 2.12 master app will throw error, because supplementary project no longer can access repo (it is bound to another project). I think it still possible, because now we have ability to add duplicate repo to supplementary project with same credentials. But it adds another burden of duplicating user-provided credentials, not to mention it looks awful.

Some project setting like permitOnlyProjectScopedRepositories will solve such problems at project level, so we don't have to use duplicate repo hacks or permit to use them globally.

[yevon]

Same here, I had to downgrade as I having random issues with this message. I use the matrix generator with git files and clusters. I specified the project in the cluster secret as stated in the upgrade instructions, but getting this anyway. The git credentials do also must have the project set in? All matrix generators working just fine in v2.11.7, but in v2.12.3 I'm receiving this. Applications not using the matrix generator do not have this message.

[blakepettersson]

@yevon in the case of applicationsets you would need to have the project parameter unset.

[blakepettersson]

@moonkaktus having a permitOnlyProjectScopedRepositories field for the project is not a bad idea, it has been previously proposed but hasn't been implemented.

In that case it would mean that for a given project it would not be allowed to use globally scoped repositories?

[moonkaktus]

@blakepettersson I think global repositories should be allowed to every project, as now. But restriction to use only project-scoped repositories of one project should be switchable for that project (like permitOnlyProjectScopedClusters).
This way we can have restricted projects where users can add their repos, and not allowed to use repos of other projects, but still have ability to create technical projects without such restriction.

[yevon]

I do use ApplicationSet, but I specify the project for the Application spec itself, not for the ApplicationSet spec. Do I must unset the Application project? I just have this but it was not working:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test
  namespace: argocd
  labels:
    name: test-appset-label
spec:
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  argocd.argoproj.io/secret-type: cluster
          - git:
              repoURL: 'https://testurl.com'
              revision: "{{metadata.annotations.targetRevision}}"
              files:
              - path: "example/path"
            selector:
                  matchExpressions:
                    - key: client.nodeBackend
                      operator: In
                      values:
                        - "true"                  
  template:
    metadata:
      name: "application-name"
      annotations:
        argocd.argoproj.io/compare-options: ServerSideDiff=false
        argocd-image-updater.argoproj.io/image-list: app=mydomain.com/{{client.backendProject}}-node
        argocd-image-updater.argoproj.io/app.force-update: "true"
        argocd-image-updater.argoproj.io/write-back-method: "git"
        argocd-image-updater.argoproj.io/git-branch: "git-source:git-destination"
        argocd-image-updater.argoproj.io/app.ignore-tags: "latest"
      labels:
        name: test-app-label
    spec:
      project: my-project
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: "namespace"

[iuculano-nh]

@yevon

I just ran into this issue and removing the project from the repository secret seemed to work. I didn't need to remove project from my ApplicationSet (and you seemingly can't anyway, it's invalid).

As soon as I did that, my ApplicationSet deployed.

This is quite a confusing error at first, because an Application will have no issue with Git, but the Git generator for an ApplicationSet mysteriously complains it can't find credentials. The logs seem useless here and don't really provide any hint.

[hamadodene]

I have the same issue with argocd 2.12.4 ...any update for this? Rolling back to 2.11.7 working for me

[mrnim94]

When I upgrade argocd to 2.12.4. It can't list private helm charts on bitbucket
time="2024-10-17T08:01:46Z" level=error msg="finished unary call with code Unknown" error="failed to get index: 404 Not Found" grpc.code=Unknown grpc.method=GetHelmCharts

[kpatelpanw]

Hi @blakepettersson , thank you for all your valuable inputs in this thread. We are facing similar error when using applicationset along with git generator on "argo v2.12.3+6b9cd82" .

We have connected the same repo in default appProject and other appProject (e.g dev or prod for other applications ) , now seeing below issue, so do we need connect the same repo without specifying anything in Project field to use applicationsets with the Git generator.

error generating params from git: error retrieving Git files: rpc error: code = Internal desc = unable to checkout git repo https://<internal_giltab_url>/dlp/dss-infrastructure/dss-argocd.git with revision b0e25833...5c476 pattern apps/dss-url-prefilter-service/envs/dev/ap-*/kustomization.yaml: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: git fetch origin --tags --force --prune failed exit status 128: fatal: could not read Username for 'https://<internal-gitlab-url>/': terminal prompts disabled

We have added https://<internal_giltab_url>/dlp/dss-infrastructure/dss-argocd.git repo in default and dev appProject. Both secret has project.

k get secret repo-2902974994 -o yaml -n dlp-argocd

apiVersion: v1
data:
  password: ...abcM=
  project: cHJvZA==     (--> decoded value: prod )
  type: Z2l0
  url: aHR0cHM6Ly9jb2RlLnBhbi5ydW4vZGxwL2Rzcy1pbmZyYXN0cnVjdHVyZS9kc3MtYXJnb2NkLmdpdA== 
           (--> decoded : https://<internal-gitlab-url>/dlp/dss-infrastructure/dss-argocd.git )
  username: YXJnbw==
kind: Secret
metadata:
  annotations:
    managed-by: argocd.argoproj.io
  creationTimestamp: "2024-10-23T09:13:30Z"
  labels:
    argocd.argoproj.io/secret-type: repository
  name: repo-2902974994
  namespace: dlp-argocd
  resourceVersion: "131584229"
  uid: 02087221-2d11-4acb-8183-070280c05730
type: Opaque

k get secret repo-1152428332 -o yaml -n dlp-argocd

apiVersion: v1
data:
  password: ...abcM=
  project: ZGVmYXVsdA==           (--> decoded value: default )
  type: Z2l0
  url: aHR0cHM6Ly9jb2RlLnBhbi5ydW4vZGxwL2Rzcy1pbmZyYXN0cnVjdHVyZS9kc3MtYXJnb2NkLmdpdA==
             (--> decoded : https://<internal-gitlab-url>/dlp/dss-infrastructure/dss-argocd.git )
  username: YXJnbw==
kind: Secret
metadata:
  annotations:
    managed-by: argocd.argoproj.io
  creationTimestamp: "2024-10-23T09:12:32Z"
  labels:
    argocd.argoproj.io/secret-type: repository
  name: repo-1152428332
  namespace: dlp-argocd
  resourceVersion: "131752057"
  uid: 3f86a3df-9d82-4827-985f-2f31a6088e67
type: Opaque

ApplicationSet

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
  creationTimestamp: '2024-10-23T13:16:47Z'
  generation: 1
  labels:
    argocd.argoproj.io/instance: dev
  name: dev-dss-url-prefilter-service
  namespace: dlp-argocd
  resourceVersion: '132179473'
  uid: 1ddf82ba-6a06-4aff-88a9-eb8bb2e06002
spec:
  generators:
    - git:
        files:
          - path: apps/dss-url-prefilter-service/envs/dev/ap-*/kustomization.yaml
          - path: apps/dss-url-prefilter-service/envs/dev/dss-*/kustomization.yaml
          - path: >-
              apps/dss-url-prefilter-service/envs/dev/dss-eks-shared-dev/dss*/kustomization.yaml
        repoURL: https://<internal-gitlab-url>/dlp/dss-infrastructure/dss-argocd.git
        revision: main
        values:
          app: dss-url-prefilter-service
          cluster: '{{ index .path.segments 4 }}'
          namespace: >-
            {{ if ge (len .path.segments) 6 }}{{ index .path.segments 5 }}{{
            else }}dss{{ end }}
          namespace_suffix: >-
            {{ if ge (len .path.segments) 6 }}-{{ index .path.segments 5 }}{{
            end }}
  goTemplate: true
  goTemplateOptions:
    - missingkey=error
  template:
    metadata:
      labels:
        app: '{{ .values.app }}'
      name: '{{ .values.cluster }}-{{ .values.app }}{{ .values.namespace_suffix }}'
    spec:
      destination:
        name: '{{ .values.cluster }}'
        namespace: '{{ .values.namespace }}'
      project: dev
      source:
        path: '{{ .path.path }}'
        repoURL: https://<internal-gitlab-url>/dlp/dss-infrastructure/dss-argocd.git
        targetRevision: main
status:
  conditions:
    - lastTransitionTime: '2024-10-23T21:25:59Z'
      message: >-
        error generating params from git: error retrieving Git files: rpc error:
        code = Internal desc = unable to checkout git repo
        https://<internal-gitlab-url>/dlp/dss-infrastructure/dss-argocd.git with revision
        ba0cb037348141e97c97085e88e051892e7d3d92 pattern
        apps/dss-url-prefilter-service/envs/dev/ap-*/kustomization.yaml: failed
        to initialize repository resources: rpc error: code = Internal desc =
        Failed to fetch default: `git fetch origin --tags --force --prune`
        failed exit status 128: fatal: could not read Username for
        'https://<internal-gitlab-url>': terminal prompts disabled
      reason: ApplicationGenerationFromParamsError
      status: 'True'
      type: ErrorOccurred

[tekno45]

creds made before 2.12 were made in the UI and just broke git generators.

I manually removed the project field from the repo secrets in the cluster and everything works.

[andrii-korotkov-verkada]

Is this resolved?

[venkat-enable]

Is this resolved ?

[yevon]

I had to delete the project from my cluster and repo secrets because if I dind't do it it won't work. I think I have all my applications and applicationsets with the same project set, but even with that it wasn't working. What should I check if I want to restore the project to my secrets?

[schlags]

I'd like to add that the path forward for preparing for this breaking change is also not ideal, and the upgrading docs are unclear that Cluster secret is referring to the secrets used to register repositories (but maybe that's just a personal problem ha).

Let's say that I have a repository secret with project: projectA defined. Projects projectB and projectC might have been using this repo secret until this point; but instead of removing the project field - I'd rather create 2 more repo secrets for that same repository scoped to projectB and another scoped to projectC.

It seems that in 2.11, this isn't possible. I get a conflict saying that it needs to be a forced operation to overwrite the existing repo secret since there are conflicts on the project field.

I'm assuming that if I upgraded to 2.12, I'd be able to add more secrets for the same repository (scoped to individually separate projects)?

If anything, and based on the issues present for this today, could we get some expanded language in the upgrading documentation that you will need to edit the secret and remove the project field since that is not exposed in the edit capabilities of the UI, and it is also impossible to add the same repo URL for a different project in a new secret?

[vertrost]

I've solved the problem by recreating secrets (I left the projects in place).

[yevon]

I found a new one with this:

1. I did not have set any project in my HTTP repository secret (Due to 2.12.x changes, because I did have set that in the past, and I had to delete it in order to work, even when it says that it should work if you have a matching project between the app and the repo secret)
2. My applications did use my git HTTP URL as RepoURL:
3. I changed my repository to be accessed by SSH, in the form of git@mygitlab:user/repo.git, and connected successfuly
4. Changed my applications to use git@mygitlab:user/repo.git. It complained that it could not be used for this project, even when the repository did not have any project set to it!
5. I had to manually specify the project in my repository secret again in order to work.

For me what I could reproduce still in latest v2.13.0+347f221 about this is:

• All my applications with a project A set
• If you use HTTP git, you must delete your project A from the repository secret, if you set it it doesn't work. (Must be allowed as they have a matching project between the app and the secret)
• If you use SSH git, you must add the project A in the repository secret, if you don't set it it doesn't work and the UI complains about it. (Must be allowed, in order to share the secret between multiple projects)

The behaviour is completely opposite depending if you are acessing your repository through HTTP or SSH(git@domain...), that's the reason about the confusion all together, it doesn't follow the behaviour described in the upgrade notes of 2.12.x.

[hamadodene]

I've solved the problem by recreating secrets (I left the projects in place).

@vertrost If I disconnect the repositories and reconnect them, will the applications be affected?

[yevon]

I had to replace sourceRepos to * in my AppProject definition, because if I did place a git@....git repository, it didn't recognize it and it says that I'm not allowed to pull changes from that git repo:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: project-name
namespace: argocd
sourceRepos:
- '*'

The behaviour of project matching with secrets is not working properly, it differs when you use http git repositories, or ssh git repositories.

[kamilpi]

We have faced with the same issue.

The main reason for this issue is here #18388

In our case, it was related to ApplicationSet, there is also a note on the official documentation:
https://argo-cd.readthedocs.io/en/latest/operator-manual/applicationset/Generators-Git/#repository-credentials-for-applicationsets

We have changed the project field in the repository secret to name and had to replace this Secret and ApplicationSet from Argo UI.