applicationset-controller update to latest version after 6.17, get fatal error: Could not read from remote repository

[ghost068]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
I use argocd:v2.12.0+fe965ed at production. At 2024.6.17, applicationset-controller pod has rebuilded, then I get a fatal error: Could not read from remote repository. see log details below.

Then I checked my ssh private key and git repo server status, it worked as well. Argocd can do refreshing and syncing operation as well, only appset can not work.
I change git repository credentials from ssh to https, and error log in applicationset-controller continues.
Finally, I rollback applicationset-controller to the version two weeks ago(v2.11.3) and error log ends.
So I think the lastest version after 6.17 maybe has some bugs in applicationset-controller. I am not fimiliar with go, and cannot findout the exact point.
https://quay.io/repository/argoproj/argocd?tab=tags&tag=latest

To Reproduce

Expected behavior
I hope the developers can review the code, and fix it

Screenshots

Version
argocd: v2.9.3+6eba5be.dirty
BuildDate: 2023-12-02T00:37:12Z
GitCommit: 6eba5be
GitTreeState: dirty
GoVersion: go1.21.4
Compiler: gc
Platform: darwin/arm64
argocd-server: v2.12.0+fe965ed

Logs
time="2024-06-20T02:29:19Z" level=error msg="error generating params" error="error generating params from git: error retrieving Git files: rpc error: code = Internal desc = unable to checkout git repo git@xxx.com:gitops.git with revision 92af11f9752df68b9b6dea5bf893c95fd781f166 pattern /*/config.json: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: git fetch origin --tags --force --prune failed exit status 128: git@xx.com: Permission denied (publickey,keyboard-interactive).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="&{0xc000b8c5c0}"

time="2024-06-20T02:29:19Z" level=error msg="error generating application from params" applicationset=argocd/appset-dev error="error generating params from git: error retrieving Git files: rpc error: code = Internal desc = unable to checkout git repo git@xx.com:common/gitops.git with revision 92af11f9752df68b9b6dea5bf893c95fd781f166 pattern /*/config-dev.json: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: git fetch origin --tags --force --prunefailed exit status 128: git@xx.com: Permission denied (publickey,keyboard-interactive).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="{nil nil &GitGenerator{RepoURL:git@xx.com:common/gitops.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:nadp/lbs/*/config-dev.json,},},Revision:HEAD,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"

[ilbarone87]

Got same error today after updating from 2.11.7 to 2.12.0, rollback fixes the error

"time=\"2024-08-05T19:23:56Z\" level=error msg=\"error generating application from params\" applicationset=argocd/infra-apps error=\"error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo https://github.com/xxxxx/deployments.git with revision 27c35f4632d6a63f5f956399f3bf08724091ad7b: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: git fetch origin --tags --force --prune failed exit status 128: fatal: could not read Username for 'https://github.com': terminal prompts disabled\" generator=\"{nil nil &GitGenerator{RepoURL:https://github.com/xxxxxx/deployments.git,Directories:[]GitDirectoryGeneratorItem{GitDirectoryGeneratorItem{Path:infra/*,Exclude:false,},},Files:[]GitFileGeneratorItem{},Revision:HEAD,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}\""

[O5ten]

☝🏻 I get the exact same error as @ilbarone87.

Rollback to 2.11.7 fixed the error. It's still prevalent in 2.12.0 that was released yesterday. We're relying on git parameters quite heavily and are using github apps as the authentication mechanism with a mix of kustomize repositories and helm charts. It only seems to affect our gitparameter appsets and none of the others.

finished unary call with code Internal" error="rpc error: code = Internal desc = unable to checkout git repo https://github.com/org/reponame-argocd with revision e72b5cdc27d48e7d2db577f0d23f0856af8dd997 pattern */apps/cluster-config/values.yaml: 
failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: 
fatal: could not read Username for 'https://github.com': terminal prompts disabled" grpc.code=Internal grpc.method=GetGitFiles grpc.service=repository.RepoServerService grpc.start_time="2024-08-06T10:59:48Z" grpc.time_ms=334.837 span.kind=server system=grpc

[whirlthesquirrel]

I'm also seeing this on 2.12.0 where a rollback fixes the issue. With the same Git repository, the application works while the application set does not.

[crenshaw-dev]

OP was on 2.9, so I'm guessing we're seeing something new with similar symptoms to the old issue.

[rtomadpg]

Similar error here. All worked on v2.11.1, but failed once upgraded to v2.12.0:

applicationset-controller time="2024-08-16T15:05:17Z" level=error msg="error generating params" error="error generating params from git: error retrieving Git files: rpc error: code = Internal desc = unable to checkout git repo git@github.com:my/repo.git with revision fd0...ad4 pattern applications/*/cluster-a/argocd.yaml: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: git fetch origin --tags --force --prune failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="&{0xc0005ae580}"

I am using an OpenSSH formatted Ed25519 key to auth to a Github repo. The upgrade did not change the Secret containing the repository configuration and key.

I reverted back to v2.11.1 and no more errors, so there does seem to be some regression in v2.12.0.

--

I did some investigation and found this:

• it only impacts the ApplicationSet controller (same credential is successfully used for Applications)
• using "Repository -> Refresh list" on v2.12.0 does not fail any repositories (confirming it's not a Repo issue)

[McHammar]

We have the same issue. Works again after rollback.

[crenshaw-dev]

For the failing repo, was the project field set in the credential Secret?

[McHammar]

For the failing repo, was the project field set in the credential Secret?

Yes it is set to "default"

[O5ten]

We have project set on all of our repository-secrets, we have some repositories that are commonly used as parents in our kustomize-hierarchy. A (Scoped team-X) extends kustomize target B (where B is scoped to team platform).

[rtomadpg]

Same here, set to default.

Our repo Secrets have the following keys:

• name
• project: default
• sshPrivateKey, formatted like:

-----BEGIN OPENSSH PRIVATE KEY-----
....
-----END OPENSSH PRIVATE KEY-----

• type: git
• url, like git@github.com:my/repo.git

[blakepettersson]

I suspect it's the same thing actually :( An applicationset doesn't have a project; it's admin-scoped. I believe the resolution is exactly the same as for #19587 - having a repo-cred that doesn't have a project set.

[rtomadpg]

I can confirm the issue on 2.12.x is gone when the repository credential used by the ApplicationSet does NOT have a project key set. Thanks @blakepettersson !

[McHammar]

Is it okay to remove it before upgrading or will it break something?

[blakepettersson]

Is it okay to remove it before upgrading or will it break something?

It's fine to remove the project even with < 2.12.

[McHammar]

I have tried now and it works after removing project and upgrading to v2.12.x

[SergeyLadutko]

me too

[reggie-k]

Looks like a dup of #19585