Upgrading to 2.12 breaks ability to use .namespace in ApplicationSet List Generator

[rknightion]

Checklist:

Describe the bug

When deploying an applicationset similar to:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name:my-appset
  namespace: argocd
spec:
  syncPolicy:
    preserveResourcesOnDeletion: false
  generators:
    - matrix:
        generators:
          - list:
              elements:
                - cluster: prod
                  namespace: prod
                - cluster: prod
                  namespace: dev
                - cluster: dev
                  namespace: dev
          - matrix:
              generators:
                - git:
                    repoURL: https://gitlab......./.git
                    revision: main
                    directories:
                      - path: helm/environments/{{.namespace}}
                    pathParamPrefix: namespace
                - git:
                    repoURL: https://gitlab......./.git
                    revision: main
                    directories:
                      - path: helm/apps/*
                    pathParamPrefix: app
  goTemplate: true
  template:
    metadata:
      name: '{{.namespace}}-{{.app.path.basenameNormalized}}'
    spec:
      destination:
        name: '{{.cluster}}'
        namespace: '{{.namespace}}'
      project: '{{.namespace}}'
 .....

We get errors in the logs:

level=error msg="error generating params" error="error getting project : AppProject.argoproj.io \"{{.namespace}}\" not found" generator="&{0x4000b24780}"

and

level=error msg="error generating application from params" applicationset=argocd/exp-prod error="failed to get params for second generator in the matrix generator: child generator returned an error on parameter generation: error getting project : AppProject.argoproj.io \"{{.namespace}}\" not found" generator="{nil nil nil nil nil nil &MatrixGenerator{Generators:[]ApplicationSetNestedGenerator{ApplicationSetNestedGenerator{List:&ListGenerator{Elements:[]JSON{{[123 34 99 108 117 115 116 101 114 34 58 34 112 117 108 115 101 112 108 97 116 102 111 114 109 45 112 114 111 100 34 44 34 110 97 109 101 115 112 97 99 101 34 58 34 101 120 112 101 100 105 97 45 112 114 111 100 117 99 116 105 111 110 34 125]},{[123 34 99 108 117 115 116 101 114 34 58 34 112 117 108 115 101 112 108 97 116 102 111 114 109 45 112 114 111 100 34 44 34 110 97 109 101 115 112 97 99 101 34 58 34 99 119 103 45 112 114 111 100 117 99 116 105 111 110 34 125]},{[123 34 99 108 117 115 116 101 114 34 58 34 119 114 45 114 105 109 115 45 112 114 111 100 34 44 34 110 97 109 101 115 112 97 99 101 34 58 34 119 114 45 114 105 109 115 45 112 114 111 100 34 125]},{[123 34 99 108 117 115 116 101 114 34 58 34 103 108 111 98 97 108 45 112 108 97 116 102 111 114 109 45 112 114 111 100 34 44 34 110 97 109 101 115 112 97 99 101 34 58 34 103 108 111 98 97 108 45 112 108 97 116 102 111 114 109 45 112 114 111 100 34 125]},{[123 34 99 108 117 115 116 101 114 34 58 34 112 117 108 115 101 112 108 97 116 102 111 114 109 45 112 114 111 100 34 44 34 110 97 109 101 115 112 97 99 101 34 58 34 112 117 108 115 101 112 108 97 116 102 111 114 109 45 112 114 111 100 117 99 116 105 111 110 34 125]},},Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},ElementsYaml:,},Clusters:nil,Git:nil,SCMProvider:nil,ClusterDecisionResource:nil,PullRequest:nil,Matrix:nil,Merge:nil,Selector:nil,Plugin:nil,},ApplicationSetNestedGenerator{List:nil,Clusters:nil,Git:&GitGenerator{RepoURL:https://gitlab.devops.pulselive.io/motor-team/pdp-experience-platform.git,Directories:[]GitDirectoryGeneratorItem{GitDirectoryGeneratorItem{Path:{{.cluster}}/{{.namespace}}/*/*,Exclude:false,},},Files:[]GitFileGeneratorItem{},Revision:prod,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},},SCMProvider:nil,ClusterDecisionResource:nil,PullRequest:nil,Matrix:nil,Merge:nil,Selector:nil,Plugin:nil,},},Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},} nil nil nil}"

level=error msg="error generating params" error="failed to get params for second generator in the matrix generator: child generator returned an error on parameter generation: error getting project : AppProject.argoproj.io \"{{.namespace}}\" not found" generator="&{map[ClusterDecisionResource:0x40009cd720 Clusters:0x40009cd6d0 Git:0x4000de5b20 List:0x74426e0 Matrix:0x4000c22a50 Merge:0x4000c22a58 Plugin:0x4000b247c0 PullRequest:0x4000dcd860 SCMProvider:0x4000dcd800]}"

This worked fine for us in v2.11.7.

To Reproduce
Use a similar applicationset to the above
Expected behavior

The applicationset provisions applications against a project based on the {{.namespace}} value generated by the list generator

Version
2.12.0

[crenshaw-dev]

I think I heard someone else say they had trouble with parameters applying in nested generators. Can you create a unit test to reproduce the issue? That would be a good first step in figuring out what broke the templating.

[VonNao]

I got similar problem which may be related. I get no error in the logs but our applicationset dosent seem to pick up new deployment. Like @rknightion 2.11.7 works without problems

[allevimi]

it is related to #18906, problem is spec.template.spec.project cannot be templated

[crenshaw-dev]

@allevimi does hard-coding the project field solve the error?

[allevimi]

didn't test it, but I think that hardcoding a valid one, it will works

	var project string
	if strings.Contains(appSet.Spec.Template.Spec.Project, "{{") {
		project = appSetGenerator.Git.Template.Spec.Project
	} else {
		project = appSet.Spec.Template.Spec.Project
	}

	appProject := &argoprojiov1alpha1.AppProject{}
	if err := client.Get(context.TODO(), types.NamespacedName{Name: appSet.Spec.Template.Spec.Project, Namespace: appSet.Namespace}, appProject); err != nil {
		return nil, fmt.Errorf("error getting project %s: %w", project, err)
	}

I'm not a GO guru, but it seems that only checks if appSet.Spec.Template.Spec.Project is an existing project

[crenshaw-dev]

Ah 100%. @ishitasequeira is there any way we can rework the signature verification logic to allow templating the project field? Maybe we can just document a limitation like "if you need signature verification, you can't template the project field"?

[crenshaw-dev]

In the meantime if anyone has time to write up a PR making that change so that we can evaluate our options, I'd appreciate it!

[MarcStorm]

We are experiencing the same error message when using git generator files. Adding here as opposed to create a new issue as we see the same error. Please let me know if you want it created in a separate issue.

Describe the bug

When deploying an applicationset similar to:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-appset
  namespace: canaria-prod
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: {{ .Values.repository.url | quote }}
        revision: {{ .Values.repository.revision | quote }}
        files:
          - path: '{{ .Values.path }}/*.yaml'
  template:
    metadata:
      name: '{{ "{{" }}.path.filename | trimSuffix ".yaml" {{ "}}" }}'
    spec:
      project: canaria-prod
      destination:
        name: in-cluster
        namespace: canaria-prod
......

ApplicationSet has been deployed with namespace and project as being hardcoded and templated. Both resulted in the same error as shown below.

We get errors in the logs:
time="2024-08-09T06:17:29Z" level=error msg="error generating params" error="error getting project canaria-prod: AppProject.argoproj.io \"canaria-prod\" not found" generator="&{0xc0001d2c40}" time="2024-08-09T06:17:29Z" level=error msg="error generating application from params" applicationset=canaria-prod/my-appset error="error getting project canaria-prod: AppProject.argoproj.io \"canaria-prod\" not found" generator="{nil nil &GitGenerator{RepoURL:git@github.com:....git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:charts/*.yaml,},},Revision:feat/add-applicationset,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"

Version
2.12.0

[amine7777]

I also bumped into this error while upgrading to 2.12.0, the solution so far is to hardcode the project name value

[allevimi]

@MarcStorm I think you have a different cause, because you have hardcoded your project value; it seems that canaria-prod doesn't exist as a Argocd AppProject.argoproj.io
What is the output of kubectl --namespace canaria-prod get AppProject.argoproj.io ?

[MarcStorm]

@allevimi - Note that I mentioned we see the same behaviour regardless of templating or hardcoding the project name.

Our AppProject are managed in a separate namespace, called core-argocd. I should emphasise that the same setup/configuration works with v. 2.11.3

kubectl --namespace canaria-prod get appprojects.argoproj.io 
No resources found in canaria-prod namespace.

kubectl get appprojects.argoproj.io -A
NAMESPACE     NAME                       AGE
core-argocd   canaria-prod   22d

[allevimi]

@MarcStorm my bad :)
so your problem is caused by this

	appProject := &argoprojiov1alpha1.AppProject{}
	if err := client.Get(context.TODO(), types.NamespacedName{Name: appSet.Spec.Template.Spec.Project, Namespace: appSet.Namespace}, appProject); err != nil {
		return nil, fmt.Errorf("error getting project %s: %w", project, err)
	}

[crenshaw-dev]

Yep, there are two bugs here.

1. New git signature verification code doesn't handle templated project fields (@ishitasequeira is looking into how to fix that)
2. The code also doesn't correctly handle appsets-in-any-namespace. It should use the controller namespace instead of appSet.Namespace to get the AppProject.

Would someone be willing to open a PR for the second bug? Should be a one-line fix (plus any tests needed).

[cilindrox]

@crenshaw-dev happy to take a stab at it - just to confirm, this is the function in question, correct?

argo-cd/applicationset/generators/git.go

Line 70 in 5ccea0d

if err := client.Get(context.TODO(), types.NamespacedName{Name: appSet.Spec.Template.Spec.Project, Namespace: appSet.Namespace}, appProject); err != nil {

appSet.Namespace -> controller namespace + tests?

[crenshaw-dev]

Yep!

[cilindrox]

It should use the controller namespace instead of appSet.Namespace to get the AppProject.

So @crenshaw-dev , after digging around for a bit, I'm thinking the NewGitGenerator function call needs an additional param, the namespace from the controller, which can be added on the GetGenerators call from the controller, similar to what's being done with NewClusterGenerator - is this what you had in mind?

[crenshaw-dev]

@cilindrox at a glance, that looks reasonable.

[crenshaw-dev]

Keeping open to track the appsets-in-any-namespace fix.

[ishitasequeira]

Yep, there are two bugs here.

1. New git signature verification code doesn't handle templated project fields (@ishitasequeira is looking into how to fix that)
2. The code also doesn't correctly handle appsets-in-any-namespace. It should use the controller namespace instead of appSet.Namespace to get the AppProject.

Would someone be willing to open a PR for the second bug? Should be a one-line fix (plus any tests needed).

@cilindrox are you working on the fix for the 2nd bug mentioned here? If not, I could take a look at it.

[ankitabhopatkar13]

We are also blocked in deploying new applications using the git generator in applicationSets. We use {{path[x]}} variables in project: property and they are no longer getting interpolated.

Thanks for the fix, is there an estimated time on when is it going to be released?

[ishitasequeira]

This should be fixed in v2.12.1.