Allow advanced templating in ApplicationSet

[crenshaw-dev]

People want to be able to do advanced things in ApplicationSet templates.

One way to accomplish this is to provide a plain ol' string field for the template instead of an Application spec. But then you lose nice IDE support.

Another way, which I think would require fewer changes, is to allow templating keys instead of just values in the Application spec.

Something like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
  goTemplate: true
  generators:
  - list:
      elements:
      - automated: "true"
      - automated: "false"
  template:
    spec:
      syncPolicy:
        "{{ ternary "automated" "turtles" (eq .automated "true") }}": {}

When automated is not "true", the key nested under syncPolicy evaluates to turtles. Since that's not part of the Application spec, it'll be dropped when the Application is unmarshaled.

Similar strategies could be applied to toggle, for example, fields under source. That would allow a git files generator to generate applications both for Helm and Kustomize.

Changes required:

First, we'd have to change the ApplicationSet CRD to preserve unknown fields. "{{ ternary "automated" "turtles" (eq .automated "true") }}" would otherwise be rejected as an invalid key (or, if validation is disabled, dropped).

Second, we'd have to change the ApplicationSet controller to handle the ApplicationSpec as an unstructured until the very last moment, when all templating has been applied. If we unmarshal to an Application spec before then, we'll lose the templated key.

Inspiration for this strange idea comes from fixing this bug, when I realized we can template keys: #11163

[crenshaw-dev]

cc @speedfl because I think you'll find this entertaining. :-)

[speedfl]

Hi @crenshaw-dev

Thank you for the notification.
Not sure I will have time to work on this one :-(

But definitely a great idea.

[cjc7373]

I have read the use cases listed in #12843, I was wondering if these use cases can be achieved in another way, like a JSON6902 patch in kustomization. In this way, we would achieve the same thing by having a new generator called Patch Generator, and it works in a way like the merge generator.

To illustrate that, let's see the use cases in #12843
Use case 1:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
spec:
  goTemplate: true
  generators:
  - patch:
      matchKey: cluster  # use this key to match an item in list generator
      baseGenerator:
        list:
          elements:
          - cluster: engineering-dev
            url: https://kubernetes.default.svc
          - cluster: engineering-prod
            url: https://kubernetes.default.svc
      patch:
      - key: engineering-dev
        action:
        - op: add
          path: /spec/source
          value: |
            helm:
              parameters:
                - some-param 
      - key: engineering-prod
        action:
        - op: add
          path: /spec/source
          value: |
            kustomize:
              images:
                - some=image
  template:
    metadata:
      name: '{{.cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: "{{.sourceType}}"
      destination:
        server: '{{.url}}'
        namespace: guestbook

Use case 2 is just like the use case 1.

In use case 3:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
spec:
  goTemplate: true
  generators:
  - patch:
      matchKey: cluster
      baseGenerator:
        list:
          elements:
          # ...
      patch:
      - key: engineering-dev
        action:
        - op: replace
          path: /spec/syncPolicy/automated/prune
          value: true
  template:
    metadata:
      name: '{{.cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: "{{.sourceType}}"
      destination:
        server: '{{.url}}'
        namespace: guestbook
      syncPolicy:
        automated:
          prune: false

What do you think of this idea? Does it look possible?

[speedfl]

This is a really good idea. Could we put patch outside generators ?

It would allow to first generate all applications, then apply patches.

Like:

spec:
  generators:
    - list:
  patch:
    -  key: engineering-dev
       action: ...

@crenshaw-dev this would also solve the issue of the Application template in the crd for ide integration

My only concern is the mix of templating for string and patch for the rest

[jwenz723]

It would be nice if the ApplicationSet template could be templated in a way similar to what helm allows. For example the example in the initial post in this issue could look like:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
  goTemplate: true
  generators:
  - list:
      elements:
      - automated: "true"
      - automated: "false"
  template:
    spec:
      syncPolicy:
        {{ toYaml .automated }}

It would be nice to allow more advanced features like control statements like an if or range.

Allowing this advanced templating might not be easily done with the current ApplicationSet spec. It might be better to add a separate attribute like spec.templatedTemplate that allows this. This way the entire spec.templatedTemplate object could be templated. I would probably prefer the key name to be spec.goTemplate, but that is already being used. I think this field would have to be a string type.

For example:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
  goTemplate: true
  generators:
  - list:
      elements:
      - syncPolicy:
          automated: "true"
      - helmValues:
          one: "1"
          two: "2"
  templatedTemplate: |
    spec:
      {{- if .helmValues }}
      helm:
        values: |
          {{- range $key, $value := .helmValues }}
          {{ $key }}: {{ $value }}
          {{- end }}
      {{- end }}
      {{- if .syncPolicy }}
        {{ toYaml .syncPolicy | nindent 6}}
      {{- end }}

[cjc7373]

@jwenz723 IMO both #11567 and #11183 are trying to do what you have proposed. The differences may be subtle. IIUC the former is less flexible and will not accept invalid YAML. The latter is more like what helm is doing.

[DanielYWoo]

cc @speedfl because I think you'll find this entertaining. :-)

Let me entertain you a bit more, I need this indeed in 50+ environments. They are all similar but with just very few differences in the helm repos and paths.

source:
  repoURL: 'https://github.com/test/{{- ternary "dev" "prod" (or (eq .env "dev") (eq .env "qa") (eq .env "staging")) -}}'
  path: '{{- ternary "./" (printf "%s/%s" .region .env) (eq .env "dev") -}}'

[DanielYWoo]

I prefer helm-styled template than kustomization patches. Template is short and expressive, especially ternary. I guess we don't need if/else like the last example from @jwenz723, that's error-prone for spaces and indent. If we really need many if/else, we should use app-of-apps, create 20 different apps instead of using application set. The whole purpose of it is to consolidate all the similar apps, right? If there are many if/else that means they are NOT similar.

[jwenz723]

if/else is not the only thing I would hope is made available, it was just an example. My main goal is that multi-field templates are made available. It appears that #11183 will accomplish this.

[speedfl]

#14893 supports it 🙂

[jetersen]

Here is a use case we came up with after wanting to use scm provider generator together with cluster generator:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: scm-k8s-app
spec:
  goTemplate: true
  generators:
    - matrix:
        generators:
          - scmProvider:
              cloneProtocol: https
              github:
                organization: moviestarplanet
                allBranches: false
                appSecretName: argocd-github-app-credentials
              filters:
                - labelMatch: k8s-app
          - clusters:
              selector:
                matchLabels:
                  k8s-app: 'true'
                matchExpressions:
                  - key: kubernetes.io/environment
                    operator: In
                    values:
                        - '{{ .labels | splitList "," | has "staging" | ternary "staging" "nothing" }}'
                        - '{{ .labels | splitList "," | has "eu" | ternary "eu" "nothing" }}'
                        - '{{ .labels | splitList "," | has "us" | ternary "us" "nothing" }}'
  template:
    metadata:
      name: '{{ .repository | replace "." "-" | lower }}-{{ .name }}'
      annotations:
        argocd.argoproj.io/manifest-generate-paths: .
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      source:
        repoURL: '{{ .url }}'
        targetRevision: '{{ .branch }}'
        path: .k8s/{{ .name }}
      project: default
      destination:
        server: '{{ .server }}'
        namespace: '{{ .repository | replace "." "-" | lower }}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true

This allows us to via GitHub topics manage our deployments to various clusters.

Although this also gets around having to check if a path exists 😖
We haven't gotten to templating out the sync policy yet 😅