Parameter overrides not honored in multi-source helm/git setup

[sgrissom-rxss]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Parameter overrides do not appear to work when using multi-source setup with a Helm repo as one source and a git repo as the second source for value files.

I know multi-source is beta so this may just be a corner case not covered yet?

In the UI, the Application manifest contains the parameter override. Snippet:

project: default
source:
  repoURL: registry-1.docker.io/bitnamicharts
  targetRevision: 15.0.2
  helm:
    valueFiles:
      - $values/path/to/my/values.yaml
    parameters:
      - name: replicaCount
        value: '3'
  chart: nginx
destination:
  server: 'https://kubernetes.default.svc'
  namespace: default
...

To Reproduce

• Create an Application with two sources:
  • Source 1 is a Helm chart. By default it has a replicaCount=1.
  • Source 2 is a git repo with values.yaml containing replicaCount=2.
• Issue command argocd app set <project> -p replicaCount=3
• Verify Application manifest contains parameter override for helm.
• Note that parameter override is not reflected in number of deployed pods.

Expected behavior

Expected: The deployed app has 3 replicas.
Actual: The deployed app has 2 replicas.

Version

argocd: v2.7.6+00c914a                                                                                                                                          │Handling connection for 8080
  BuildDate: 2023-06-20T21:18:20Z                                                                                                                               │Handling connection for 8080
  GitCommit: 00c914a948d9e8ad99be8bd82a368fbdeba12f88                                                                                                           │Handling connection for 8080
  GitTreeState: clean                                                                                                                                           │Handling connection for 8080
  GoVersion: go1.19.10                                                                                                                                          │Handling connection for 8080
  Compiler: gc                                                                                                                                                  │Handling connection for 8080
  Platform: darwin/arm64                                                                                                                                        │Handling connection for 8080
argocd-server: v2.7.6+00c914a.dirty                                                                                                                             │Handling connection for 8080
  BuildDate: 2023-06-20T20:51:13Z                                                                                                                               │Handling connection for 8080
  GitCommit: 00c914a948d9e8ad99be8bd82a368fbdeba12f88                                                                                                           │Handling connection for 8080
  GitTreeState: dirty                                                                                                                                           │Handling connection for 8080
  GoVersion: go1.19.10                                                                                                                                          │Handling connection for 8080
  Compiler: gc                                                                                                                                                  │Handling connection for 8080
  Platform: linux/arm64                                                                                                                                         │Handling connection for 8080
  Kustomize Version: v5.0.1 2023-03-14T01:32:48Z                                                                                                                │Handling connection for 8080
  Helm Version: v3.11.2+g912ebc1                                                                                                                                │Handling connection for 8080
  Kubectl Version: v0.24.2                                                                                                                                      │Handling connection for 8080
  Jsonnet Version: v0.19.1

Logs

The only relevant thing I could find was in the repo-server, which seems to indicate the HelmParameter is not honored.

time="2023-06-30T22:09:54Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:registry-1.docker.io/bitnamicharts,Path:,TargetRevision:15.0.2,Helm:&ApplicationSourceHelm{ValueFiles:[$values/path/to/my/values.yaml],Parameters:[]HelmParameter{},ReleaseName:,Values:,FileParameters:[]HelmFileParameter{},Version:,PassCredentials:false,IgnoreMissingValueFiles:false,SkipCrds:false,},Kustomize:nil,Directory:nil,Plugin:nil,Chart:nginx,Ref:,}/15.0.2"

[keramblock]

You using source instead of sources? Also I do not see second repo with values

[sgrissom-rxss]

@keramblock Good eye. I inadvertently chopped off some import pieces of that yaml, which may point to the underlying issue.

After deploying the multi-source app, this is the manifest rendered in the UI appears like this:

project: default
destination:
  server: 'https://kubernetes.default.svc'
  namespace: default
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  retry:
    limit: 5
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m
sources:
  - repoURL: registry-1.docker.io/bitnamicharts
    targetRevision: 15.0.2
    helm:
      valueFiles:
        - $values/path/to/my/values.yaml
    chart: nginx
  - repoURL: 'https://git.example.com/my/project.git'
    targetRevision: poc
    ref: values

Through the command line, a parameter is added to the app:

argocd app set argocd/my-app -p replicaCount=3

The resulting manifest in the UI:

project: default
source:
  repoURL: registry-1.docker.io/bitnamicharts
  targetRevision: 15.0.2
  helm:
    valueFiles:
      - $values/path/to/my/values.yaml
    parameters:
      - name: replicaCount
        value: '3'
  chart: nginx
destination:
  server: 'https://kubernetes.default.svc'
  namespace: default
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  retry:
    limit: 5
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m
sources:
  - repoURL: registry-1.docker.io/bitnamicharts
    targetRevision: 15.0.2
    helm:
      valueFiles:
        - $values/path/to/my/values.yaml
    chart: nginx
  - repoURL: 'https://git.example.com/my/project.git'
    targetRevision: poc
    ref: values

The result of adding a parameter to the application via the command line argocd is the generation of a source field utilizing values from existing sources[0].

[crenshaw-dev]

Looks like the CLI never persisted the parameter to the Application resource. In general, the UI/CLI support of multi-source apps is incomplete. In this case, I would have expected the CLI to throw an error saying that it could not set the parameter on a multi-source app. I think this is a CLI bug, and the lack of full CLI support is a secondary issue.

As a workaround, you can manually edit the Application resource to add the desired override.

[sgrissom-rxss]

@crenshaw-dev Thanks for looking.

In general, the UI/CLI support of multi-source apps is incomplete.

I figured as much given the beta warnings on the feature! I realized a bit late that the command is currently ambiguous to the target source. Appears we need to teach the CLI a new trick to indicate the target source for some of these commands (at least set).

[naimo84]

As a workaround, you can manually edit the Application resource to add the desired override.

@crenshaw-dev We're using the app of apps concept und we've tried the workaround today, but the overrides are disappearing after syncing the parent app. As you can hopefully see in the gif.

What are we doing wrong? 😁

Version

argocd: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.8.4+c279299
  BuildDate: 2023-09-13T19:12:09Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.1.0 2023-06-19T16:58:18Z
  Helm Version: v3.12.1+gf32a527
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.20.0

[naimo84]

@crenshaw-dev Hey Michael, might this be fixed with the upcoming 2.9? 😁 The workaround definitely does not work 😕

[alexelent-carta]

I think #14743 might solve this.

[crenshaw-dev]

@naimo84 since you're using an app-of-apps, you need to modify the parameter in the parent app's deployed manifests so that they're not overridden when you sync. Basically, go full-gitops instead of doing overrides via the UI/CLI.

[naimo84]

Thanks a lot for your answer 👍
we have 4 cluster, each with it'a own argocd and roundabout 1000 argocd apps) for customer test and production cluster we're going full-gitops of course 😉 There, multisources are working perfectly 👍

But we also have a Dev and internal Test cluster. In order to not commit every single change, like an app get's many times an update per day with a ...-SNAPSHOT-... version, we're using the 'argocd app set ...' to set the "image.tag" of the child apps since the first day of our argocd journey. This is also working perfectly until we're using multisources.

We (@RalphFeichtner an me) will try to change your dev cicd pipeline to change the parent app's manifest. But that might not be easy 😉

[ishitasequeira]

PR #17425 added support to set unset commands for multiple sources.