Argocd-2.4.17 CMP plugin fails to "kustomize build ." on remote repo. Getting Error: accumulating resources due to git credentials

[kxs-sindrakumar]

Checklist:

• [ x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ x] I've included steps to reproduce the bug.
• [x ] I've pasted the output of argocd version.

Describe the bug

1. We upgraded argocd from 2.2.8 to 2.4.17 maintaining the same cmp (defined in configmap below).
2. Once we completed the upgrade, apps fail to sync generating an accumulating resources error (see logs below)

What I have tried

1. I copied the below application kustomization.yaml into my repo server and ran "kustomize build ." This resulted in me being prompted for the git username and password. If entered correctly, it works and i get a manifest back. If I just hit enter twice, then i get the following error:

Error: accumulating resources: accumulation err='accumulating resources from 'https://github.com/myorg/ce-cluster-addons/cert-manager/overlays/mytenant/myapp-qa?ref=v0.3.5': yaml: line 175: mapping values are not allowed in this context': git cmd = '/usr/bin/git fetch --depth=1 origin v0.3.5': exit status 128

2. Through the ongoing discussion on this thread 9443, I tried the gitaskpass route. Again copying the application kustomization.yaml to repo server and manually running "kustomize build ." works after adding gitaskpass.
   HOWEVER, automatically through argo repo server, it does not work and produces the same error code shown in the logs. below.

To Reproduce

Kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/argoproj/argo-cd/manifests/base?ref=v2.4.17&timeout=90
- github.com/argoproj/argo-cd/manifests/crds?ref=v2.4.17&timeout=90
- github.com/argoproj/argo-cd/manifests/cluster-rbac?ref=v2.4.17&timeout=90
- namespace.yaml
namespace: argo-cd
patches:
- patch: |-
    - op: replace
      path: "/subjects/0/namespace"
      value: argo-cd
  target:
    group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    version: v1
patchesStrategicMerge:
- patches/argocd-repo-server-deploy.yaml
- patches/argocd-cm.yaml
- patches/argocd-resource-limits.yaml

namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: argo-cd

patches/argocd-cm.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  configManagementPlugins: |
    - name: argocd-vault-plugin
      generate:
        command: ["argocd-vault-plugin"]
        args: ["generate", "./"]
    - name: argocd-kustomize
      generate:
        command: ["sh", "-c"]
        args: ["kustomize build ."]
      lockRepo: true

patches/argocd-repo-server-deploy.yaml:

# # containers:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
      - name: argocd-repo-server
        volumeMounts:
        - name: custom-tools
          mountPath: /usr/local/bin/argocd-vault-plugin
          subPath: argocd-vault-plugin
        # - name: custom-tools
        #   mountPath: /usr/local/bin/kustomize
        #   subPath: kustomize
        # Note: AVP config (for the secret manager, etc) can be passed in several ways. This is just one example
        # https://argocd-vault-plugin.readthedocs.io/en/stable/config/
        envFrom:
          - secretRef:
              name: argocd-vault-plugin-credentials
      volumes:
      - name: custom-tools
        emptyDir: {}
      initContainers:
      - name: download-tools
        image: alpine:3.8
        command: [sh, -c]

        # Don't forget to update this to whatever the stable release version is
        # Note the lack of the `v` prefix unlike the git tag
        env:
          - name: AVP_VERSION
            value: "1.7.0"
        args:
          - >-
            wget -O argocd-vault-plugin
            https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools

      # Not strictly necessary, but required for passing AVP configuration from a secret and for using Kubernetes auth to Hashicorp Vault
      automountServiceAccountToken: true

patches/argocd-resource-limites.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
      - name: argocd-repo-server
        resources:
          requests:
            memory: "100Mi"
            cpu: "50m"
          limits:
            memory: "200Mi"
            cpu: "100m"
      automountServiceAccountToken: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-redis
spec:
  template:
    spec:
      containers:
      - name: redis
        resources:
          requests:
            memory: "30Mi"
            cpu: "4m"
          limits:
            memory: "60Mi"
            cpu: "8m"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-notifications-controller
spec:
  template:
    spec:
      containers:
      - name: argocd-notifications-controller
        resources:
          requests:
            memory: "70Mi"
            cpu: "2m"
          limits:
            memory: "90Mi"
            cpu: "6m"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-dex-server
spec:
  template:
    spec:
      containers:
      - name: dex
        resources:
          requests:
            memory: "30Mi"
            cpu: "3m"
          limits:
            memory: "60Mi"
            cpu: "6m"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: argocd-application-controller
spec:
  template:
    spec:
      containers:
      - name: argocd-application-controller
        resources:
          requests:
            memory: "270Mi"
            cpu: "50m"
          limits:
            memory: "540Mi"
            cpu: "150m"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-applicationset-controller
spec:
  template:
    spec:
      containers:
      - name: argocd-applicationset-controller
        resources:
          requests:
            memory: "65Mi"
            cpu: "10m"
          limits:
            memory: "130Mi"
            cpu: "30m"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
spec:
  template:
    spec:
      containers:
      - name: argocd-server
        resources:
          requests:
            memory: "100Mi"
            cpu: "50m"
          limits:
            memory: "300Mi"
            cpu: "100m"

The application kustomization.yaml:
Note: this repo doesn't exist, i changed some of the names to protect identity.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- https://github.com/myorg/ce-cluster-addons/cert-manager/overlays/mytenant/myapp-qa?ref=v0.3.5

Expected behavior

As per version 2.2.8, I except the kustomize build to create the manifest and sync the application

Screenshots

N/A

Version

argocd: v2.4.17+d330967
  BuildDate: 2022-11-07T16:37:42Z
  GitCommit: d3309679605ddd1ef99651aef0090c7abfcd784f
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64

Logs

Note: url changed slightly to avoid showing confidential information.

argocd-repo-server time="2022-11-16T17:19:40Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `sh -c kustomize build .` failed exit status 1: Error: accumulating r │
│ esources: accumulation err='accumulating resources from 'https://github.com/myorg/ce-cluster-addons/mytenant/overlays/tenant/myapp-qa?ref=v0.3.5': yaml: line 175: mapping values are not allowed in this c │
│ ontext': git cmd = '/usr/bin/git fetch --depth=1 origin v0.3.5': exit status 128" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2022-11-16T17:19:39Z" grpc. │
│ time_ms=906.686 span.kind=server system=grp

[jessesuen]

yaml: line 175: mapping values are not allowed in this context

I think this may have more to do with change in kustomize version from argo-cd v2.2 to v2.4 rather than Argo CD bug. I suggest downgrading kustomize to the version in v2.2 to verify.

However, another problem is that you appear to be using the legacy URL format of resource references.
https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md#legacy-url-format

Newer kustomize resource references should look like:

• https://github.com/kubernetes-sigs/kustomize//examples/multibases/dev/?ref=v1.0.6
• git@github.com:owner/repo
• ssh://git@github.com/owner/repo

[kxs-sindrakumar]

yaml: line 175: mapping values are not allowed in this context

I think this may have more to do with change in kustomize version from argo-cd v2.2 to v2.4 rather than Argo CD bug. I suggest downgrading kustomize to the version in v2.2 to verify.

However, another problem is that you appear to be using the legacy URL format of resource references. https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md#legacy-url-format

Newer kustomize resource references should look like:

• https://github.com/kubernetes-sigs/kustomize//examples/multibases/dev/?ref=v1.0.6
• git@github.com:owner/repo
• ssh://git@github.com/owner/repo

1. I downgraded to v4.2.0, which is in v2.2 of argo. Doing kustomize version in the pod shows:

{Version:kustomize/v4.2.0 GitCommit:d53a2ad45d04b0264bcee9e19879437d851cb778 BuildDate:2021-06-30T22:49:26Z GoOs:linux GoArch:amd64}

2. In the existing deployment (v2.4.17) I formatted the url so it looks like below in terms of the two forward slashes, but still the same error. Just as a sanity check I also tried it with v2.2 but still the same error

https://github.com/kubernetes-sigs/kustomize//examples/multibases/dev/?ref=v1.0.6

My kustomization.yaml used the following to create the manifest:

- github.com/argoproj/argo-cd/manifests/base?ref=v2.2.0&timeout=90
- github.com/argoproj/argo-cd/manifests/crds?ref=v2.2.0&timeout=90
- github.com/argoproj/argo-cd/manifests/cluster-rbac?ref=v2.2.0&timeout=90