Error Generating Manifest: argocd-repo-server service account doesn't have access to argocd namespace to get secret for AVP

[tharpooljha]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

argocd-repo-server service account cannot access the argocd namespace 'vault-configuration' secret.

ComparisonError: rpc error: code = Unknown desc = plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `sh -c kustomize build . | argocd-vault-plugin generate -s vault-configuration -` failed exit status 1: Error: secrets "vault-configuration" is forbidden: User "system:serviceaccount:argocd:argocd-repo-server" cannot get resource "secrets" in API group "" in the namespace "argocd" Usage: argocd-vault-plugin generate <path> [flags] Flags: -c, --config-path string path to a file containing Vault configuration (YAML, JSON, envfile) to use -h, --help help for generate -s, --secret-name string name of a Kubernetes Secret in the argocd namespace containing Vault configuration data in the argocd namespace of your ArgoCD host (Only available when used in ArgoCD). The namespace can be overridden by using the format <namespace>:<name> --verbose-sensitive-output enable verbose mode for detailed info to help with debugging. Includes sensitive data (credentials), logged to stderr

To Reproduce

create vault-configuration secret in argocd namespace.

apiVersion: v1
stringData:
  AVP_AUTH_TYPE: k8s
  AVP_K8S_ROLE: argocd
  VAULT_ADDR: http://vault:8200
  AVP_TYPE: vault
kind: Secret
metadata:
  name: vault-configuration
  namespace: argocd
type: Opaque

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd

resources:
- namespace.yaml
- https://github.com/argoproj/argo-cd/manifests/cluster-install?ref=v2.4.11
- cmp-plugin.yaml
- vault-configuration.yaml
- repository.yaml
#- secrets.yaml


patchesStrategicMerge:
- argocd-secret.yaml
- argocd-repo-server.yaml

argocd-repo-server.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:

      # Mount SA token for Kubernets auth
      # Note: In 2.4.0 onward, there is a dedicated SA for repo-server (not default)
      # Note: This is not fully supported for Kubernetes < v1.19
      automountServiceAccountToken: true

      # Each of the embedded YAMLs inside cmp-plugin ConfigMap will be mounted into it's respective plugin sidecar
      volumes:
        - configMap:
            name: cmp-plugin
          name: cmp-plugin
        - name: custom-tools
          emptyDir: {}

      # Download tools
      initContainers:
      - name: download-tools
        image: registry.access.redhat.com/ubi8
        env:
          - name: AVP_VERSION
            value: 1.12.0
        command: [sh, -c]
        args:
          - >-
            curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools

      # argocd-vault-plugin with Helm
      containers:
      - name: avp-helm
        command: [/var/run/argocd/argocd-cmp-server]
        image: quay.io/argoproj/argocd:v2.4.11
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp

          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: avp-helm.yaml
            name: cmp-plugin

          # Important: Mount tools into $PATH
          - name: custom-tools
            subPath: argocd-vault-plugin
            mountPath: /usr/local/bin/argocd-vault-plugin

      # argocd-vault-plugin with Kustomize
      - name: avp-kustomize
        command: [/var/run/argocd/argocd-cmp-server]
        image: quay.io/argoproj/argocd:v2.4.11
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp

          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: avp-kustomize.yaml
            name: cmp-plugin

          # Important: Mount tools into $PATH
          - name: custom-tools
            subPath: argocd-vault-plugin
            mountPath: /usr/local/bin/argocd-vault-plugin

      # argocd-vault-plugin with plain YAML
      - name: avp
        command: [/var/run/argocd/argocd-cmp-server]
        image: quay.io/argoproj/argocd:v2.4.11
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp

          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: avp.yaml
            name: cmp-plugin

          # Important: Mount tools into $PATH
          - name: custom-tools
            subPath: argocd-vault-plugin
            mountPath: /usr/local/bin/argocd-vault-plugin

cmp-plugin.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  avp-kustomize.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-kustomize
    spec:
      allowConcurrency: true
      # Note: this command is run _before_ anything is done, therefore the logic is to check
      # if this looks like a Kustomize bundle
      discover:
        find:
          command:
            - find
            - "."
            - -name
            - kustomization.yaml
      generate:
        command:
          - sh
          - "-c"
          - "kustomize build . | argocd-vault-plugin generate -s argocd:vault-configuration -"
      lockRepo: false
  avp-helm.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      # Note: this command is run _before_ any Helm templating is done, therefore the logic is to check
      # if this looks like a Helm chart
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      generate:
        command:
          - sh
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |
            argocd-vault-plugin generate -s argocd:vault-configuration -
      lockRepo: false
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
      generate:
        command:
          - argocd-vault-plugin
          - generate
          - "."
      lockRepo: false

argocd-AppSet.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  name: argocd-appsets
spec:
  generators:
  - list:
      elements:
      - cluster: core
        cluster-name: in-cluster
        overlay: overlays/core
        targetRevision: HEAD



  template:
    metadata:
      name: '{{cluster}}-argo'
      labels:
        app: '{{cluster}}-argo'
    spec:
      project: infra-projects
      source:
        repoURL: https://github.com/asdf/DevOps.git
        targetRevision: '{{ targetRevision }}'
        path: apps/argocd/{{overlay}}
      destination:
        name: '{{ cluster-name }}'
        namespace: argocd
      # Sync policy
      syncPolicy:
        # disabled autoSync with A1ODT-461 to avoid accidental damage
       automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
         prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
         selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
         allowEmpty: true # Allows deleting all application resources during automatic syncing ( false by default ).
        syncOptions: # Sync options which modifies sync behavior
          - Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).
          - CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
          - PrunePropagationPolicy=foreground # Supported policies are background, foreground and orphan.
          - ApplyOutOfSyncOnly=true
        # The retry feature is available since v1.7
        retry:
          limit: 5 # number of failed sync attempt retries; unlimited number of attempts if less than 0
          backoff:
            duration: 5s # the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
            factor: 2 # a factor to multiply the base duration after each failed retry
            maxDuration: 3m # the maximum amount of time allowed for the backoff strategy

---

Expected behavior

I'm trying to deploy my argocd git ops folder, and its unable to deploy applicationsets, projects, repos from the environments core folder because of the issue. I would expect to be able to deploy a argo application manifest. I'm not defining any environmental variables or secrets in any of the argo-appSet applications.

Screenshots

Version

argocd v 2.4.11

Logs

time="2022-09-01T20:48:27Z" level=error msg="finished streaming call with code Unknown" error="error generating manifests: `sh -c kustomize build . | argocd-vault-plugin generate -s argocd:vault-configuration -` failed exit status 1: Error: secrets \"vault-configuration\" is forbidden: User \"system:serviceaccount:argocd:argocd-repo-server\" cannot get resource \"secrets\" in API group \"\" in the namespace \"argocd\"\nUsage:\n  argocd-vault-plugin generate <path> [flags]\n\nFlags:\n  -c, --config-path string         path to a file containing Vault configuration (YAML, JSON, envfile) to use\n  -h, --help                       help for generate\n  -s, --secret-name string         name of a Kubernetes Secret in the argocd namespace containing Vault configuration data in the argocd namespace of your ArgoCD host (Only available when used in ArgoCD). The namespace can be overridden by using the format <namespace>:<name>\n      --verbose-sensitive-output   enable verbose mode for detailed info to help with debugging. Includes sensitive data (credentials), logged to stderr" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2022-09-01T20:48:26Z" grpc.time_ms=1034.048 span.kind=server system=grpc

Paste any relevant application logs here.

[TheAnachronism]

I have the same issue. How can I tell the repo-server service account that it can access a secret aswell?

[TheAnachronism]

Alright, I solved it by adding these two files in my kustomize deployment:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd-repo-server-role
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["secrets"]
  verbs: ["get"]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd-repo-server-rolebinding
subjects:
- kind: ServiceAccount
  name: argocd-repo-server
roleRef:
  # "roleRef" specifies the binding to a Role / ClusterRole
  kind: Role #this must be Role or ClusterRole
  name: argocd-repo-server-role # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

I discovered, that there is a service account for the repo server, but it doesn't have any role bindings. So, I created one, so it could access a secret in the same namespace.

[crenshaw-dev]

Yep, by default, the repo-server has no access. That's intentional. If your plugin needs more access, you'll have to grant that in your environment.