ArgoCD v2.3.0 - ARGOCD_GIT_ASKPASS_NONCE is not set

[jasonmacdonald]

Describe the bug

I get the following error in my ApplicationSet even though I can confirm that the repo settings show connectionstatus as successful.

Error during fetching repo: `git fetch origin HEAD --tags --force`
failed exit status 128: time="2022-03-08T16:02:47Z" level=fatal
msg="ARGOCD_GIT_ASKPASS_NONCE is not set"

error: unable to read askpass response from 'argocd'

fatal: could not read Username for 'https://gitlab.<mycompany>.com':
 terminal prompts disabled
reason: ApplicationGenerationFromParamsError
status: 'True'
type: ErrorOccurred

This is in a Kubernetes EKS cluster in AWS.

Expected behavior

it should load the repository and create the applications

Screenshots

Version

argocd: v2.2.5+8f981cc.dirty
  BuildDate: 2022-02-05T05:20:39Z
  GitCommit: 8f981ccfcf942a9eb00bc466649f8499ba0455f5
  GitTreeState: dirty
  GoVersion: go1.17.6
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.3.0+fe42780
  BuildDate: 2022-03-06T06:27:40Z
  GitCommit: fe427802293b090f43f91f5839393174df6c3b3a
  GitTreeState: clean
  GoVersion: go1.17.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
  Helm Version: v3.8.0+gd141386
  Kubectl Version: v0.23.1
  Jsonnet Version: v0.18.0

ApplicationSet Output

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: '0'
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"argoproj.io/v1alpha1","kind":"ApplicationSet","metadata":{"annotations":{"argocd.argoproj.io/sync-wave":"0"},"creationTimestamp":null,"name":"cluster-addons","namespace":"argocd"},"spec":{"generators":[{"git":{"files":[{"path":"cluster-addons/**/dev.chart.yaml"}],"repoURL":"https://gitlab.******.com/******/infra/applications.git","requeueAfterSeconds":20,"revision":"HEAD","template":{"metadata":{},"spec":{"destination":{},"project":"","source":{"chart":"{{chart}}","helm":{"ignoreMissingValueFiles":true,"releaseName":"{{helmReleaseName}}","skipCrds":true,"valueFiles":["{{valueFilesGlobals}}","{{valueFiles}}"],"values":"{{values}}\n"},"path":"{{srcPath}}","repoURL":"{{repoURL}}","targetRevision":"{{targetRevision}}"},"syncPolicy":{"syncOptions":["Replace={{syncOptionsReplace}}","CreateNamespace=true"]}}}}},{"git":{"files":[{"path":"cluster-addons/**/dev.app.yaml"}],"repoURL":"https://gitlab.******.com/******/infra/applications.git","requeueAfterSeconds":20,"revision":"HEAD","template":{"metadata":{},"spec":{"destination":{},"project":"","source":{"directory":{"recurse":true},"path":"{{srcPath}}","repoURL":"{{repoURL}}"},"syncPolicy":{"syncOptions":["Replace={{syncOptionsReplace}}","CreateNamespace=true"]}}}}}],"syncPolicy":{"preserveResourcesOnDeletion":false},"template":{"metadata":{"labels":{"app.kubernetes.io/name":"{{
      appName }}"},"name":"{{ appName
      }}","namespace":"{{namespace}}"},"spec":{"destination":{"namespace":"{{namespace}}","server":"https://kubernetes.default.svc"},"ignoreDifferences":[{"group":"argoproj.io","jsonPointers":["/status"],"kind":"Application"}],"project":"cluster-addons","source":{"repoURL":"{{repoURL}}","targetRevision":"{{targetRevision}}"},"syncPolicy":{"automated":{"allowEmpty":true,"prune":true,"selfHeal":true}}}}},"status":{}}
  creationTimestamp: '2022-03-08T15:42:50Z'
  generation: 4
  managedFields:
    - apiVersion: argoproj.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
      manager: applicationset-controller
      operation: Update
      time: '2022-03-08T15:42:50Z'
    - apiVersion: argoproj.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:argocd.argoproj.io/sync-wave: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:generators: {}
          f:syncPolicy:
            .: {}
            f:preserveResourcesOnDeletion: {}
          f:template:
            .: {}
            f:metadata:
              .: {}
              f:labels:
                .: {}
                f:app.kubernetes.io/name: {}
              f:name: {}
              f:namespace: {}
            f:spec:
              .: {}
              f:destination:
                .: {}
                f:namespace: {}
                f:server: {}
              f:ignoreDifferences: {}
              f:project: {}
              f:source:
                .: {}
                f:repoURL: {}
                f:targetRevision: {}
              f:syncPolicy:
                .: {}
                f:automated:
                  .: {}
                  f:allowEmpty: {}
                  f:prune: {}
                  f:selfHeal: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2022-03-08T15:42:50Z'
  name: cluster-addons
  namespace: argocd
  resourceVersion: '951827'
  uid: 514358e5-3ce5-44ca-ac40-5de9d303cae2
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/argocd/applicationsets/cluster-addons
status:
  conditions:
    - lastTransitionTime: '2022-03-08T16:19:27Z'
      message: >-
        Error during fetching repo: `git fetch origin HEAD --tags --force`
        failed exit status 128: time="2022-03-08T16:19:27Z" level=fatal
        msg="ARGOCD_GIT_ASKPASS_NONCE is not set"

        error: unable to read askpass response from 'argocd'

        fatal: could not read Username for 'https://gitlab.******.com':
        terminal prompts disabled
      reason: ApplicationGenerationFromParamsError
      status: 'True'
      type: ErrorOccurred
    - lastTransitionTime: '2022-03-08T16:19:27Z'
      message: >-
        Error during fetching repo: `git fetch origin HEAD --tags --force`
        failed exit status 128: time="2022-03-08T16:19:27Z" level=fatal
        msg="ARGOCD_GIT_ASKPASS_NONCE is not set"

        error: unable to read askpass response from 'argocd'

        fatal: could not read Username for 'https://gitlab.******.com':
        terminal prompts disabled
      reason: ErrorOccurred
      status: 'False'
      type: ParametersGenerated
    - lastTransitionTime: '2022-03-08T16:19:27Z'
      message: >-
        Error during fetching repo: `git fetch origin HEAD --tags --force`
        failed exit status 128: time="2022-03-08T16:19:27Z" level=fatal
        msg="ARGOCD_GIT_ASKPASS_NONCE is not set"

        error: unable to read askpass response from 'argocd'

        fatal: could not read Username for 'https://gitlab.******.com':
        terminal prompts disabled
      reason: ApplicationGenerationFromParamsError
      status: 'False'
      type: ResourcesUpToDate
spec:
  generators:
    - git:
        files:
          - path: cluster-addons/**/dev.chart.yaml
        repoURL: https://gitlab.******.com/******/infra/applications.git
        requeueAfterSeconds: 20
        revision: HEAD
        template:
          metadata: {}
          spec:
            destination: {}
            project: ''
            source:
              chart: '{{chart}}'
              helm:
                ignoreMissingValueFiles: true
                releaseName: '{{helmReleaseName}}'
                skipCrds: true
                valueFiles:
                  - '{{valueFilesGlobals}}'
                  - '{{valueFiles}}'
                values: |
                  {{values}}
              path: '{{srcPath}}'
              repoURL: '{{repoURL}}'
              targetRevision: '{{targetRevision}}'
            syncPolicy:
              syncOptions:
                - Replace={{syncOptionsReplace}}
                - CreateNamespace=true
    - git:
        files:
          - path: cluster-addons/**/dev.app.yaml
        repoURL: https://gitlab.******.com/******/infra/applications.git
        requeueAfterSeconds: 20
        revision: HEAD
        template:
          metadata: {}
          spec:
            destination: {}
            project: ''
            source:
              directory:
                recurse: true
              path: '{{srcPath}}'
              repoURL: '{{repoURL}}'
            syncPolicy:
              syncOptions:
                - Replace={{syncOptionsReplace}}
                - CreateNamespace=true
  syncPolicy:
    preserveResourcesOnDeletion: false
  template:
    metadata:
      labels:
        app.kubernetes.io/name: '{{ appName }}'
      name: '{{ appName }}'
      namespace: '{{namespace}}'
    spec:
      destination:
        namespace: '{{namespace}}'
        server: https://kubernetes.default.svc
      ignoreDifferences:
        - group: argoproj.io
          jsonPointers:
            - /status
          kind: Application
      project: cluster-addons
      source:
        repoURL: '{{repoURL}}'
        targetRevision: '{{targetRevision}}'
      syncPolicy:
        automated:
          allowEmpty: true
          prune: true
          selfHeal: true

[jasonmacdonald]

I confirmed that rolling back to 2.2.6 fixes the issue. If I then try to apply the new manifests, it breaks again. Nothing else changing between the two other than the newer manifests.

This is using a Gitlab deploy token for authentication. Not sure if that affects it or not.

[jasonmacdonald]

Further investigations, if I apply the older version of ApplicationSet CRDs overtop of the newer ones deployed by Argo-cd things also work again. So it seems to have something to do with the newer CRD manifests at https://raw.githubusercontent.com/argoproj/applicationset/v0.4.0/manifests/install.yaml

[K890]

I can confirm your investigation is right.
Just by applying 0.3.0 manifests of argocd applicationset, the error went away.

[jasonmacdonald]

Just an update, they are trying to address the issue with a 0.4.1 release of the applicationset CRDs with this merge argoproj/applicationset#535. I assume once we apply that it should resolve the issue. Not sure if this will require an update to ArgoCD 2.3.0 though

[sharon-hunters]

To my understanding starting from ArgoCD 2.3.0 applicationset controller is bundled in the installation manifests, so I would expect to have a new ArgoCD release for this fix.

[tooptoop4]

@sharon-hunters does 2.3.1 fix this?

[an-tex]

@tooptoop4 yep

[alexmt]

v2.3.1 release with fixed application set version has been published.

[cherniukandrei]

Please update the documentation accordingly to reflect the fix. The command should be
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/applicationset/v0.4.1/manifests/install.yaml