Warning: Conflict when getting shard from shard mapping configMap

[fmunteanu]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I'm deploying ArgoCD with the Helm chart, running on 2 application controllers with dynamicClusterDistribution enabled and controller.sharding.algorithm set to round-robin. I always see the following warnings (see logs below):

• conflict when getting shard from shard mapping configMap. Retrying
• The cluster https://kubernetes.default.svc has no assigned shard

I'm not sure what the root cause of these warnings, I was wondering if anyone can provide some guidance if this is considered a bug or a configuration issue.

CM Details

$ kubectl get cm argocd-app-controller-shard-cm -n kube-system -o json | jq .data
{
  "shardControllerMapping": "[{\"ShardNumber\":0,\"ControllerName\":\"argo-cd-argocd-application-controller-6c5b5d5d66-dh2pt\",\"HeartbeatTime\":\"2024-12-15T01:13:04Z\"},{\"ShardNumber\":1,\"ControllerName\":\"argo-cd-argocd-application-controller-6c5b5d5d66-g4dzj\",\"HeartbeatTime\":\"2024-12-15T01:13:03Z\"}]"
}

$ kubectl get cm argocd-cmd-params-cm -n kube-system -o json | jq .data
{
  "application.namespaces": "kube-system",
  "applicationsetcontroller.enable.leader.election": "true",
  "applicationsetcontroller.enable.new.git.file.globbing": "true",
  "applicationsetcontroller.enable.progressive.syncs": "true",
  "applicationsetcontroller.log.format": "text",
  "applicationsetcontroller.log.level": "warn",
  "applicationsetcontroller.namespaces": "",
  "applicationsetcontroller.policy": "sync",
  "controller.ignore.normalizer.jq.timeout": "1s",
  "controller.log.format": "text",
  "controller.log.level": "warn",
  "controller.operation.processors": "10",
  "controller.repo.server.timeout.seconds": "60",
  "controller.self.heal.timeout.seconds": "5",
  "controller.sharding.algorithm": "round-robin",
  "controller.status.processors": "20",
  "otlp.address": "",
  "redis.server": "argo-cd-argocd-redis:6379",
  "repo.server": "argo-cd-argocd-repo-server:8081",
  "reposerver.log.format": "text",
  "reposerver.log.level": "warn",
  "reposerver.parallelism.limit": "0",
  "server.basehref": "/",
  "server.dex.server": "https://argo-cd-argocd-dex-server:5556",
  "server.dex.server.strict.tls": "false",
  "server.disable.auth": "false",
  "server.enable.gzip": "true",
  "server.enable.proxy.extension": "false",
  "server.insecure": "true",
  "server.log.format": "text",
  "server.log.level": "warn",
  "server.repo.server.strict.tls": "false",
  "server.rootpath": "",
  "server.staticassets": "/shared/app",
  "server.x.frame.options": "sameorigin"
}

$ kubectl get cm argocd-cm -n kube-system -o json | jq .data
{
  "application.instanceLabelKey": "argocd.argoproj.io/instance",
  "exec.enabled": "true",
  "server.rbac.log.enforce.enable": "false",
  "statusbadge.enabled": "false",
  "timeout.hard.reconciliation": "0s",
  "timeout.reconciliation": "180s",
  "url": "https://argocd.noty.cc"
}

Helm Chart Custom Values

applicationSet:
  pdb:
    enabled: true
    minAvailable: 1
  replicas: 1
  resources:
    limits:
      cpu: null
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 128Mi
configs:
  cm:
    exec.enabled: true
    statusbadge.enabled: false
  params:
    application.namespaces: kube-system
    applicationsetcontroller.enable.leader.election: true
    applicationsetcontroller.enable.new.git.file.globbing: true
    applicationsetcontroller.enable.progressive.syncs: true
    controller.sharding.algorithm: round-robin
    server.insecure: true
  rbac:
    policy.default: role:readonly
  secret:
    argocdServerAdminPassword: redacted
controller:
  dynamicClusterDistribution: true
  pdb:
    enabled: true
    minAvailable: 1
  replicas: 2
  resources:
    limits:
      cpu: null
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 128Mi
dex:
  pdb:
    enabled: true
    minAvailable: 1
  resources:
    limits:
      cpu: null
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 128Mi
global:
  domain: argocd.noty.cc
  logging:
    level: warn
notifications:
  pdb:
    enabled: true
    minAvailable: 1
  resources:
    limits:
      cpu: null
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 128Mi
redis:
  pdb:
    enabled: true
    minAvailable: 1
  resources:
    limits:
      cpu: null
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 128Mi
repoServer:
  autoscaling:
    enabled: true
    maxReplicas: 3
    minReplicas: 1
    targetMemoryUtilizationPercentage: 80
  pdb:
    enabled: true
    minAvailable: 1
  resources:
    limits:
      cpu: null
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 128Mi
server:
  autoscaling:
    enabled: true
    maxReplicas: 3
    minReplicas: 1
    targetMemoryUtilizationPercentage: 80
  pdb:
    enabled: true
    minAvailable: 1
  resources:
    limits:
      cpu: null
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 128Mi

Version

argocd: v2.13.2+dc43124
  BuildDate: 2024-12-11T19:01:33Z
  GitCommit: dc43124058130db9a747d141d86d7c2f4aac7bf9
  GitTreeState: clean
  GoVersion: go1.22.9
  Compiler: gc
  Platform: linux/arm64

Logs

time="2024-12-15T00:49:55Z" level=warning msg="conflict when getting shard from shard mapping configMap. Retrying (0/3)"
time="2024-12-15T00:49:55Z" level=warning msg="conflict when getting shard from shard mapping configMap. Retrying (1/3)"
time="2024-12-15T00:49:56Z" level=warning msg="conflict when getting shard from shard mapping configMap. Retrying (2/3)"
time="2024-12-15T00:49:56Z" level=warning msg="conflict when getting shard from shard mapping configMap. Retrying (3/3)"
time="2024-12-15T00:49:58Z" level=warning msg="The cluster https://kubernetes.default.svc has no assigned shard."

[andrii-korotkov-verkada]

How is argocd-app-controller-shard-cm updated? Is it a dynamic process?

[fmunteanu]

I'm deploying everything with the official Helm chart.

[andrii-korotkov-verkada]

I just see specific names like argo-cd-argocd-application-controller-6c5b5d5d66-dh2pt, so guess it's dynamically specified.
Can you share your application controller manifest, please?

[fmunteanu]

When I set:

controller:
  dynamicClusterDistribution: true

I see the two pod names changing from argo-cd-argocd-application-controller-0/1 to the above listed format. For now I disabled the dynamicClusterDistribution option, it order to address the warnings.

The only warning I see now is, which is not right either:

time="2024-12-22T02:37:07Z" level=warning msg="The cluster https://kubernetes.default.svc has no assigned shard."

The argo-cd-argocd-applicationset-controller-85694c5f6-g9t9p pod does not have any warnings.

I'm going to re-deploy ArgoCd with dynamicClusterDistribution enabled and post the new spec. Current spec for argo-cd-argocd-application-controller-0:

spec:
  volumes:
    - name: argocd-home
      emptyDir: {}
    - name: argocd-repo-server-tls
      secret:
        secretName: argocd-repo-server-tls
        items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          - key: ca.crt
            path: ca.crt
        defaultMode: 420
        optional: true
    - name: argocd-cmd-params-cm
      configMap:
        name: argocd-cmd-params-cm
        items:
          - key: controller.profile.enabled
            path: profiler.enabled
        defaultMode: 420
        optional: true
    - name: kube-api-access-d77x5
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: application-controller
      image: quay.io/argoproj/argocd:v2.13.2
      args:
        - /usr/local/bin/argocd-application-controller
        - '--metrics-port=8082'
      workingDir: /home/argocd
      ports:
        - name: metrics
          containerPort: 8082
          protocol: TCP
      env:
        - name: ARGOCD_CONTROLLER_REPLICAS
          value: '2'
        - name: ARGOCD_APPLICATION_CONTROLLER_NAME
          value: argo-cd-argocd-application-controller
        - name: ARGOCD_RECONCILIATION_TIMEOUT
          valueFrom:
            configMapKeyRef:
              name: argocd-cm
              key: timeout.reconciliation
              optional: true
        - name: ARGOCD_HARD_RECONCILIATION_TIMEOUT
          valueFrom:
            configMapKeyRef:
              name: argocd-cm
              key: timeout.hard.reconciliation
              optional: true
        - name: ARGOCD_RECONCILIATION_JITTER
          valueFrom:
            configMapKeyRef:
              name: argocd-cm
              key: timeout.reconciliation.jitter
              optional: true
        - name: ARGOCD_REPO_ERROR_GRACE_PERIOD_SECONDS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.repo.error.grace.period.seconds
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_REPO_SERVER
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: repo.server
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_REPO_SERVER_TIMEOUT_SECONDS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.repo.server.timeout.seconds
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_STATUS_PROCESSORS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.status.processors
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_OPERATION_PROCESSORS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.operation.processors
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_LOGFORMAT
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.log.format
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_LOGLEVEL
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.log.level
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_METRICS_CACHE_EXPIRATION
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.metrics.cache.expiration
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_SELF_HEAL_TIMEOUT_SECONDS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.self.heal.timeout.seconds
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_SELF_HEAL_BACKOFF_TIMEOUT_SECONDS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.self.heal.backoff.timeout.seconds
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_SELF_HEAL_BACKOFF_FACTOR
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.self.heal.backoff.factor
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_SELF_HEAL_BACKOFF_CAP_SECONDS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.self.heal.backoff.cap.seconds
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_REPO_SERVER_PLAINTEXT
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.repo.server.plaintext
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_REPO_SERVER_STRICT_TLS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.repo.server.strict.tls
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_PERSIST_RESOURCE_HEALTH
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.resource.health.persist
              optional: true
        - name: ARGOCD_APP_STATE_CACHE_EXPIRATION
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.app.state.cache.expiration
              optional: true
        - name: REDIS_SERVER
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: redis.server
              optional: true
        - name: REDIS_COMPRESSION
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: redis.compression
              optional: true
        - name: REDISDB
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: redis.db
              optional: true
        - name: REDIS_USERNAME
          valueFrom:
            secretKeyRef:
              name: argocd-redis
              key: redis-username
              optional: true
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: argocd-redis
              key: auth
              optional: true
        - name: REDIS_SENTINEL_USERNAME
          valueFrom:
            secretKeyRef:
              name: argo-cd-argocd-redis
              key: redis-sentinel-username
              optional: true
        - name: REDIS_SENTINEL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: argo-cd-argocd-redis
              key: redis-sentinel-password
              optional: true
        - name: ARGOCD_DEFAULT_CACHE_EXPIRATION
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.default.cache.expiration
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_OTLP_ADDRESS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: otlp.address
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_OTLP_INSECURE
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: otlp.insecure
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_OTLP_HEADERS
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: otlp.headers
              optional: true
        - name: ARGOCD_APPLICATION_NAMESPACES
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: application.namespaces
              optional: true
        - name: ARGOCD_CONTROLLER_SHARDING_ALGORITHM
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.sharding.algorithm
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_KUBECTL_PARALLELISM_LIMIT
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.kubectl.parallelism.limit
              optional: true
        - name: ARGOCD_K8SCLIENT_RETRY_MAX
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.k8sclient.retry.max
              optional: true
        - name: ARGOCD_K8SCLIENT_RETRY_BASE_BACKOFF
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.k8sclient.retry.base.backoff
              optional: true
        - name: ARGOCD_APPLICATION_CONTROLLER_SERVER_SIDE_DIFF
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.diff.server.side
              optional: true
        - name: ARGOCD_IGNORE_NORMALIZER_JQ_TIMEOUT
          valueFrom:
            configMapKeyRef:
              name: argocd-cmd-params-cm
              key: controller.ignore.normalizer.jq.timeout
              optional: true
      resources:
        limits:
          memory: 256Mi
        requests:
          cpu: 10m
          memory: 256Mi
      volumeMounts:
        - name: argocd-repo-server-tls
          mountPath: /app/config/controller/tls
        - name: argocd-home
          mountPath: /home/argocd
        - name: argocd-cmd-params-cm
          mountPath: /home/argocd/params
        - name: kube-api-access-d77x5
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readinessProbe:
        httpGet:
          path: /healthz
          port: metrics
          scheme: HTTP
        initialDelaySeconds: 10
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
        seccompProfile:
          type: RuntimeDefault
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: argocd-application-controller
  serviceAccount: argocd-application-controller
  automountServiceAccountToken: true
  nodeName: chaos
  securityContext: {}
  hostname: argo-cd-argocd-application-controller-0
  subdomain: argo-cd-argocd-application-controller
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/name: argocd-application-controller
            topologyKey: kubernetes.io/hostname
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority