Using master and server-side-apply I get a inline-diff for resources: {} for all my apps

[Cowboy-coder]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

In order to get access to server-side-apply I tried out master of argocd. This caused me to get an inline diff like this on all my apps like this

This causes argocd to continuously try to sync without any luck.

To Reproduce

Seems to happen on all my apps. I also asked on Slack and seems one person also had seen similar issue with server-side-apply. To be fair I never tried without SSA so not sure if it's connected.

Expected behavior

I expect no diff and no sync

Screenshots

Version

argocd: v2.4.11+3d9e9f2.dirty
  BuildDate: 2022-08-22T19:32:10Z
  GitCommit: 3d9e9f2f95b7801b90377ecfc4073e5f0f07205b
  GitTreeState: dirty
  GoVersion: go1.19
  Compiler: gc
  Platform: darwin/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
argocd-server: v2.5.0+192bc50
  BuildDate: 2022-09-16T15:51:15Z
  GitCommit: 192bc5093ad4b410b84d23b70ea953f0d60512ab
  GitTreeState: clean
  GoVersion: go1.18.6
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.5 2022-05-20T20:25:40Z
  Helm Version: v3.9.3+g414ff28
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

Logs

Paste any relevant application logs here.

[leoluz]

Can you please provide an example of ArgoCD application that causes this issue? I didn't have this behavior during my tests.

[Cowboy-coder]

It's a private repo, but I can copy and paste the relevant parts

project: infra
source:
  repoURL: 'https://<privaterepo>.git'
  path: infra/local-path-provisioner
  targetRevision: HEAD
destination:
  server: 'https://kubernetes.default.svc'
syncPolicy:
  automated:
    prune: true
    selfHeal: true
    allowEmpty: true
  syncOptions:
    - ServerSideApply=true

Which picks ups and installs this yaml

apiVersion: v1
kind: Namespace
metadata:
  name: local-path-storage

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: local-path-provisioner-service-account
  namespace: local-path-storage

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: local-path-provisioner-role
rules:
  - apiGroups: [""]
    resources: ["nodes", "persistentvolumeclaims", "configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["endpoints", "persistentvolumes", "pods"]
    verbs: ["*"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: local-path-provisioner-bind
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: local-path-provisioner-role
subjects:
  - kind: ServiceAccount
    name: local-path-provisioner-service-account
    namespace: local-path-storage

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: local-path-provisioner
  namespace: local-path-storage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: local-path-provisioner
  template:
    metadata:
      labels:
        app: local-path-provisioner
    spec:
      serviceAccountName: local-path-provisioner-service-account
      containers:
        - name: local-path-provisioner
          image: rancher/local-path-provisioner:v0.0.22
          imagePullPolicy: IfNotPresent
          command:
            - local-path-provisioner
            - --debug
            - start
            - --config
            - /etc/config/config.json
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config/
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      volumes:
        - name: config-volume
          configMap:
            name: local-path-config

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: local-path-config
  namespace: local-path-storage
data:
  config.json: |-
    {
            "nodePathMap":[
            {
                    "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
                    "paths":["/opt/local-path-provisioner"]
            }
            ]
    }
  setup: |-
    #!/bin/sh
    set -eu
    mkdir -m 0777 -p "$VOL_DIR"
  teardown: |-
    #!/bin/sh
    set -eu
    rm -rf "$VOL_DIR"
  helperPod.yaml: |-
    apiVersion: v1
    kind: Pod
    metadata:
      name: helper-pod
    spec:
      containers:
      - name: helper-pod
        image: busybox
        imagePullPolicy: IfNotPresent

And this is the result. Always spec.containers[].resources being out of sync on the Deployment

[leoluz]

@Cowboy-coder @iam-veeramalla
We just merged the fix in Argo CD master.
Can you please give it another try and re-open this issue if the problem persists?
Thank you!

[yogeek]

Hello @leoluz , is this fixed only in master for now ? and if it is, when will this fix be released please ?
because we just enable serverSide Apply in our ArgoCD 2.5.2 and we have a flapping Sync<=>OutOfSync issue on multiple resources like kyverno ClusterPolicy because of empty fields.