Unexpected (and Constant) Application Updates

[DaytonG]

Checklist:

• [ X ] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• [ X ] I've included steps to reproduce the bug.
• [ X ] I've pasted the output of argocd version.

Describe the bug

When using helm, found that our Application object's "generation" field was constantly updating. After some help, we discovered it was because of a few empty fields: source.path (was set to empty string), source.helm.parameters (empty list), and source.helm.valuesFiles (empty list). I removed all three fields from our Application objects, and things stopped refreshing.

Some other things to note:

• sync policy is set to automated
• we are using the app-of-apps pattern. The issue was present in the sub-apps. Didn't notice it in the parent app objects.
• the UI showed no diffs.
• Once I removed the empty fields, I had to manually trigger a sync. Argo did not roll out the change itself.
• the only reason I noticed was because the YAML manifests in the UI would not scroll because the text was constantly changing. There was no other indication a problem was occurring.

To Reproduce

Deploy an Application like so, using app-of-apps:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  name: opining-molly-redis
  namespace: argocd
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  ignoreDifferences:
  - jsonPointers:
    - "/spec/clusterIP"
    kind: Service
  project: default
  source:
    chart: redis
    helm:
      releaseName: opining-molly
      valuesFiles: []
      parameters: []
      path: '' 
      values: |-
        "master":
          "persistence":
            "enabled": false
        "metrics":
          "enabled": true
        "password": "redacted"
        "rbac":
          "create": true
        "sentinel":
          "enabled": false
        "serviceAccount":
          "create": true
        "slave":
          "persistence":
            "enabled": false
    repoURL: https://charts.bitnami.com/bitnami
    targetRevision: 10.4.1
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Expected behavior

Application allows for empty fields to be present without causing behind-the-scenes refreshes.

Version

# Took this from within the container, as we haven't hooked up the client from outside kubernetes yet
argocd@mothy-horse-argocd-repo-server-7bf9d9f6dc-m52td:~$ argocd version --client
argocd: v1.4.2+48cced9
  BuildDate: 2020-01-24T01:04:04Z
  GitCommit: 48cced9d925b5bc94f6aa9fa4a8a19b2a59e128a
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64

Logs

From the application-controller pod:

time="2020-03-20T19:35:43Z" level=info msg="Update successful" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:43Z" level=info msg="Reconciliation completed" application=zany-antelope-dds-tls-secret dest-namespace=argocd dest-server="https://kubernetes.default.svc" fields.level=2 time_ms=807.498233
time="2020-03-20T19:35:43Z" level=info msg="Refreshing app status (spec.source differs), level (2)" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:43Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:43Z" level=info msg="Skipping auto-sync: application status is Synced" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:44Z" level=info msg="Update successful" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:44Z" level=info msg="Reconciliation completed" application=zany-antelope-dds-tls-secret dest-namespace=argocd dest-server="https://kubernetes.default.svc" fields.level=2 time_ms=802.009486
time="2020-03-20T19:35:44Z" level=info msg="Refreshing app status (spec.source differs), level (2)" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:44Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:44Z" level=info msg="Skipping auto-sync: application status is Synced" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:45Z" level=info msg="Update successful" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:45Z" level=info msg="Reconciliation completed" application=zany-antelope-dds-tls-secret dest-namespace=argocd dest-server="https://kubernetes.default.svc" fields.level=2 time_ms=801.774317
time="2020-03-20T19:35:46Z" level=info msg="Refreshing app status (spec.source differs), level (2)" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:46Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=zany-antelope-dds-tls-secret
time="2020-03-20T19:35:46Z" level=info msg="Skipping auto-sync: application status is Synced" application=zany-antelope-dds-tls-secret

[jessesuen]

Looking into it

[jessesuen]

I could not reproduce this in both v1.4.2 and v1.5.0-rc1. After the initial deploy where the application reconciles very frequently (because there are changes happening with the managed objects), the app eventually "settled" and there were no refreshes.

Notice the reason for refreshes is logged in the controller. In my case it is because controller refresh requested. Whereas in your case, it is spec.source differs

$ kl argocd-application-controller-6c869db745-z9hmv | grep application=opining-molly-redis | grep "Refreshing app status"
time="2020-03-24T02:09:27Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: never, expiry: 3m0s), level (2)" application=opining-molly-redis
time="2020-03-24T02:10:08Z" level=info msg="Refreshing app status (normal refresh requested), level (2)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:10Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:11Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:11Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:11Z" level=info msg="Refreshing app status (controller refresh requested), level (2)" application=opining-molly-redis
time="2020-03-24T02:10:12Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=opining-molly-redis
time="2020-03-24T02:10:12Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=opining-molly-redis
time="2020-03-24T02:10:21Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=opining-molly-redis
time="2020-03-24T02:10:21Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:52Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=opining-molly-redis
time="2020-03-24T02:10:52Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:10:53Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=opining-molly-redis
time="2020-03-24T02:11:03Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=opining-molly-redis
time="2020-03-24T02:11:04Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=opining-molly-redis
time="2020-03-24T02:11:09Z" level=info msg="Refreshing app status (normal refresh requested), level (2)" application=opining-molly-redis

Can you share any argocd-cm configuration that you might have set?

[jessesuen]

Here is the relevant code where we decide whether or not we need to reconcile:

	} else if !app.Spec.Source.Equals(app.Status.Sync.ComparedTo.Source) {
		reason = "spec.source differs"

Equals is simply a DeepEquals against the structured objects:

// Equals compares two instances of ApplicationSource and return true if instances are equal.
func (source *ApplicationSource) Equals(other ApplicationSource) bool {
	return reflect.DeepEqual(*source, other)
}

So it seems in your case, app.spec.source does not equal to app.status.sync.comparedTo.source.

Can you share the output of:

kubectl get app -o yaml zany-antelope-dds-tls-secret

Please do this when the app is in the refresh loop to catch why the comparison is not equal.

[DaytonG]

application-opining-molly-redis.txt

Here you go. That's from one of our test clusters.

[chiragthaker]

i can see the same thing with our setup , is this solved with new versions ?

[JonathanSerafini]

Seeing the same symptoms but in a slightly different situation ... while running argoproj/argocd:v1.5.4

Symptoms:

• high cpu usage on app controller
• Refreshing app status (spec.source differs), level (2) logs for the app-of-apps and it's descendant app objects
• generation incrementing on app-of-apps and descendant app objects

Setup :

• apps deployed via kubectl ( no issues )
• app-of-apps deployed and synced ( issues begin )
• helm charts with git sources

After having removed the extraneous fields as mention by OP, CPU usage has dropped and the log messages are no longer present. The generation field seems to be increasing gradually but is doing so at a much slower rate than before.

[jessesuen]

@alexmt is working on unnecessary app reconciliations in v1.7 (duping to #1533)

[ebuildy]

Still see it, argocd 2.8

The application is basically deployed like that, nothing seem changing:

How can I "force" a refresh interval ?