Autopilot installed "root" and "autopilot-bootstrap" applications keep reporting health degraded

[endophage]

We have ArgoCD installed using argocd-autopilot. We recently setup slack notifications and have since been seeing regular (a few times per day) notifications that the root and autopilot-bootstrap applications are degraded.

These applications are purely static resources. There's nothing that could be "degraded", the resources are either present or missing.

I did think possibly this could be if one of the Application resources under the root or autopilot-bootstrap applications becomes degraded. However we're not getting any notifications on any other apps becoming degraded.

Has anyone else experienced this and is there any solution beyond silencing the notifications for these specific apps?

[noam-codefresh]

do these applications keep the "degraded" state for a while, before getting back to normal? if so, can you please take a look at their status in that time? it should have more information about which resource is not in a synced state? if you can paste the status here it might be helpful as well.

i am not %100 sure this is related to autopilot, as it is being used only to commit a specific git structure to your repository. if the repo doesn't change (in the relevant directories, at least), and the apps become "degraded", i'd look for answer in argo-cd itself.

[endophage]

Thanks for the response, I just included the autopilot info for context.

In the UI I never actually see either root or autopilot-bootstrap become degraded. It seems like it recovers immediately (the healthy notifications are set at the default scope to only deliver once per revision). They're also not out of sync, these apps, especially autopilot-bootstrap rarely get any changes.

We've seen these notifications come through overnight when nobody in our org is making any changes, though they are more frequent during the day when there's more activity both from the user side and our engineers. We also see it flap sometimes, where it reports degraded (and presumably goes healthy again) multiple times inside a minute or so.

[roi-codefresh]

@endophage maybe you should check the kubernetes Events to see if you can see those transitions there also, maybe there will be more context there.