Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manage Argo CD Using Argo CD Reconciliation loop #1533

Closed
KarstenSiemer opened this issue Apr 26, 2019 · 17 comments · Fixed by #1547 or #1608
Closed

Manage Argo CD Using Argo CD Reconciliation loop #1533

KarstenSiemer opened this issue Apr 26, 2019 · 17 comments · Fixed by #1547 or #1608
Labels
bug Something isn't working component:application-sets Bulk application management related
Milestone

Comments

@KarstenSiemer
Copy link
Contributor

KarstenSiemer commented Apr 26, 2019

Describe the bug
Apparently i've somehow build a reconciliation loop by managing argocd using argocd.

argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Update successful" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=189.357799
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Refreshing app status (controller refresh requested)" application=argo-cd
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:29Z" level=info msg="received unary call /application.ApplicationService/ResourceTree" grpc.method=ResourceTree grpc.request.claims="{\"acr\":\"0\",\"aud\":\"cd.svc.xxx.de\",\"auth_time\":1556267652,\"azp\":\"cd.svc.xxx.de\",\"email\":\"k.siemer@domain.de\",\"exp\":1556276350,\"groups\":[\"/teams/black-ops\",\"/orgs/engineering\",\"/teams/operations\",\"/k8s/cluster/admins\"],\"iat\":1556275450,\"iss\":\"https://id.domain.net/auth/realms/xxx\",\"jti\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"nbf\":0,\"nonce\":\"xxxxxxxxx\",\"session_state\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"sub\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"typ\":\"ID\"}" grpc.request.content="applicationName:\"argo-cd\" " grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:29Z" span.kind=server system=grpc
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:29Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:29Z" grpc.time_ms=6.115 span.kind=server system=grpc
argocd-application-controller-65d85974d-t9kwx argocd-application-controller W0426 10:53:29.569909       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Update successful" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=201.867236
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Refreshing app status (controller refresh requested)" application=argo-cd
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:29Z" level=info msg="received unary call /application.ApplicationService/ResourceTree" grpc.method=ResourceTree grpc.request.claims="{\"acr\":\"0\",\"aud\":\"cd.svc.xxx.de\",\"auth_time\":1556267652,\"azp\":\"cd.svc.xxx.de\",\"email\":\"k.siemer@domain.de\",\"exp\":1556276350,\"groups\":[\"/teams/black-ops\",\"/orgs/engineering\",\"/teams/operations\",\"/k8s/cluster/admins\"],\"iat\":1556275450,\"iss\":\"https://id.domain.net/auth/realms/xxx\",\"jti\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"nbf\":0,\"nonce\":\"xxxxxxxxx\",\"session_state\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"sub\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"typ\":\"ID\"}" grpc.request.content="applicationName:\"argo-cd\" " grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:29Z" span.kind=server system=grpc
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:29Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:29Z" grpc.time_ms=6.487 span.kind=server system=grpc
argocd-application-controller-65d85974d-t9kwx argocd-application-controller W0426 10:53:29.772895       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Update successful" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=187.965437
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Refreshing app status (controller refresh requested)" application=argo-cd
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:29Z" level=info msg="received unary call /application.ApplicationService/ResourceTree" grpc.method=ResourceTree grpc.request.claims="{\"acr\":\"0\",\"aud\":\"cd.svc.xxx.de\",\"auth_time\":1556267652,\"azp\":\"cd.svc.xxx.de\",\"email\":\"k.siemer@domain.de\",\"exp\":1556276350,\"groups\":[\"/teams/black-ops\",\"/orgs/engineering\",\"/teams/operations\",\"/k8s/cluster/admins\"],\"iat\":1556275450,\"iss\":\"https://id.domain.net/auth/realms/xxx\",\"jti\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"nbf\":0,\"nonce\":\"xxxxxxxxx\",\"session_state\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"sub\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"typ\":\"ID\"}" grpc.request.content="applicationName:\"argo-cd\" " grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:29Z" span.kind=server system=grpc
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:29Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:29Z" grpc.time_ms=5.521 span.kind=server system=grpc
argocd-application-controller-65d85974d-t9kwx argocd-application-controller W0426 10:53:29.960502       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:29Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Update successful" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=183.817583
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Refreshing app status (controller refresh requested)" application=argo-cd
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="received unary call /application.ApplicationService/ResourceTree" grpc.method=ResourceTree grpc.request.claims="{\"acr\":\"0\",\"aud\":\"cd.svc.xxx.de\",\"auth_time\":1556267652,\"azp\":\"cd.svc.xxx.de\",\"email\":\"k.siemer@domain.de\",\"exp\":1556276350,\"groups\":[\"/teams/black-ops\",\"/orgs/engineering\",\"/teams/operations\",\"/k8s/cluster/admins\"],\"iat\":1556275450,\"iss\":\"https://id.domain.net/auth/realms/xxx\",\"jti\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"nbf\":0,\"nonce\":\"xxxxxxxxx\",\"session_state\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"sub\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"typ\":\"ID\"}" grpc.request.content="applicationName:\"argo-cd\" " grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" span.kind=server system=grpc
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" grpc.time_ms=26.681 span.kind=server system=grpc
argocd-application-controller-65d85974d-t9kwx argocd-application-controller W0426 10:53:30.153956       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Update successful" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=190.51399899999998
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Refreshing app status (controller refresh requested)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller W0426 10:53:30.338137       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="received unary call /application.ApplicationService/ResourceTree" grpc.method=ResourceTree grpc.request.claims="{\"acr\":\"0\",\"aud\":\"cd.svc.xxx.de\",\"auth_time\":1556267652,\"azp\":\"cd.svc.xxx.de\",\"email\":\"k.siemer@domain.de\",\"exp\":1556276350,\"groups\":[\"/teams/black-ops\",\"/orgs/engineering\",\"/teams/operations\",\"/k8s/cluster/admins\"],\"iat\":1556275450,\"iss\":\"https://id.domain.net/auth/realms/xxx\",\"jti\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"nbf\":0,\"nonce\":\"xxxxxxxxx\",\"session_state\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"sub\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"typ\":\"ID\"}" grpc.request.content="applicationName:\"argo-cd\" " grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" span.kind=server system=grpc
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" grpc.time_ms=6.126 span.kind=server system=grpc
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Update successful" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=207.910877
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Refreshing app status (controller refresh requested)" application=argo-cd
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="received unary call /application.ApplicationService/ResourceTree" grpc.method=ResourceTree grpc.request.claims="{\"acr\":\"0\",\"aud\":\"cd.svc.xxx.de\",\"auth_time\":1556267652,\"azp\":\"cd.svc.xxx.de\",\"email\":\"k.siemer@domain.de\",\"exp\":1556276350,\"groups\":[\"/teams/black-ops\",\"/orgs/engineering\",\"/teams/operations\",\"/k8s/cluster/admins\"],\"iat\":1556275450,\"iss\":\"https://id.domain.net/auth/realms/xxx\",\"jti\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"nbf\":0,\"nonce\":\"xxxxxxxxx\",\"session_state\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"sub\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"typ\":\"ID\"}" grpc.request.content="applicationName:\"argo-cd\" " grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" span.kind=server system=grpc
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" grpc.time_ms=5.991 span.kind=server system=grpc
argocd-application-controller-65d85974d-t9kwx argocd-application-controller W0426 10:53:30.547050       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Update successful" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=186.86920700000002
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Refreshing app status (controller refresh requested)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller W0426 10:53:30.738720       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="received unary call /application.ApplicationService/ResourceTree" grpc.method=ResourceTree grpc.request.claims="{\"acr\":\"0\",\"aud\":\"cd.svc.xxx.de\",\"auth_time\":1556267652,\"azp\":\"cd.svc.xxx.de\",\"email\":\"k.siemer@domain.de\",\"exp\":1556276350,\"groups\":[\"/teams/black-ops\",\"/orgs/engineering\",\"/teams/operations\",\"/k8s/cluster/admins\"],\"iat\":1556275450,\"iss\":\"https://id.domain.net/auth/realms/xxx\",\"jti\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"nbf\":0,\"nonce\":\"xxxxxxxxx\",\"session_state\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"sub\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"typ\":\"ID\"}" grpc.request.content="applicationName:\"argo-cd\" " grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" span.kind=server system=grpc
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" grpc.time_ms=9.8 span.kind=server system=grpc
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Update successful" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=190.88876299999998
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Refreshing app status (controller refresh requested)" application=argo-cd
argocd-application-controller-65d85974d-t9kwx argocd-application-controller W0426 10:53:30.927467       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:53:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: system-cicd)" application=argo-cd
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="received unary call /application.ApplicationService/ResourceTree" grpc.method=ResourceTree grpc.request.claims="{\"acr\":\"0\",\"aud\":\"cd.svc.xxx.de\",\"auth_time\":1556267652,\"azp\":\"cd.svc.xxx.de\",\"email\":\"k.siemer@domain.de\",\"exp\":1556276350,\"groups\":[\"/teams/black-ops\",\"/orgs/engineering\",\"/teams/operations\",\"/k8s/cluster/admins\"],\"iat\":1556275450,\"iss\":\"https://id.domain.net/auth/realms/xxx\",\"jti\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"nbf\":0,\"nonce\":\"xxxxxxxxx\",\"session_state\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"sub\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"typ\":\"ID\"}" grpc.request.content="applicationName:\"argo-cd\" " grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" span.kind=server system=grpc
argocd-server-7dc6666d45-mk8lp argocd-server time="2019-04-26T10:53:30Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2019-04-26T10:53:30Z" grpc.time_ms=7.142 span.kind=server system=grpc

As it can be seen here, the controller constantly requests a refresh of the argo-cd application.
Even though it is already healthy and in sync. The auto sync feature isn't even on.

Screenshots
argocd

The argo-cd application is owned by the system-cicd appproject.
The argocd-apps application is owned by argocd-objects appproject and used to create applications that are lying inside git.
The argocd-projects application is also owned by argocd-objects and used to create appprojects that are lying inside git.

I do not see any reason why it would be in a loop.
Here you can see, that it is trying to reconcile very often:

argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:45:51Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=186.208233
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:45:51Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=196.083342
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:45:52Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=192.908786
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:45:52Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=192.06886
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:45:52Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=205.887309
argocd-application-controller-65d85974d-t9kwx argocd-application-controller time="2019-04-26T10:45:52Z" level=info msg="Reconciliation completed" application=argo-cd full=true time_ms=196.757855

I am also unsure what can not retrieve list of objects using index : Index with name namespace does not exist means.
Since argo-cd works wonderfully on all applications that i've passed to it.

Versions:
argo-cd: v1.0.0rc0
kubernetes: v1.14.1

I am having this problem for a few releases now. I am not sure when it started, but in version v0.10.3 this was working just fine

@KarstenSiemer KarstenSiemer added the bug Something isn't working label Apr 26, 2019
@KarstenSiemer
Copy link
Contributor Author

I can't find why it is happening. And thus cannot use that feature, which is quite sad, since it is actually the most important feature and why i use argocd

@Oded-B
Copy link

Oded-B commented Apr 30, 2019

We had the same issue when trying to have an ArgoCD application manage all application including the aforementioned application.
We had the same looped arrow in the UI and the same reconciliation loop.
Once we removed the recursive configuration, so we manage all other application declaratively, and the argocd configuration application manually, the reconciliation loop stopped:

image

@alexec
Copy link
Contributor

alexec commented Apr 30, 2019

After some discussion, we're not sure if this is a feature of a bug. Certainly, people have Argo CD self-manage, but due to how the plumbing now works inside it, this will result in a reconciliation loop.

There's no simple fix to this AFAIK.

This workaround is to manage Argo CD outside of itself, be that manually or with another Argo CD instance.

@alexmt
Copy link
Collaborator

alexmt commented Apr 30, 2019

There's a fix. It is just not trivial :)

@alexmt alexmt self-assigned this Apr 30, 2019
@alexmt
Copy link
Collaborator

alexmt commented Apr 30, 2019

Trying to find simple way to improve it.

@alexmt
Copy link
Collaborator

alexmt commented Jun 5, 2019

Looks like it is broken again. See #1547 (comment)

@alexmt alexmt reopened this Jun 5, 2019
alexmt pushed a commit to alexmt/argo-cd that referenced this issue Jun 10, 2019
@alexmt
Copy link
Collaborator

alexmt commented Jun 10, 2019

Hello @KarstenSiemer , finally got back to this issue, but unfortunately cannot reproduce it. Implemented e2e test which creates self-referenced app and makes sure there is not reconciliation loop: #1724

PR deploys app which is pointing to itself. Can you please check if I'm missing something?

@alexec alexec removed this from the v1.1 milestone Jun 13, 2019
@KarstenSiemer
Copy link
Contributor Author

Nope, i have to reopen.
This was gone when i first created the app for self reconciliation. But the problem is back after some time:
argoreconcile
here the red line is actually two lines, one is the argo-cd self reconciling app the other one is the seeder application which is pulling applications from git and also pulls the argo-cd app

hence the green line being twice the amount of the red one

it feels less bad then before but definitely destroys argo-cd in a way of having the two apps extremely busy so that nothing gets rolled out any more

@KarstenSiemer KarstenSiemer reopened this Aug 9, 2019
@KarstenSiemer
Copy link
Contributor Author

KarstenSiemer commented Aug 16, 2019

We had the same issue when trying to have an ArgoCD application manage all application including the aforementioned application.
We had the same looped arrow in the UI and the same reconciliation loop.
Once we removed the recursive configuration, so we manage all other application declaratively, and the argocd configuration application manually, the reconciliation loop stopped:

image

He is correct, if the self reconciling app isn't being pulled by the seeding application, then there is no reconciliation loop.
Last time i created it by hand and merged it to git in seeding directory at about the same time, which is why the problem wasn't there at first but reappeared.

The last thing that is a problem with seeding applications is the heredity of application state and that argocd doesn't do another reconciliation as long as an app is in progressing state.

When a seeded application becomes progressing, the seeder will be in progressing state, too.
Our seeder now governs around 80 applications, there is one in progressing state almost always.
That is hindering updates for all applications.
Can this be prevented?
The selfhealing starting from 1.2 would definitely help with this i think. But they'd still remain in progressing

@alexec alexec added the M label Oct 4, 2019
@alexec alexec added this to the v1.4 milestone Oct 4, 2019
@v-gu
Copy link

v-gu commented Oct 31, 2019

some app use configmap map as leader-election helper, like cert-manager, nginx-ingress, this will trigger ArgoCD reconciliations, those configmap usually are un-managed resources.

@alexmt alexmt modified the milestones: v1.4, v1.5 Jan 23, 2020
@alexmt alexmt modified the milestones: v1.5, v1.6 Feb 18, 2020
@alexmt alexmt removed the M label May 13, 2020
@jannfis jannfis added the component:core Syncing, diffing, cluster state cache label May 14, 2020
@alexmt alexmt added component:application-sets Bulk application management related and removed component:core Syncing, diffing, cluster state cache labels May 15, 2020
@alexmt alexmt modified the milestones: v1.6 GitOps Engine, v1.7 May 27, 2020
@yann-soubeyrand
Copy link
Contributor

Hello,

We're hit by this issue too. We have app of apps (with the root of them managing itself), an app which manages Argo CD, and apps with leader election config maps or lock config maps. That is, we collection all the pattern which seem to lead to high reconciliation rate and CPU usage (Argo CD uses a full CPU)!

@alexmt alexmt modified the milestones: v1.7 , v1.8 Aug 25, 2020
@alexmt alexmt removed their assignment Sep 17, 2020
@alexmt alexmt modified the milestones: v1.8, v1.9 Oct 29, 2020
@jessesuen
Copy link
Member

This bug is quite old, since then there was an improvement that prevented unnecessary update of status.observedAt which was the primary contributor to this reconciliation loop. In other words, we believe this is fixed (at least in v1.8)

@Aenima4six2
Copy link

This is still happening to use in ArgoCD 2.0.4.

@agrevtcev
Copy link

Hello!
This is still happening to use in ArgoCD 2.0.4. We are managing argocd w/ argocd, and noticed unusual high CPU usage on argocd-application-controller pod (on version 2.0.3). So decided to try 2.0.4, and got the same result.

Br, Alexey

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working component:application-sets Bulk application management related
Projects
None yet
10 participants