Getting "No operation is in progress" when trying to Terminate sync

[teddy-wahle]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

The UI and the API say a sync is happening:

But when I try terminate, it gives me this error

Unable to terminate operation: Unable to terminate operation. No operation is in progress

This is happening across all our clusters. I see nothing in the logs

To Reproduce

All I do is try to Terminate a running sync. Same error through CI and UI.

I tried re-creating the app. I've tried deleting application controller pods. It doesn't help.

Expected behavior

I expect the sync get terminated

Screenshots

See above

Version

➜  argocd version
argocd: v2.9.3+6eba5be.dirty
  BuildDate: 2023-12-02T00:45:51Z
  GitCommit: 6eba5be864b7e031871ed7698f5233336dfe75c7
  GitTreeState: dirty
  GoVersion: go1.21.4
  Compiler: gc
  Platform: darwin/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
argocd-server: v2.9.5+f943664
  BuildDate: 2024-01-19T17:57:57Z
  GitCommit: f9436641a616d277ab1f98694e5ce4c986d4ea05
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.13.2+g2a2fb3b
  Kubectl Version: v0.24.17
  Jsonnet Version: v0.20.0

Logs

➜  argocd app get vkms
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
Name:               argocd/vkms
Project:            streaming
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://<argo-domain>/applications/vkms
Repo:               <argo-repo>
Target:             main
Path:              <argo-path>
SyncWindow:         Sync Allowed
Sync Policy:        Automated
Sync Status:        OutOfSync from main (51e98a1)
Health Status:      Healthy

CONDITION  MESSAGE                                                                                                                                 LAST TRANSITION
SyncError  Failed sync attempt to 51e98a1cd67fed6e1b954cfa3fbecf95c8909913: waiting for healthy state of autoscaling/HorizontalPodAutoscaler/vkms  2024-02-08 15:11:47 -0800 PST


GROUP        KIND                     NAMESPACE  NAME                 STATUS     HEALTH   HOOK  MESSAGE
             ConfigMap                default    vkms-config-3499602  OutOfSync                 ignored (requires pruning)
             ServiceAccount           default    vkms                 OutOfSync                 serviceaccount/vkms unchanged
             ConfigMap                default    vkms-config-de552ac  OutOfSync                 configmap/vkms-config-de552ac unchanged
apps         Deployment               default    vkms                 OutOfSync  Healthy        deployment.apps/vkms configured
argoproj.io  Rollout                  default    vkms                 OutOfSync  Healthy        rollout.argoproj.io/vkms unchanged
             Service                  default    vkms                 OutOfSync  Healthy        service/vkms unchanged
autoscaling  HorizontalPodAutoscaler  default    vkms                 OutOfSync  Healthy

➜  argocd app terminate-op vkms
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
FATA[0000] rpc error: code = InvalidArgument desc = Unable to terminate operation. No operation is in progress

[jgwest]

Can you provide the pod logs for server and application controller pods after reproducing the issue?

[teddy-wahle]

@jgwest Here are logs from the two application controller pods and the two server pods. To create these logs I deleted all server and app controller pods so that all the logs were fresh. Then I went to UI and tried to terminate a sync in an app called vconductor and got the same error:

argocd-application-controller-1.txt
argocd-application-controller-0.txt

argocd-server-55464bc74-zmhvr.txt
argocd-server-55464bc74-jw67s.txt

You can see this in the logs:

time="2024-02-12T18:02:33Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = Unable to terminate operation. No operation is in progress" grpc.code=InvalidArgument grpc.method=TerminateOperation grpc.service=application.ApplicationService grpc.start_time="2024-02-12T18:02:33Z" grpc.time_ms=8.452 span.kind=server system=grpc

[AnubhavSabarwal]

This might be because of sharding functionality which is known issue in 2.10.0 version.

PR is raised for this.
#17124

[morawat]

we are on 2.10.5 and still facing this issue.

[NPastorale]

I'm seeing this behaviour on v2.10.9+c071af8 too

[NPastorale]

It is still present on v2.11.0+d3f33c0

[morawat]

Is there a way to upvote and get some traction on this?

[jonra82]

This has been a problem since the inception of ArgoCD. When will we get a workaround ??

[dongbeiqing91]

same issue on v2.11.4+e1284e1
I often see this issue in such scenario:
A deployment is already in the cluster but we are onboarding it to ArgoCD for the first time. It has the annotation "argocd.argoproj.io/sync-options: Replace=true". I guess this "replace" behavior causes some bug, leading to sync error like "waiting for healthy state of apps/Deployment/..."

[Jack200062]

Hi Folks! Any new info on this one? Facing the same problem on v2.10.5+335875d version

[ramakrishnakommuri]

I came across similar issue on v2.11.5+c4b283c. Any workaround ?

[andreosipov]

have the same problem on v2.10.5

[nicl-dev]

same problem on v2.10.8+37b1cf5

[sandeep-ongit]

same issue on v2.11.10
Any update on this please?

[raidlman]

Same problem here for v2.13.0+347f221

[andrii-korotkov-verkada]

Can you check if the field operation is present on the top level of application resource when this happens, please? If it's not present, there must be some bug with keeping operation and status.operationState in sync.