"Configure plugin via sidecar" ⇒ child resources not pruned on deletion

[pmorch]

Describe the bug

The behavior differs between a plugin via Argo CD configmap and a plugin via sidecar if the application manifest has:

  syncPolicy:
    automated:
      prune: true

When the plugin is installed in the argocd-cm ConfigMap, then resources installed by the application are pruned when the application is deleted. If the plugin is in a sidecar, the application's resources are not pruned as expected when the application is removed.

Also, the logs from ArgoCD are much shorter when deleting the application that uses the sidecar plugin (see "Logs" below).

To Reproduce

In this gist, i've included all the files referenced below. Short version: I've created an ArgoCD installation with two plugins. One installed in the Argo CD configmap and (almost) the same plugin in a sidecar.

$ kubectl apply -f cmp-plugin.yaml 
configmap/cmp-plugin configured

$ helm upgrade -n argocd --install argocd --version 3.30.0 --values values.yaml argo/argo-cd
(helm output removed)

Now there are two application .yaml files, one for the configmap plugin (with a name) and one for the sidecar plugin (without a name).

diff -u app-cmp-plugin.yaml nameless-app-cmp-plugin.yaml
--- app-cmp-plugin.yaml	2022-01-20 23:13:16.811774500 +0100
+++ nameless-app-cmp-plugin.yaml	2022-01-20 23:13:16.811774500 +0100
@@ -14,7 +14,6 @@
     path: .
     targetRevision: HEAD
     plugin:
-      name: cmp-plugin
       env:
       - name: FOO
         value: myfoo

If I apply app-cmp-plugin.yaml, the plugin from the argocd-cm config map installs a test-cmd-plugin config map in the default namespace. When I delete app-cmp-plugin.yaml, the test-cmd-plugin config map is pruned along with the application as expected.

If I apply nameless-app-cmp-plugin.yaml, the plugin in the sidecar likewise installs a test-cmd-plugin config map in the default namespace. But when I delete nameless-app-cmp-plugin.yaml, the test-cmd-plugin config map is not pruned along with the application. I expected it to be pruned.

Expected behavior

I expect argocd-cm and sidecar plugins to behave identically and both should prune resources when specified to do so in the Application's declaration.

Version

$ argocd version
argocd: v2.1.7+a408e29.dirty
  BuildDate: 2021-11-19T05:20:05Z
  GitCommit: a408e299ffa743213df3aa9135bf7945644ec936
  GitTreeState: dirty
  GoVersion: go1.17.2
  Compiler: gc
  Platform: linux/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
argocd-server: v2.2.2+03b17e0
  BuildDate: 2022-01-01T06:27:52Z
  GitCommit: 03b17e0233e64787ffb5fcf65c740cc2a20822ba
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.22.2
  Jsonnet Version: v0.17.0

Logs

Logs are available in the gist and were created with kubetail for all pods in the argocd namespace. What I found most interesting was that the log for deleting the application for the argocd-cm configmap plugin, delete-with-name.log was 34 lines, but the one for deleting with the the sidecar plugin, delete-without-name.log, was 3 lines.

[crenshaw-dev]

     Foo: myfoo
     KubeVersion: "1.21"
-    in-sidecar: "true"
   creationTimestamp: "2022-02-02T18:07:43Z"
+  labels:
+    app.kubernetes.io/instance: test-cmd-plugin
   name: test-cmd-plugin
   namespace: default

The (partial) ConfigMap diff is interesting. I wonder if the problem is that the sidecar CMP is failing to apply the necessary label.

[crenshaw-dev]

I think I've found the fix. The most difficult part will be writing an e2e test for it. Should be done tomorrow.