Workflow of workflow containing artifacts cannot be deleted after completion due to finalizer

[Ezzahhh]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When implementing a workflow of workflows pattern and the child workflow uses artifacts, I was not able to delete the jobs after they are complete. Usually able to delete the child pod (the artifact GC finalizer is removed for it) but the parent seems to never have its finalizer removed so it cannot be deleted unless with manual intervention.
Removal of the artifact output from the child wf step resolves this issue.

Version(s)

v3.5.10 and index digest on container f61c2eaf06294f38dc66f7519ae2c3af916ce7dcde8322999789bd39c29e983c

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: bug-test
  namespace: argo-workflows
spec:
  entrypoint: main
  templates:
    - name: main
      outputs:
        parameters:
        - name: customers
          valueFrom:
            path: /tmp/customers.json
      script:
        image: python:3.9-bookworm
        command:
          - python
        source: |
          import json
          list_1 = ["test"]
          with open("/tmp/customers.json", "w") as f:
              json.dump(list_1, f)
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: bug-test-start
  namespace: argo-workflows
spec:
  entrypoint: main
  templates:
  - name: main
    dag:
      tasks:
      - name: bug-test
        templateRef:
          name: bug-test
          template: main
      - name: child-wfs
        depends: bug-test
        template: child-wfs
        arguments:
          parameters:
          - name: customer-oops-path
            value: "{{item}}"
        withParam: "{{tasks.bug-test.outputs.parameters.customers}}"
  - name: child-wfs
    inputs:
      parameters:
        - name: customer-oops-path
    resource:
      action: create
      manifest: |
        apiVersion: argoproj.io/v1alpha1
        kind: Workflow
        metadata:
          generateName: child-wf-
          namespace: argo-workflows
        spec:
          entrypoint: child-main
          artifactGC:
            strategy: OnWorkflowDeletion
          arguments:
            parameters:
            - name: customer-oops-path
              value: "{{inputs.parameters.customer-oops-path}}"
          workflowTemplateRef:
            name: bug-test-start
      successCondition: status.phase == Succeeded
      failureCondition: status.phase in (Failed, Error)

  - name: child-main
    inputs:
      parameters:
        - name: customer-oops-path
    dag:
      tasks:
        - name: get-kops-and-customer-info
          template: get-customer-data
          arguments:
            parameters:
            - name: customer-oops-path
              value: "{{inputs.parameters.customer-oops-path}}"
  - name: get-customer-data
    inputs:
      parameters:
        - name: customer-oops-path
    outputs:
      artifacts:
        - name: bug-test
          path: /tmp/result.json
          s3:
            key: '{{workflow.name}}.txt.tar.gz'
          artifactGC:
            strategy: OnWorkflowDeletion
    script:
      image: python:3.9-bookworm
      command: [python]
      source: |
        import json
        list_1 = []
        list_1.append("{{inputs.parameters.customer-oops-path}}")
        print("{{inputs.parameters.customer-oops-path}}")
        with open("/tmp/result.json", "w") as f:
            json.dump(list_1, f)

Logs from the workflow controller

time="2024-09-05T00:55:39.801Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=779822173 namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.801Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=1 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.806Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=bug-test-start-s4gnp-3496475705 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.808Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=bug-test-start-s4gnp-3496475705 old.message= old.phase=Pending old.progress=0/1 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.809Z" level=warning msg="was unable to obtain the node for bug-test-start-s4gnp-1844965441, taskName child-wfs"
time="2024-09-05T00:55:39.811Z" level=warning msg="was unable to obtain the node for bug-test-start-s4gnp-1844965441, taskName child-wfs"
time="2024-09-05T00:55:39.812Z" level=info msg="TaskGroup node bug-test-start-s4gnp-1844965441 initialized Running (message: )" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.813Z" level=warning msg="was unable to obtain the node for bug-test-start-s4gnp-2759189168, taskName child-wfs(0:test)"
time="2024-09-05T00:55:39.814Z" level=info msg="All of node bug-test-start-s4gnp.child-wfs(0:test) dependencies [bug-test] completed" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.814Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.823Z" level=info msg="argo-workflows/ConfigMap/krr-semaphore/semaphore acquired by argo-workflows/bug-test-start-s4gnp/bug-test-start-s4gnp-2759189168. Lock availability: 9/10" semaphore=argo-workflows/ConfigMap/krr-semaphore/semaphore
time="2024-09-05T00:55:39.823Z" level=info msg="Node bug-test-start-s4gnp.child-wfs(0:test) acquired synchronization lock" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.823Z" level=info msg="Pod node bug-test-start-s4gnp-2759189168 initialized Pending" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.843Z" level=info msg="Created pod: bug-test-start-s4gnp.child-wfs(0:test) (bug-test-start-s4gnp-child-wfs-2759189168)" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.843Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.843Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.843Z" level=info msg="Workflow to be dehydrated" Workflow Size=5916
time="2024-09-05T00:55:39.875Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=779822313 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:39.883Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/bug-test-start-s4gnp-main-3496475705/labelPodCompleted
time="2024-09-05T00:55:44.899Z" level=info msg="Processing workflow" Phase= ResourceVersion=779822385 namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.047Z" level=info msg="resolved artifact repository" artifactRepositoryRef="argo-workflows/#"
time="2024-09-05T00:55:45.047Z" level=info msg="adding artifact GC finalizer" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.047Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.047Z" level=info msg="Updated phase  -> Running" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.047Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.047Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.047Z" level=info msg="DAG node child-wf-bdvxw initialized Running" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.047Z" level=warning msg="was unable to obtain the node for child-wf-bdvxw-1127732427, taskName get-kops-and-customer-info"
time="2024-09-05T00:55:45.047Z" level=warning msg="was unable to obtain the node for child-wf-bdvxw-1127732427, taskName get-kops-and-customer-info"
time="2024-09-05T00:55:45.048Z" level=info msg="All of node child-wf-bdvxw.get-kops-and-customer-info dependencies [] completed" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.048Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.048Z" level=info msg="Pod node child-wf-bdvxw-1127732427 initialized Pending" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.096Z" level=info msg="Created pod: child-wf-bdvxw.get-kops-and-customer-info (child-wf-bdvxw-get-customer-data-1127732427)" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.100Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.100Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:45.101Z" level=info msg="Workflow to be dehydrated" Workflow Size=4688
time="2024-09-05T00:55:45.119Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=779822391 workflow=child-wf-bdvxw
time="2024-09-05T00:55:49.836Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=779822313 namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:49.836Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=2 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:49.837Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=bug-test-start-s4gnp-2759189168 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:49.837Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=bug-test-start-s4gnp-3496475705 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:49.837Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Running new.progress=0/1 nodeID=bug-test-start-s4gnp-2759189168 old.message= old.phase=Pending old.progress=0/1 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:49.878Z" level=info msg="Node bug-test-start-s4gnp.child-wfs(0:test) acquired synchronization lock" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:49.878Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:49.878Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:49.878Z" level=info msg="Workflow to be dehydrated" Workflow Size=5982
time="2024-09-05T00:55:49.907Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=779822454 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:55.101Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=779822391 namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.101Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=1 workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.101Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=child-wf-bdvxw-1127732427 workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.101Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=child-wf-bdvxw-1127732427 old.message= old.phase=Pending old.progress=0/1 workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.102Z" level=info msg="Outbound nodes of child-wf-bdvxw set to [child-wf-bdvxw-1127732427]" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.102Z" level=info msg="node child-wf-bdvxw phase Running -> Succeeded" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.102Z" level=info msg="node child-wf-bdvxw finished: 2024-09-05 00:55:55.102706952 +0000 UTC" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.102Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.102Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.102Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.102Z" level=info msg="Marking workflow completed" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.102Z" level=info msg="Workflow to be dehydrated" Workflow Size=5381
time="2024-09-05T00:55:55.249Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=779822527 workflow=child-wf-bdvxw
time="2024-09-05T00:55:55.341Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/child-wf-bdvxw-get-customer-data-1127732427/labelPodCompleted
time="2024-09-05T00:55:59.907Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=779822454 namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:59.908Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=2 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:59.908Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=bug-test-start-s4gnp-3496475705 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:59.908Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=bug-test-start-s4gnp-2759189168 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:59.908Z" level=info msg="node unchanged" namespace=argo-workflows nodeID=bug-test-start-s4gnp-2759189168 workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:59.941Z" level=info msg="Node bug-test-start-s4gnp.child-wfs(0:test) acquired synchronization lock" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:59.941Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:59.941Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:55:59.941Z" level=info msg="Workflow to be dehydrated" Workflow Size=5983
time="2024-09-05T00:55:59.960Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=779822596 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:00.332Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=779822527 namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:56:00.333Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=child-wf-bdvxw
time="2024-09-05T00:56:00.333Z" level=info msg="Workflow to be dehydrated" Workflow Size=5407
time="2024-09-05T00:56:00.385Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=779822602 workflow=child-wf-bdvxw
time="2024-09-05T00:56:05.399Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=779822602 namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:56:05.399Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=child-wf-bdvxw
time="2024-09-05T00:56:09.961Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=779822596 namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.961Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=2 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.961Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=bug-test-start-s4gnp-3496475705 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.961Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=bug-test-start-s4gnp-2759189168 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.961Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=bug-test-start-s4gnp-2759189168 old.message= old.phase=Running old.progress=0/1 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="Lock has been released by argo-workflows/bug-test-start-s4gnp/bug-test-start-s4gnp-2759189168. Available locks: 10" semaphore=argo-workflows/ConfigMap/krr-semaphore/semaphore
time="2024-09-05T00:56:09.962Z" level=info msg="node bug-test-start-s4gnp-1844965441 phase Running -> Succeeded" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="node bug-test-start-s4gnp-1844965441 finished: 2024-09-05 00:56:09.962537905 +0000 UTC" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="Outbound nodes of bug-test-start-s4gnp set to [bug-test-start-s4gnp-2759189168]" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="node bug-test-start-s4gnp phase Running -> Succeeded" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="node bug-test-start-s4gnp finished: 2024-09-05 00:56:09.962667964 +0000 UTC" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="Marking workflow completed" namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.962Z" level=info msg="Workflow to be dehydrated" Workflow Size=6120
time="2024-09-05T00:56:09.963Z" level=info msg="Released all acquired locks" key=bug-test-start-s4gnp namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:09.983Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=779822723 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:10.047Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/bug-test-start-s4gnp-child-wfs-2759189168/labelPodCompleted
time="2024-09-05T00:56:15.044Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=779822723 namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:15.044Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:15.044Z" level=info msg="Workflow to be dehydrated" Workflow Size=6090
time="2024-09-05T00:56:15.053Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=779822800 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:20.064Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=779822800 namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:20.065Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:41.222Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=779822994 namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:56:41.222Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=child-wf-bdvxw
time="2024-09-05T00:56:41.222Z" level=info msg="Creating Artifact GC Task child-wf-bdvxw-artgc-wfdel-1685253302-0" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:56:41.225Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=779822995 namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:41.225Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:41.226Z" level=info msg="Workflow to be dehydrated" Workflow Size=6126
time="2024-09-05T00:56:41.239Z" level=info msg="creating pod to delete artifacts: child-wf-bdvxw-artgc-wfdel-1685253302" namespace=argo-workflows strategy=OnWorkflowDeletion workflow=child-wf-bdvxw
time="2024-09-05T00:56:41.280Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=779823121 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:41.528Z" level=info msg="Workflow to be dehydrated" Workflow Size=5443
time="2024-09-05T00:56:41.725Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=779823131 workflow=child-wf-bdvxw
time="2024-09-05T00:56:46.287Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=779823121 namespace=argo-workflows workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:46.288Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=bug-test-start-s4gnp
time="2024-09-05T00:56:46.782Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=779823131 namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:56:46.783Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=child-wf-bdvxw
time="2024-09-05T00:56:46.783Z" level=info msg="reconciling artifact-gc pod" message= namespace=argo-workflows phase=Succeeded pod=child-wf-bdvxw-artgc-wfdel-1685253302 workflow=child-wf-bdvxw
time="2024-09-05T00:56:46.783Z" level=info msg="processing completed Artifact GC Pod \"child-wf-bdvxw-artgc-wfdel-1685253302\"" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:56:46.803Z" level=info msg="no remaining artifacts to GC, removing artifact GC finalizer (forceFinalizerRemoval=false)" namespace=argo-workflows workflow=child-wf-bdvxw
time="2024-09-05T00:56:46.803Z" level=info msg="Workflow to be dehydrated" Workflow Size=5451
time="2024-09-05T00:56:46.827Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=779823131 workflow=child-wf-bdvxw
time="2024-09-05T00:56:46.938Z" level=error msg="Failed to enable pod for deletion" error="an empty namespace may not be set when a resource name is provided"
time="2024-09-05T00:56:46.943Z" level=error msg="Failed to enable pod for deletion" error="an empty namespace may not be set when a resource name is provided"

Logs from in your workflow's wait container

time="2024-09-05T00:55:33.049Z" level=info msg="Starting Workflow Executor" version=v3.5.10                                                                                                                                                                                                                                                             
time="2024-09-05T00:55:33.087Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5                                                                                                                                                                                                                                
time="2024-09-05T00:55:33.087Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-workflows podName=bug-test-start-s4gnp-main-3496475705 templateName=main version="&Version{Version:v3.5.10,BuildDate:2024-08-01T05:11:25Z,GitCommit:25829927431d9a0f46d17b72ae74aedb8d700884,Git
time="2024-09-05T00:55:33.266Z" level=info msg="Starting deadline monitor"                                                                                                                                                                                                                                                                              
time="2024-09-05T00:55:35.266Z" level=info msg="Main container completed" error="<nil>"                                                                                                                                                                                                                                                                 
time="2024-09-05T00:55:35.266Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"                                                                                                                                                                                                                                
time="2024-09-05T00:55:35.266Z" level=info msg="Saving output parameters"                                                                                                                                                                                                                                                                               
time="2024-09-05T00:55:35.266Z" level=info msg="Saving path output parameter: customers"                                                                                                                                                                                                                                                                
time="2024-09-05T00:55:35.266Z" level=info msg="Copying /tmp/customers.json from base image layer"                                                                                                                                                                                                                                                      
time="2024-09-05T00:55:35.266Z" level=info msg="Successfully saved output parameter: customers"                                                                                                                                                                                                                                                         
time="2024-09-05T00:55:35.267Z" level=info msg="No output artifacts"                                                                                                                                                                                                                                                                                    
time="2024-09-05T00:55:35.297Z" level=info msg="Alloc=7486 TotalAlloc=13535 Sys=23397 NumGC=4 Goroutines=8"                                                                                                                                                                                                                                             
time="2024-09-05T00:55:35.313Z" level=info msg="Deadline monitor stopped"

[chengjoey]

time="2024-09-05T00:55:35.267Z" level=info msg="No output artifacts"

It seems no output artifacts. This may have been fixed by #13500. If you use the latest controller, has this issue been resolved?

[Ezzahhh]

time="2024-09-05T00:55:35.267Z" level=info msg="No output artifacts"

It seems no output artifacts. This may have been fixed by #13500. If you use the latest controller, has this issue been resolved?

This was also tested on latest tag (index digest f61c2eaf06294f38dc66f7519ae2c3af916ce7dcde8322999789bd39c29e983c) and the finalizer is still not automatically removed

Just a note to others that encounter this. The latest docker tag does not point to main. I had to use latest-linux-amd64. Is this intended? It could be a CI deployment error somewhere. In any case, I believe #13500 should have resolved this but I was testing on the wrong docker tag so will be closing.

Thanks for the help @chengjoey!

[Ezzahhh]

Looks like Actions failing https://github.com/argoproj/argo-workflows/actions/runs/10715657770/job/29711546100#step:5:515 due to windows build that's halting the entire job.

[agilgur5]

@Ezzahhh thanks for analyzing that! I had seen that the Windows build was failing but didn't realize it was causing the combined manifest to fail to run too.

I've filed an issue to get that fixed: #13562

[agilgur5]

Also since this is indeed resolved on main, marking this as a duplicate of #13499