controller panic: nil pointer dereference on DAG processedTmpl

[chengjoey]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

kubectl get workflow

NAME   STATUS   AGE   MESSAGE
a      Error    14m   runtime error: invalid memory address or nil pointer dereference

kubectl get workflow a -o yaml

# ...
    outputs: {}
status:
  artifactGCStatus:
    notSpecified: true
  artifactRepositoryRef:
    artifactRepository: {}
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  - status: "True"
    type: Completed
  finishedAt: "2024-09-02T08:21:01Z"
  message: 'runtime error: invalid memory address or nil pointer dereference'
  nodes:
# ...

Version(s)

latest(sha256:f61c2eaf06294f38dc66f7519ae2c3af916ce7dcde8322999789bd39c29e983c)

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: a
spec:
  templates:
    - name: echo2
      container:
        name: ''
        image: alpine:3.7
        command:
          - echo
          - abc
    - name: echo1
      inputs:
        parameters:
          - name: abc
            valueFrom:
              configMapKeyRef:
                name: abc
                key: abc
      container:
        name: ''
        image: alpine:3.7
        command:
          - echo
          - abc
    - name: diamond
      dag:
        tasks:
          - name: A
            template: echo1
            when: 1 == 0      
          - name: B
            template: echo2         
            dependencies:
              - A
  entrypoint: diamond

Logs from the workflow controller

time="2024-09-02T08:21:01.019Z" level=info msg="node changed" namespace=default new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=a-814491500 old.message= old.phase=Pending old.progress=0/1 workflow=a
time="2024-09-02T08:21:01.019Z" level=error msg="Mark error node" error="unable to retrieve inputs.parameters.abc from ConfigMap: ConfigMap 'abc' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=default nodeName=a.A workflow=a
time="2024-09-02T08:21:01.019Z" level=error msg="node is already fulfilled" fromPhase=Skipped namespace=default nodeName=a.A toPhase=Error workflow=a
time="2024-09-02T08:21:01.019Z" level=info msg="node a-864824357 phase Skipped -> Error" namespace=default workflow=a
time="2024-09-02T08:21:01.019Z" level=info msg="node a-864824357 message: unable to retrieve inputs.parameters.abc from ConfigMap: ConfigMap 'abc' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=default workflow=a
time="2024-09-02T08:21:01.020Z" level=error msg="Recovered from panic" namespace=default r="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 268 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x64\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2()\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:196 +0x88\npanic({0x1dc2200?, 0x390cab0?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x124\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAGTask(0x40006f40c0, {0x2566fe0, 0x39ce820}, 0x400031e770, {0x40008ba421, 0x1})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:453 +0xeb8\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAGTask(0x40006f40c0, {0x2566fe0, 0x39ce820}, 0x400031e770, {0x3897ef0, 0x1})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:517 +0x2600\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAG(0x40006f40c0, {0x2566fe0, 0x39ce820}, {0x3897fe8, 0x1}, 0x400087e2c0, {0x4000a8e370, 0x7}, 0x4000cefd48, {0x256d550, ...}, ...)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:269 +0x31c\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeTemplate(0x40006f40c0, {0x2566fe0, 0x39ce820}, {0x3897fe8, 0x1}, {0x256d550, 0x4000972000}, 0x400087e240, {{0x0, 0x0, ...}, ...}, ...)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2207 +0x2594\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0x40006f40c0, {0x2566fe0, 0x39ce820})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:372 +0x1744\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0x4000471608, {0x2566fe0, 0x39ce820})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:888 +0x57c\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0x4000471608)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:803 +0x88\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.15/pkg/util/wait/wait.go:157 +0x40\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x40009614e0, {0x253b0e0, 0x40008d6150}, 0x1, 0x4000563b60)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.15/pkg/util/wait/wait.go:158 +0x90\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0x40009614e0, 0x3b9aca00, 0x0, 0x1, 0x4000563b60)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.15/pkg/util/wait/wait.go:135 +0x80\nk8s.io/apimachinery/pkg/util/wait.Until(...)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.15/pkg/util/wait/wait.go:92\ncreated by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).Run in goroutine 56\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:367 +0x128c\n" workflow=a
time="2024-09-02T08:21:01.020Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=a
time="2024-09-02T08:21:01.020Z" level=info msg="Updated message  -> runtime error: invalid memory address or nil pointer dereference" namespace=default workflow=a
time="2024-09-02T08:21:01.020Z" level=info msg="Marking workflow completed" namespace=default workflow=a
time="2024-09-02T08:21:01.020Z" level=info msg="Workflow to be dehydrated" Workflow Size=2359

Logs from in your workflow's wait container

None

[chengjoey]

Recovered from panic stack:

goroutine 268 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x64
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2()
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:196 +0x88
panic({0x1dc2200?, 0x390cab0?})
	/usr/local/go/src/runtime/panic.go:770 +0x124
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAGTask(0x40006f40c0, {0x2566fe0, 0x39ce820}, 0x400031e770, {0x40008ba421, 0x1})
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:453 +0xeb8
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAGTask(0x40006f40c0, {0x2566fe0, 0x39ce820}, 0x400031e770, {0x3897ef0, 0x1})
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:517 +0x2600
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAG(0x40006f40c0, {0x2566fe0, 0x39ce820}, {0x3897fe8, 0x1}, 0x400087e2c0, {0x4000a8e370, 0x7}, 0x4000cefd48, {0x256d550, ...}, ...)
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:269 +0x31c
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeTemplate(0x40006f40c0, {0x2566fe0, 0x39ce820}, {0x3897fe8, 0x1}, {0x256d550, 0x4000972000}, 0x400087e240, {{0x0, 0x0, ...}, ...}, ...)
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2207 +0x2594
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0x40006f40c0, {0x2566fe0, 0x39ce820})
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:372 +0x1744
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0x4000471608, {0x2566fe0, 0x39ce820})
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:888 +0x57c
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0x4000471608)
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:803 +0x88
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.15/pkg/util/wait/wait.go:157 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x40009614e0, {0x253b0e0, 0x40008d6150}, 0x1, 0x4000563b60)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.15/pkg/util/wait/wait.go:158 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x40009614e0, 0x3b9aca00, 0x0, 0x1, 0x4000563b60)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.15/pkg/util/wait/wait.go:135 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.15/pkg/util/wait/wait.go:92
created by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).Run in goroutine 56
	/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:367 +0x128c

cause panic:

argo-workflows/workflow/controller/dag.go

Lines 452 to 454 in 8a67009

	if tmpl != nil {
	woc.controller.syncManager.Release(woc.wf, node.ID, processedTmpl.Synchronization)
	}