podMetadata is not applied in a WorkflowTemplate executed by a Sensor with a templateRef

[jdinsel-xealth]

Describe the bug

podMetadata from a referenced WorkflowTemplate is not applied to the containers executed when a Sensor triggers a Workflow with a templateRef.

Most of the Sensor examples show how to create a Workflow that runs a single container. See this k8s resources example. It works great, and you can add the following to the spec (line 25 of the current example).

             podGC:
                  deleteDelaySeconds: 59s
                  strategy: OnWorkflowCompletion

The problem occurs when the Workflow resource has a template that includes a templateRef such as the example below.

            spec:
                entrypoint: my-example
                serviceAccountName: my-sa
                templates:
                - name: my-example
                  steps:
                  - - name: my-example
                      templateRef:
                        name: update-my-example
                        template: update-my-example

If the referenced template has a podMetadata applying annotations to containers, these are ignored.

podMetadata:
    annotations:
      my-example/xyz: true

To fix this, the annotation must be applied directly to the containers in the WorkflowTemplate being referenced. This is not true if the WorkflowTemplate is executed with argo submit.

To Reproduce

Steps to reproduce the behavior:

1. Create a Sensor with the a reference to a templateRef
2. Configure the templateRef with a container or two and configure the podMetadata to globally apply annotations to those containers
3. Cause the Sensor to be triggered
4. Watch the pods execute; they will not have the expected annotations

Expected behavior

A valid WorkflowTemplate, when referenced by a Sensor, should apply all of the podMetadata settings.

Environment (please complete the following information):

• Kubernetes: v1.30
• Argo Events: v2.4.7
• Argo Workflows: v0.41.1

Additional context

See the issue described here #2044 for the history of adding support for WorkflowTemplate's via TemplateRef and the pull request #2120

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.