Linter fails on templateRef for WorkflowTemplate

[bmaggi]

Summary

I got a "template reference not found" when using the linter on the following example.
I was expecting a valid workflowTemplate, so I suspect an error in the linter.

Diagnostics

What Kubernetes provider are you using?
Version 1.16.15 Self Hosted

What version of Argo Workflows are you running?
2.12.9

Example

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate # used to be Workflow
metadata:
  generateName: workflow-template-hello-world-
spec:
  entrypoint: whalesay
  templates:
    - name: whalesay
      steps:                              # You should only reference external "templates" in a "steps" or "dag" "template".
        - - name: call-whalesay-template
            templateRef:                  # You can reference a "template" from another "WorkflowTemplate" using this field
              name: workflow-template-1   # This is the name of the "WorkflowTemplate" CRD that contains the "template" you want
              template: whalesay-template # This is the name of the "template" you want to reference
            arguments:                    # You can pass in arguments as normal
              parameters:
                - name: message
                  value: "hello world"

---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-1
spec:
  entrypoint: whalesay-template     # Fields other than "arguments" and "templates" not supported in v2.4 - v2.6
  arguments:
    parameters:
      - name: message
        value: hello world
  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]

Result in the command line :

λ argo-windows-amd64-2.12.9.exe -v
time="2021-02-23T15:16:34.878Z" level=debug msg="CLI version" version="{v2.12.9 2021-02-16T22:46:55Z 737905345d70ba1ebd566ce1230e4f971993dfd0 v2.12.9 clean go1.13 gc windows/amd64}"

λ argo-windows-amd64-2.12.9.exe template lint ref-example.yaml
time="2021-02-23T15:40:54.449Z" level=error msg="templates.whalesay.steps[0].call-whalesay-template template reference workflow-template-1.whalesay-template not found"
time="2021-02-23T15:40:54.492Z" level=fatal msg="Errors encountered in validation"

Notes :

• This was originally a dicussion
• I transformed one of the official example from kind Workflow to kind : WorkflowTemplate
  (Should be correct according to Workflow template Spec)
• I checked the discussions around "templateRef" and from my understanding the use cas inside a step is still valid.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[simster7]

Is your template on the cluster? That checks verifies that the reference workflow template is accessible from the cluster, not just the file. If not, can you try adding the workflow template to the cluster first (using argo template create), then running the lint again?

[bmaggi]

Here are the steps

λ argo-windows-amd64-2.12.9.exe template create ref-example.yaml
2021/02/23 18:33:28 Failed to create workflow template: templates.whalesay.steps[0].call-whalesay-template template reference workflow-template-1.whalesay-template not found

=> error when creating the template whith a single file

• Spliting the file in 2 (part1 with workflow-template-1 and part2 with workflow-template-hello-world-)

argo-windows-amd64-2.12.9.exe template create ref-example-part1.yaml
Name:                workflow-template-1
Namespace:           mine
Created:             Tue Feb 23 18:35:20 +0100 (now)

=> creation ok

λ argo-windows-amd64-2.12.9.exe template lint ref-example.yaml
WorkflowTemplate manifests validated
WorkflowTemplate manifests validated

=> linter ok

So your guess is right the linter seems failing due to the missing template in the cluster.

So what is this the correct workflow for deployment ?

• using the linter on template 1
• deploy the template 1
• using the linter on template 2
• deploy the template 2

I was expecting to be able to lint everything before deployment and deploying everything on the cluster in one go.

[simster7]

I was expecting to be able to lint everything before deployment and deploying everything on the cluster in one go.

Yes, I agree that that is desired behavior. I'll work on a fix

[simster7]

@bmaggi Another workaround: you don't actually need two files, you can do it on the same file as long as the WorkflowTemplate that is being referenced goes before (i.e. on top) of the WorkflowTemplate that is doing the referencing

[bmaggi]

Thanks for your answer, maybe I missed something but I still have the linter error with

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: github-issue-example-lib
spec:
  entrypoint: whalesay-template     # Fields other than "arguments" and "templates" not supported in v2.4 - v2.6
  arguments:
    parameters:
      - name: message
        value: hello world
  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate # used to be Workflow
metadata:
  generateName: github-issue-example-main
spec:
  entrypoint: whalesay
  templates:
    - name: whalesay
      steps:                              # You should only reference external "templates" in a "steps" or "dag" "template".
        - - name: call-whalesay-template
            templateRef:                  # You can reference a "template" from another "WorkflowTemplate" using this field
              name: github-issue-example-lib  # This is the name of the "WorkflowTemplate" CRD that contains the "template" you want
              template: whalesay-template # This is the name of the "template" you want to reference
            arguments:                    # You can pass in arguments as normal
              parameters:
                - name: message
                  value: "hello world"

λ argo-windows-amd64-2.12.9.exe template lint github-example.yaml
time="2021-02-25T17:04:44.704Z" level=error msg="templates.whalesay.steps[0].call-whalesay-template template reference github-issue-example-lib.whalesay-template not found"
time="2021-02-25T17:04:44.744Z" level=fatal msg="Errors encountered in validation"

[simster7]

Low priority, but #5198 stands as a PoC to fix this issue

[msmerc]

@simster7 - is this still considered a low priority issue / wontfix?

It's a showstopper for us since we like to do our tests vs a clean environment.

Would you consider a partial fix whereby so long as the template files are specified in order such that no template is referred to before it is defined then it would pass? That would be immensely helpful!

[tvalasek]

broken argo lint makes troubleshooting argo workflows really really hard for us. It is like looking for needle in haystack. Would it be possible to bump up the priority? Thanks

[sarabala1979]

@tvalasek I am trying to understand your use-case and how this issue will be a showstopper. If you split templates into separate files it will work what you expect like lint and create each resource. I understand keep everything in one file is kind of convenient and maintenance-free. But currently, our team has under-resource to work on this kind of issue. But we will try to prioritize but you or @simster7 like to submit the PR, we are happy to review it.
argo template create will do lint and create each resource as you expect.

$ argo template create wft1.yaml 
Name:                github-issue-example-lib
Namespace:           argo
Created:             Tue Jul 13 23:49:47 -0700 (now)
Name:                github-issue-example-mainv2gqh
Namespace:           argo
Created:             Tue Jul 13 23:49:47 -0700 (now)

[msmerc]

@sarabala1979 I've found the opposite. If I glue the scripts into a single file in an appropriate order it works ok [workaround], but if you split the files it never works because the templates are never defined. I image if I were to do it like so:

1. Lint templates
2. Submit templates
3. Lint scripts

It would work - but that complicates our CI!

@tvalasek as an ugly workaround, we rename all our templates to start with _ and then do the following:

# Use awk to glue all the ago files into a single yaml file to work around argo bug
awk 'FNR==1 && NR!=1 {print "----"}{print}' argo/* > combined.yaml
argo lint --all-kinds combined.yaml

I'm not proud!

[tvalasek]

@tvalasek as an ugly workaround, we rename all our templates to start with _ and then do the following:

# Use awk to glue all the ago files into a single yaml file to work around argo bug
awk 'FNR==1 && NR!=1 {print "----"}{print}' argo/* > combined.yaml
argo lint --all-kinds combined.yaml

I'm not proud!

Hi. I tried your workaround but it does not work for me in v3.1.12. We use following reference chain: Workflow (wrapper) -> WorkflowTemplate (logical groups) -> WorkflowTemplate (pod specs). This article explains our use case: https://tomas-valasek.medium.com/building-kubernetes-clusters-using-kubernetes-dde332c77d65

[vatine]

I read "If you split templates into separate files it will work what you expect like lint and create each resource" as "lint one file, if OK, push it to the cluster".

This would be inconvenient with a VCS-based flow that want to ensure that workflows (and WorkflowTemplates) are well-formed at the PR stage. I suspect it would require loading all referenced files before the linting starts, and use internally-stored Workflows and WorkflowTemplates before the linting starts, so as it avoid ordering issues between them.

[SamOh]

+1 to all of the above.

We're trying to use the argo linting in our CI pipeline to catch errors in the manifests before we apply resources to our cluster. The linting fails for any references to workflow templates even if they are defined within the same file. This isn't actually an error and shouldn't be caught by the linting tool as one, given that the template will exist if it were applied above.

It would be very helpful if we could get a fix in that allows the linting to check above in the file for resources and pass if they exist so that references to them won't error out. Or, at the very least switch these errors to warnings instead of errors.

[apapia]

I'm running into the same issue but the workaround of combining them into one yaml in dependency order doesn't work for me. In this case, they are all kind: WorkflowTemplate. It still attempts to connect to kubernetes to retrieve the other templates.

[joebowbeer]

This seems like a big problem.

I'd like to see a mode that did not communicate with the cluster.

However, I did discover that if I added the namespace where the interdependent wftmpls are deployed, then the lint passed!

argo lint myfolder -n mynamespace

[artemklevtsov]

I consider to up the minikube in the CI/CD to apply templates and lint it.
Minikube CI/CD examples here: https://github.com/minikube-ci/examples

[bijenkins]

This is still a large issue in 2024. argo lint should be linting against static files. Combining multiple files into one, results in the same behavior still of not being able to find WorkflowTemplate resources. It is not documented anywhere how linter actually works, one would expect every example in this project to pass a lint with minimal effort.

If these WorkflowTemplates don't exist in the cluster you are linting against, it will fail. Even if it's namespace in the resources to signify they indeed are all on the same

For example the workflow-template example in this repository, with the independent resources put in their dependent order in ONE file:

# WorkflowTemplate for printing a message
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-print-message
spec:
  entrypoint: print-message
  templates:
  - container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - echo
      image: busybox
    inputs:
      parameters:
      - name: message
    name: print-message
---
# WorkflowTemplate for random fail template
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-random-fail-template
spec:
  templates:
  - container:
      args:
      - import random; import sys; exit_code = random.choice([0, 1, 1]); sys.exit(exit_code)
      command:
      - python
      - -c
      image: python:alpine3.6
    name: random-fail-template
    retryStrategy:
      limit: 10
---
# WorkflowTemplate for inner DAG
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-inner-dag
spec:
  templates:
  - container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - echo
      image: busybox
    inputs:
      parameters:
      - name: message
    name: print-message
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: message
            value: inner-A
        name: inner-A
        templateRef:
          name: workflow-template-print-message
          template: print-message
      - arguments:
          parameters:
          - name: message
            value: inner-B
        depends: inner-A
        name: inner-B
        template: print-message
      - arguments:
          parameters:
          - name: message
            value: inner-C
        depends: inner-A
        name: inner-C
        template: print-message
      - arguments:
          parameters:
          - name: message
            value: inner-D
        depends: inner-B && inner-C
        name: inner-D
        templateRef:
          name: workflow-template-print-message
          template: print-message
    name: inner-diamond
---
# WorkflowTemplate for inner steps
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-inner-steps
spec:
  templates:
  - container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - echo
      image: busybox
    inputs:
      parameters:
      - name: message
    name: print-message
  - name: inner-steps
    steps:
    - - arguments:
          parameters:
          - name: message
            value: inner-hello1
        name: inner-hello1
        templateRef:
          name: workflow-template-print-message
          template: print-message
    - - arguments:
          parameters:
          - name: message
            value: inner-hello2a
        name: inner-hello2a
        templateRef:
          name: workflow-template-print-message
          template: print-message
      - arguments:
          parameters:
          - name: message
            value: inner-hello2b
        name: inner-hello2b
        template: print-message
---
# WorkflowTemplate for submittable workflow
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-submittable
spec:
  arguments:
    parameters:
    - name: message
      value: hello world
  entrypoint: print-message
  templates:
  - container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - echo
      image: busybox
    inputs:
      parameters:
      - name: message
    name: print-message
---
# Now the Workflows that use the templates
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: archive-location
spec:
  archiveLogs: true
  entrypoint: hello-world
  templates:
  - container:
      args:
      - hello world
      command:
      - echo
      image: busybox
    name: hello-world
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: workflow-template-dag-diamond
spec:
  entrypoint: diamond
  templates:
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: message
            value: A
        name: A
        templateRef:
          name: workflow-template-print-message
          template: print-message
      - arguments:
          parameters:
          - name: message
            value: B
        depends: A
        name: B
        templateRef:
          name: workflow-template-print-message
          template: print-message
      - depends: A
        name: C
        templateRef:
          name: workflow-template-inner-dag
          template: inner-diamond
      - arguments:
          parameters:
          - name: message
            value: D
        depends: B && C
        name: D
        templateRef:
          name: workflow-template-print-message
          template: print-message
    name: diamond
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: workflow-template-hello-world
spec:
  entrypoint: hello-world-from-templateRef
  templates:
  - name: hello-world-from-templateRef
    steps:
    - - arguments:
          parameters:
          - name: message
            value: hello world
        name: call-print-message
        templateRef:
          name: workflow-template-print-message
          template: print-message
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: workflow-template-hello-world-1
spec:
  arguments:
    parameters:
    - name: message
      value: hello world
  entrypoint: print-message
  workflowTemplateRef:
    name: workflow-template-print-message
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: workflow-template-retry-with-steps
spec:
  entrypoint: retry-with-steps
  templates:
  - name: retry-with-steps
    steps:
    - - name: hello1
        templateRef:
          name: workflow-template-random-fail-template
          template: random-fail-template
    - - name: hello2a
        templateRef:
          name: workflow-template-random-fail-template
          template: random-fail-template
      - name: hello2b
        templateRef:
          name: workflow-template-random-fail-template
          template: random-fail-template
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: workflow-template-steps
spec:
  entrypoint: hello-hello-hello
  templates:
  - name: hello-hello-hello
    steps:
    - - arguments:
          parameters:
          - name: message
            value: hello1
        name: hello1
        templateRef:
          name: workflow-template-print-message
          template: print-message
    - - arguments:
          parameters:
          - name: message
            value: hello2a
        name: hello2a
        templateRef:
          name: workflow-template-inner-steps
          template: inner-steps
      - arguments:
          parameters:
          - name: message
            value: hello2b
        name: hello2b
        templateRef:
          name: workflow-template-print-message
          template: print-message

Results in:

argo lint single_ordered.yaml                                                                                                                                                                   130 ↵ ──(Tue,Sep10)─┘
single_ordered.yaml:
   ✖ in "workflow-template-inner-dag" (WorkflowTemplate): rpc error: code = InvalidArgument desc = templates.inner-diamond templates.inner-diamond.tasks.inner-A template reference workflow-template-print-message.print-message not found
   ✖ in "workflow-template-inner-steps" (WorkflowTemplate): rpc error: code = InvalidArgument desc = templates.inner-steps templates.inner-steps.steps[0].inner-hello1 template reference workflow-template-print-message.print-message not found
   ✖ in "workflow-template-dag-diamond" (Workflow): templates.diamond.tasks.A template reference workflow-template-print-message.print-message not found
   ✖ in "workflow-template-hello-world" (Workflow): templates.hello-world-from-templateRef.steps[0].call-print-message template reference workflow-template-print-message.print-message not found
   ✖ in "workflow-template-hello-world-1" (Workflow): rpc error: code = NotFound desc = workflowtemplates.argoproj.io "workflow-template-print-message" not found
   ✖ in "workflow-template-retry-with-steps" (Workflow): templates.retry-with-steps.steps[0].hello1 template reference workflow-template-random-fail-template.random-fail-template not found
   ✖ in "workflow-template-steps" (Workflow): templates.hello-hello-hello.steps[0].hello1 template reference workflow-template-print-message.print-message not found

✖ 7 linting errors found!

This is a big blocker using argo-lint, it's hard to find good static analysis tools and this isn't usable.

[agilgur5]

one would expect every example in this project to pass a lint with minimal effort.

They are all separate files in the examples, as you noted.

it's hard to find good static analysis tools

I don't think any contributor would disagree with that. There's a reason argo lint is an existing feature.

This is listed as a P1 bug with workarounds and Argo is lacking contributors, per the Sustainability Effort. Anyone is welcome to contribute fixes, improvements to CI, etc