Testing tools for Tekton Pipelines

[bobcatfish]

I'm not exactly sure what this looks like but I wanted to start tracking this here if ppl have ideas.

Basically I think that we should have a great testing story around Tekton Pipelines, e.g.:

• People should be able to write "unit tests" for their Tasks and Pipelines (I have no idea what these would actually look like!)
• People should be able to have some level of confidence in their Tasks without having to plug them into an entire Pipeline, run them, and observe what happened
• People should be able to express what the intent of their Tasks is, and verify it in some way that isn't manual

For inspiration, I really like the way that OPA polices can have tests written for them: https://www.openpolicyagent.org/docs/latest/how-do-i-test-policies/ Unfortunately Tekton Pipelines is totally different b/c it isn't a language (or is it????) and it's mostly side effect driven (the whole point is to execute stuff that causes changes in the state of the world) but I feel confident that if we really spend some time thinking about this we can make something cool!

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[bobcatfish]

This is a roadmap item: https://github.com/tektoncd/pipeline/blob/master/roadmap.md

/lifecycle frozen

[bobcatfish]

People should be able to write "unit tests" for their Tasks and Pipelines (I have no idea what these would actually look like!)

Quick passing thought - now that we have both params and results (and also workspaces), we could provide tools / libs to write "tests" that exercise Tasks and Pipelines with certain inputs and check for expected outputs and workspace contents. Maybe not as much of "unit test" as we'd like but probably a useful test for many cases 🤔

(Also wondering if there might be some overlap with the catalog tiers work @jerop )

[sfzylad]

It would be great to have some mechanism for testing tasks and pipelines. I think there are two scenarios worth considering:

1. testing against the task's or pipeline's results, maybe with a CR like TaskRunTest or PipelineRunTest that would take the TaskRun or PipelineRun definition and list of assertions to be tested and at the end indicate whether the assertions are satisfied.
2. testing tasks outside of tekton. I think having a mechanism that can evaluate a TaskRun and execute task without deploying it into tekton could be useful. Something that will interpolate parameters and run the image and script from each step. Perhaps incorporating some form of tests like bats.

[vdemeester]

It would be great to have some mechanism for testing tasks and pipelines. I think there are two scenarios worth considering:

1. testing against the task's or pipeline's results, maybe with a CR like TaskRunTest or PipelineRunTest that would take the TaskRun or PipelineRun definition and list of assertions to be tested and at the end indicate whether the assertions are satisfied.
2. testing tasks outside of tekton. I think having a mechanism that can evaluate a TaskRun and execute task without deploying it into tekton could be useful. Something that will interpolate parameters and run the image and script from each step. Perhaps incorporating some form of tests like bats.

For (2), something like buildkit-tekton could probably help 👼🏼

[cdbkr]

This would be really helpful when maintaining a catalog with a lot of Tasks and Pipelines.
It would be great to be able to actually test the following scenarios in isolation (without a cluster):

• Tasks:

  • interpolated variables coming from params (which could be optional/required)
  • order of steps
  • produces results
  • general task validation

• Pipelines:

  • order of tasks execution (sync/parallel/chained)
  • tasks are configured properly respecting required/optional params
  • all params referred in the task configurations are actually declared in the pipeline's spec-params section
  • workspaces required by the tasks are actually declared in the pipeline
  • general pipeline validation

  I'm a bit puzzled whether this would be "configuration testing" or not but, something like this would really help automating checks/validations when maintaining a huge catalog of tasks/pipelines.
  Also, we had a look at https://github.com/tektoncd/catlin and it might cover what I described above (not sure if that's what you are planning to do with that tool or if you'd like to enhance the testing tools separately).