Support using a PipelineTask in the Pipeline CRD to run other Pipelines the same way we run a Task (nest pipelines in pipelines)

[skaegi]

Pipelines group tasks together in a workflow. It would be incredibly useful to be able to re-use that mechanism to create sub-workflows.

Pipelines and Tasks both take Params, Workspaces, and generate Results so with a little work it should be possible to use a PipelineRef in place of a TaskRef when defining a Pipeline Task. For the other aspects required to generate a "sub"-PipelineRun these could be inherited from the parent PipelineRun.

Here's a basic example to illustrate syntax...

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline-calling-pipeline
spec:
  params:
  - name: best-param-ever
  workspaces:
  - name: pipeline-ws1
  tasks:
  - name: pipeline-in-task-clothing
    pipelineRef:
      name: some-sub-pipeline
    workspaces:
    - name: sub-pipeline-ws0
      workspace: pipeline-ws1
  - name: regular-task
    taskRef:
      name: some-task
    params:
    - name: new-input
      value: $(tasks.pipeline-in-task-clothing.results.some-ouput)
    workspaces:
    - name: src
      workspace: pipeline-ws1
    runAfter:
    - pipeline-in-task-clothing

In a nutshell, other than using pipelineRef in-place of taskRef the definitions behave the same.

[vdemeester]

/kind feature

[bobcatfish]

pipeline-in-task-clothing

[jlpettersson]

Pipelines and Tasks both take Params, Workspaces, and generate Results so with a little work it should be possible to use a PipelineRef in place of a TaskRef when defining a Pipeline Task. For the other aspects required to generate a "sub"-PipelineRun these could be inherited from the parent PipelineRun.

I feel the use case is a bit unclear here. But I do have things that reminds me about this. E.g. The use-case I describe in #2505 (comment) - there I use (or "link") two PipelineRun and reuse the workspace volume.

On the other hand, I will have another use-case that also reminds a bit. But one fundamental difference is that I probably want to have it asynchronous. When talking about "linking" Pipelines or "chain", it may be done synchronous or asynchronous. That is also related to e.g. introducing asynchronous tasks - but I need to think about it more.

[madbence]

This feature would be really useful for us. eg. right now we have a task that can set the status on a given commit, and a task that can execute unit tests (eg. run npm test, etc.). We'd like to distribute a task/pipeline that can combine these (eg. set the commit to pending, run unit tests, set commit status to success/failure).

we'd like to provide to option to use set-status and run-unit-tests in a pipeline, and to be able to use a "higher-level" task, like "run-unit-tests-with-commit-status".

[bobcatfish]

Thanks for the use case @madbence this looks exactly along the lines we're thinking as well - just to double check, set-status sets the status of something like a GitHub check (or similar using non-GitHub)?

[bobcatfish]

oh and i forgot to post here some relevant links:

• WIP design doc for pipelines in pipelines - visible to tekton-dev
• Also @imjasonh has been looking into some thoughts around Design alternative Task Implementations inside a pipeline #215 which may let us conquer 2 issues with one stone

[madbence]

@bobcatfish exactly, set-status creates a commit status.

[vdemeester]

cc @mattmoor

[bobcatfish]

Tekton custom tasks would let us have this feature and more; planning to put this feature on hold until we have custom tasks and then we can decide if we want it as a top level feature or not.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vdemeester]

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

[tekton-robot]

@vdemeester: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dbazhal]

Couldn't this issue be done without custom tasks? We're in pain from disability to include pipelines into pipelines :) Problem is that we have to include task "run_pipeline" each time when we want to include pipeline inside pipeline. Each time such "incuded" pipeline fails, there is no easy way to find out why subpipeline failed, or what tasks it contained, etc.

[JPEWdev]

I also would really like support for running pipelines inside of pipelines. This is really useful for the case were you want some "super pipeline" that runs different variants of the same sub pipeline with different arguments (i.e. like a matrix/parameterized build from most other CI systems), provided that it can run all the sub-pipelines in parallel with different workspaces.

[JPEWdev]

I also would really like support for running pipelines inside of pipelines. This is really useful for the case were you want some "super pipeline" that runs different variants of the same sub pipeline with different arguments (i.e. like a matrix/parameterized build from most other CI systems), provided that it can run all the sub-pipelines in parallel with different workspaces.

Also I would like to point out that it would only really be useful if the subpipelines could be scheduled on different nodes. My (limited) understanding of tekton is that this might be difficult if the parent pipeline has to specify the workspace

[dbazhal]

Also I would like to point out that it would only really be useful if the subpipelines could be scheduled on different nodes. My (limited) understanding of tekton is that this might be difficult if the parent pipeline has to specify the workspace

agreed. Work with workspaces is sometimes pain, double pain if super-pipeline can't create ws, so Super-pipeline runs 1) task to create ws 2) each sub-pipeline with passing it workspace as argument 3) has to finish with task that deleted ws.

Also, i dont know how it could be done, but it would be better to aggregate sub-pipelines bodies into single pipelinerun. Creating separate pipeline run for each "sub"-pipeline makes it difficult to read logs of each task in each sub-pipeline(even more difficult with openshift tekton pipelines web-interface), it makes sub-pipelines execute slower(running pipelinerun isnt flash-fast itself, and it makes sometimes passing same arguments to each sub-pipeline a lot of copy-paste work.

[aelbarkani]

+1 for this feature, very helpful for reusing pipelines !

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[jerop]

/lifecycle frozen

TEP-0056: Pipelines in Pipelines
Experimental Project

[jerop]

TEP-0056 proposing Pipelines in Pipelines is implementable - we welcome contributions in implementing it

https://github.com/tektoncd/community/blob/main/teps/0056-pipelines-in-pipelines.md

/help-wanted

[EmmaMunley]

Happy to start working on implementing TEP-0056.

/assign @EmmaMunley

[ChrisPaul33]

Hi, there are any updates about the implementation of the feature?

[pritidesai]

Hi @ChrisPaul33, we have started implementing this feature. What API version are you currently using? Looking for some insight for - #7055 (comment)

[ChrisPaul33]

Hello @pritidesai, we are currently using v1beta1.
We are still trying to migrate to Tekton so migrating from v1beta1 to v1 for us won't be a big deal.
I will try my best to look at #7055.

[uqix]

Is matrix with pipelineRef supported?