Design for triggering PipelineRuns from events

[bobcatfish]

Expected Behavior

We need to be able to trigger the creation of PipelineRuns from events, e.g. from knative/eventing events.

This is a placeholder issue we can use to gather requirements + ideas, which we can synthesize in this doc (note doc editable by members of knative-dev@googlegroups.com).

The most recent and relevant proposal

Now THIS is the most recent proposal/design which proposes creating a top level project based on the POC at https://github.com/tektoncd/experimental/tree/master/tekton-listener 🎉🎉🎉

Actual Behavior

PipelineRuns, and all PipelineResources required to execute them, must be created manually.

[bobcatfish]

#263 has some relevant suggestions around overriding values with a "parameters" type (ala PipelineParams for those who remember, #282 for those who don't!)

[afrittoli]

Brainstorming about this topic...

With the existing API, the way to trigger a Pipeline (or a Task) is to define the PipelineResources, define the PipelineRun and run it, so the service that defines the PipelineRun also triggers it.
I can image the use cases where a PipelineRun is defined but not yet executed, for instance if there are parts of the pipeline that we want to decouple, we can have pipelineA to publish an event when it's done (e.g. #587), and pipelineB to be triggered by that event. A real life example could be pipelineA runs some tests, pipelineB does test result post-processing, or publishes a comment on github or so. If pipelineA is used as a build template for kservice, once it completes the kservice will be deployed. pipelineB will run asynchronously and perform any postprocessing that is needed but that should not block the service deployment.

A simplifying assumption would be that service that generates PipelineRun objects knows everything needed by pipelineA and pipelineB from the beginning. If we remove that assumption, need we need to create a way for a PipelineRun to receive inputs via a CloudEvent.

In terms of implementations, if the PipelineRun specifies a channel as trigger, I think there are two options.

1. The PipelineRun controller becomes an Addressable and it subscribes to all channels defined in any PipelineRun that has not been executed yet
2. When the PipelineRun is created, the PipelineRun controller generates and Addressable that subscribes to the channel. When the Addressable receives the event, it patches the PipelineRun with info from the Cloud Event, and it sets a field that tells the PipelineRun controller that the PipelineRun is ready to be executed now.
   The same applies for TaskRun.

In option (1) the controller needs to process Cloud Events. Having one place that receives all events may be more efficient, but it may become an issue on a large scale system. Having one component doing reconciliation and listening for triggers may also be not ideal as the two functions may interfere which each other.

In option (2) the controller logic is not changed that much and there is a good separation of concerns. It may be possible to implement multiple Addressables that react to different kind of events... even though the same could be achieved by using Adapters that convert events to cloud events. The downside is that we create one service just to receive a single event.

An option (3) in the middle could be to have a single external addressable that subscribes to all events of all pipelines and tasks, which lives as long as the controller lives, with subscriptions added and removed on demand.

[dlorenc]

Ref to a recent design doc: https://docs.google.com/document/d/1vxFEovcLxDbypYMTAS1r6t8KCL4ltGmck266tbS1mTw/edit?disco=AAAACu6Ex1I&ts=5c90ad27

[bobcatfish]

I think we should consider this issue completed once we have something like agreement on a design; then open subsequent issues to tackle the implementation.

[vincent-pli]

@afrittoli
I like the option #2
I working on a demo with gitlab + knative + tekton pipeline.
We expect the pipeline could be triggered by knative eventing, seems not support, so we have to build a "subscribe" (a knative service) and bind a "pipelinerun.yaml" to it's image, it's hack.

One question:
Do you prefer to let pipeline controller know everything needed by a pipelinerun in advance but hold on(not do the really work, like create pod .etc)
or let payload of "event" (couldevent) tell everything?

[bobcatfish]

Related proposal from @iancoffey in #783

[bobcatfish]

Linking @iancoffey's proposal here: https://docs.google.com/document/d/1tfG6dkARP2xko-5trDV3D62DJZiXQBZgLtLH7GyUopM/edit#

[vtereso]

I have heard from @iancoffey that the above proposal (currently tektoncd/experimental/tekton-lister) is intended to live outside this repository? From the repo naming (at the very least), it would seem that anything in experimental is "experimental" and, at the discretion of the community, a potential candidate for upstreaming? We were looking to unify into one approach in the experimental repo and since there is still no supported eventing story in the Tekton core (this open issue), it seems worthwhile to have a discussion on this.

[bobcatfish]

hey @vtereso everything you've said is accurate! There's a bit more info about the process at https://github.com/tektoncd/community/blob/master/process.md#experimental-repo

We were looking to unify into one approach in the experimental repo and since there is still no supported eventing story in the Tekton core (this open issue), it seems worthwhile to have a discussion on this.

I'm currently working on a proposal to promote the project from experimental to a top level dir! I'll share the proposal shortly and then we can keep discussing :D

[vtereso]

I see that you added a slot on the schedule to discuss this 😄 I believe this issue effectively breaks down the differences between the two experimental approaches, notably that the current tekton-listener does not implement any mechanism for resource templating (creating resources at event trigger time). Since Tekton is already templating for params/input/output, it seems in-line with the design approach for eventing to provide the same for pipelineResources. Maybe, maybe not, but I look forward to a discussion.

[vtereso]

I'm currently working on a proposal to promote the project from experimental to a top level dir! I'll share the proposal shortly and then we can keep discussing :D

IIUC, is the idea to make a tektoncd/eventing repo rather than working it into tektoncd/pipeline? Would there be a further integration at a later step? Assuming some consensus could be reached on the spec, this seems like a necessary component of Tekton that would warrant combining the two. With the core logic of the proposal adding a few CRDs/controllers, it doesn't seem entirely off base.

[vincent-pli]

I think this discussion could help on the topic:
tektoncd/experimental#35

[bobcatfish]

With @iancoffey @ncskier @vtereso @dlorenc and @kimsterv we have created an updated proposal/design at https://docs.google.com/document/d/1fngeNn3kGD4P_FTZjAnfERcEajS7zQhSEUaN7BYIlTw 🎉

Will discuss at the next couple working group meetings.

[vdemeester]

@bobcatfish should this issue be closed now that tektoncd/triggers exists ?

[bobcatfish]

kk let's do it! The main reason I was keeping this open was b/c I still need to expand that design doc to include conditions/filtering but that's just an additive change :D

🎉 woot!