Add Cluster scope pipeline support

[zhangtbj]

Expected Behavior

We can also define Cluster scope pipeline as ClusterTask

Right now, Tekton has Cluster scope task ClusterTask, but If I also want to run a Cluster scope pipeline which include that ClusterTask.

So that people can define Cluster scope pipeline for all cluster-level users

Actual Behavior

Something like:

apiVersion: tekton.dev/v1alpha1
kind: ClusterPipeline
metadata:
  name: clusterpipeline
  name: demo-pipeline
spec:
  resources:
  - name: source-repo
    type: git
  tasks:
  - name: skaffold-unit-tests
    taskRef:
      name: unit-tests
...

[vdemeester]

/kind feature
/kind question

[tomgeorge]

If nobody is working on this, I will start this evening.

[imjasonh]

I believe there was a question about whether ClusterPipelines would be allowed to contain namespaced Tasks when run in a namespace. I think this is probably confusing enough we should punt on it until someone has a specific use case, and only support ClusterTasks in ClusterPipelines at first.

[tomgeorge]

got it, thanks!

[zhangtbj]

I think ClusterTask is just for some sample usage.

If it is an advantage or complex task or job, we need to define a pipeline to do more steps.

Such as cluster scope or level operation by using on the cluster level.

Or as we are in a multiple-tenants environment, we have a cluster scope pipeline that allows all namespaces multiple-tenants can use this cluster-level pipeline together. Or we have to define the pipeline in each namespace one by one.

[dbazhal]

Here is our case to use ClusterPipelines.

We're making multitenant system, each team has it's own project in "control" cluster. Each project has multiple environments(namespaces) like testing, prod, etc, in multiple clusters.
So, each service in each project should have pipeline that builds artifact, and delivers it promoting from environment to environment to production unchanged(because what we test on one environment should get the same as tested to production).
So what does this CI/CD pipeline? Launches some "build" pipeline. Then for each namespace in testing environment launches "deploy" pipeline. Then for each namespace in production environment launches same "deploy" pipeline.
Every service pipeline in every project uses that "build" and "deploy" pipelines.
And when the same "build"/"deploy" pipeline should be used in different namespaces, we want to code, test, and publish that pipelines once for all users, and make it available as ClusterPipeline.

In such case i agree that ClusterPipeline should be able to refer only to ClusterTasks, but not namespaced Tasks.
And maybe, if #2134 will be done, ClusterPipeline could refer to other ClusterPipelines.

[jsimao71]

Support for cluster scoped Pipeline is really critical. A cluster will typically have many namespaces being created (in multi-tenant or even single-tenant), while only a small number of pipelines. Required the commonly used pipeline to have to be deployed in all such namespaces is a big burden on the use of tekon, and a waste of resources. Not to mention upgrading pipelines across all namespaces. Please add this...My believe is that this is a critical practical features for the ease of use and adoption of Tekton. Ideally a admin should setup common pipelines at cluster level, and all devs using the cluster can benefit from those pipelines.
Much appreciated..

[jsimao71]

BTW, the issue of the scope of tasks in cluster level pipeline is in concept easy to deal with.
A enum property can fine-tune the behavior. e.g.
task-scope: cluster-only | namespace-only | cluster-first | namespace-first
with the following meaning:
cluster-only -- use always cluster tasks (which need to be setup by admin)
namespace-only -- assume relevant tasks have been added to namespace
cluster-first [not as useful] -- check cluster task first and use if it exists, otherwise check namespace
namespace-first -- check namespace task (if defined), fallback to cluster task
The default could be namespace-first (but other are also reasonable).
Much appreciated!!!!

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[dbazhal]

na-ah, please don't
/remove-lifecycle stale

[jsimao71]

@dbazhal can you plese clarify why you think cluster support for pipelines is not a good idea?! Thanks!
Personally..I find really bad pipelines can not be installed at cluster level like task.
For exemple, if you create a simple but very common and useful pipeline with two task to git clone and build with maven and push image, this limitation of tekton implies that the pipeline needs to be installed in all namespaces.
Fundamentally, there is not much difference between pipelines and tasks. A pipeline of two tasks is almost a task, just a composite. Why is task support at cluster level, but not pipeline?!? Does not make much sense to me.
It really ruins the tekton experience, because a cluster admin can not setup tekton for easy install in all namespaces.
As it is requires all devs that own a namespace to reinstall all relevant pipelines.
So I insist please consider adding this feature.
And if not, please provide some rationale why not...
Thanks.

[dbazhal]

wat :) i was talking to bot that tried to stale issue.

[zhangtbj]

Yes, Our service leverages Tekton a lot.

And our service is not used for personal user, it is a multiple tenant case, it means we provide some shared Tekton tasks as ClusterTask, so that each namespace user can share or use them.

So it is the same case for Cluster scope pipeline. Such as for a common CI/CD or build relate processes.

[dbazhal]

We are currently compelled to copy each pipeline into user namespace to execute :D
That kinda says something about importance of cluster-scoped pipelines :)
I think feature importance is not an issue, there must be some trouble with adding this functionality. I'd like to help but i don't know how

[bobcatfish]

Hey all!

I think it is likely we'll be moving away from cluster scoped resources ( not sure if @vdemeester and others might disagree) as we add more support for tekton bundles (more tools and support coming soon!)

The idea is that there are a lot of limitations to referring to Tasks and Pipelines in the cluster (including namespace scoping and it being hard to version them (#1839)) so we are moving toward a model where you can reference Tasks and Pipelines other places, e.g. the feature we recently added allows you to store them in OCI registries (like images) and reference them there, and we'll likely add support for other locations such as git in the future.

[dbazhal]

Aw. Talking about solving concrete problem - absence of reusable pipelines mechanism - how soon could it be done? Wouldn't it be faster to add cluster-scoped pipelines first, and then calm and move to bright future with bundles?) (namespaced pipelines hurt really much)

[bobcatfish]

I think you can start using it today but I don't know what the mechanics of that look like b/c I think you'd have to create the Bundle by hand? Not 100% sure but I think @imjasonh @vdemeester know more

[jsimao71]

tekton bundles makes sense if you could reference directly in taskruns and pipelineruns. If the can only be used to define pipeline specs -- as the docs suggest -- it does does solve the problem, because it requires the pipeline to be defined in each namespace again -- which is the problem at hand.
Even with bundles, one cloud argue that bundles are really a complimentary mechanism. It also adds extra complexity related with authentication in remote OCI registy.
Moreover if you don't add cluster level pipelines, and still keep cluster level tasks, the runtime becomes completely asymmetric. On the other hand, if you remove cluster tasks you loose backward compatibility of the model. Kubernetes has other native resources that are scoped at namespace and cluster level -- e.g. roles and role bindings. So it makes sense for tekton to maintain the spirit of K8s model.
The question is: how hard can it be to add pipelines at cluster level, for somebody that is familiar with the code?!?
If you think this is really hard to do, please provide some explanation why this is the case..otherwise the issue will continue to popup, people get confused, and you have lots of upset users.

[vdemeester]

tekton bundles makes sense if you could reference directly in taskruns and pipelineruns.

This is the case (https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#specifying-the-target-pipeline). You can use "reference" to tekton bundles in PipelineRun, Pipeline and TaskRun 😉

That said, I do not think we should move away from cluster scoped resources, and the more I think about, the more I don't see why we wouldn't have ClusterPipeline.

The question is: how hard can it be to add pipelines at cluster level, for somebody that is familiar with the code?!?
If you think this is really hard to do, please provide some explanation why this is the case..otherwise the issue will continue to popup, people get confused, and you have lots of upset users.

This is not really the question. The code is a "detail", what is important is "does ClusterPipeline solve a problem for users ?" or does it make sense. I think it makes sense and as @jsimao71 @dbazhal @zhangtbj wrote, I think there is some need for it even. In OpenShift, having ClusterPipeline would simplify even more our "first time user" flows.

One problem to think about is @imjasonh's comment

I believe there was a question about whether ClusterPipelines would be allowed to contain namespaced Tasks when run in a namespace. I think this is probably confusing enough we should punt on it until someone has a specific use case, and only support ClusterTasks in ClusterPipelines at first.

Should we allow only ClusterTask and bundle reference from ClusterPipeline ? If we do, what is the impact / confusion on the user ? If that's the case, it would mean, for example, that converting a Pipeline into a ClusterPipeline is more involved than changing the type (because you would also need to ensure tasks referenced are ClusterTask.

Anyway, I think it would make sense to discuss this and implement if we reach a consensus 😉. It would also need to pass through a TEP.

[vdemeester]

I think you can start using it today but I don't know what the mechanics of that look like b/c I think you'd have to create the Bundle by hand? Not 100% sure but I think @imjasonh @vdemeester know more

You can use oci for now, until we integrate this more into "official" tooling.

[jsimao71]

Thanks for the update and discussion @vdemeester.
As you point out, I think this thread shows clearly that lots of people in the community think this is a great idea.
This simplifies life for first time users --- which can really on high-level tools or clusters pre-setup by an admin). And also saves possible duplicate installs across namespaces, which is wasteful at best and a pain to maintain and update in the worst case.

On cluster scoped pipelines refering to namespace scoped tasks as you point out, this can lead to confusions and is somewhat error prone. The exception is when it works kind of "inheritance with override" -- that is, there is a cluster scoped task defined, but it is allowed for a namespace scope task with same name to override the cluster one.
Note, however, that because (cluster)tasks can have parameters, most uses cases where this "inheritance with override" could be considered, probably can be implemented more simply setting parameters in the task when running a task run.
So I suggest that, at least in a first iteration, not to support cluster pipelines references to namespace tasks.
When we have a stable implementation of cluster pipelines, and people start to use it, it might become clear if there are some valid use cases for references of type cluster pipeline->namespace task.
Thanks everybody for the comments...specially the "heroes" that are going to do the code implementation ;).

[dbazhal]

Any ideas on this? :) Saw 2021 roadmap, there are no cluster pipelines :(

[jsimao71]

Yes..its taking some time for Tekton devs realize how important/critical this is. Let's hope it is added to the roadmap.
In the mean time, and without affording to wait, my team developed a PaaS toolset UI + CLI on top of Tekton, to at least make this a bit more manageable. If you interested, see my blog thread.
On CI/CD:
https://medium.com/swlh/building-java-spring-apps-in-the-cloud-with-kubernetes-cloud-manager-tekton-pipelines-681407bcd8aa
On the CLI:
https://einnovator.medium.com/a-cli-tool-for-multi-cluster-kubernetes-cloud-manager-rocks-af17124af2a6

[bobcatfish]

I've added this to the API working group discussion next week if any interested folks are able to attend! and/or we can discuss at an upcoming Wednesday working group

https://github.com/tektoncd/community/blob/main/working-groups.md#general
https://github.com/tektoncd/community/blob/main/working-groups.md#api

my 2 cents is that I'd like to move away from using the Cluster as a source of truth (e.g. bundles are an alternative to this but still WIP) but also I could see adding this feature unblock all of you in the meantime

[vdemeester]

So we discussed this a bit during the API WG. In a gist, it seems we are leaning towards getting away from Cluster scoped object once we have something that is better (OCI bundles, …). Thus adding support to ClusterPipeline would add more complextiy and technical dept for the future. Instead we should focus on making OCI bundle enabled by default. Especially given the current open question : what can a ClusterPipeline refer (only clustertask, …) ?

my 2 cents is that I'd like to move away from using the Cluster as a source of truth (e.g. bundles are an alternative to this but still WIP) but also I could see adding this feature unblock all of you in the meantime

One thing here though, ClusterPipeline and ClusterTask features are totally different than using the "cluster" as a source of truth. When reading "Cluster as a source of truth", for me this also includes Task, Pipeline, … that are in the Cluster — which is not what is being discussed here. Even if we enable OCI bundle by default, having CRDs, object in the cluster is in the ADN of the tektoncd/pipeline project — it enables us to have CustomTask, … I don't see this going away anytime soon. And, even if it is in the Cluster, it doesn't necessarly mean it is the source of truth ; on a side note, we need to define what we mean by source of truth : from the user perspective or from the pipeline controller/webhook perspective ? (because those may be very different)

[dbazhal]

Ah, too bad :) We were really hoping pipelines will live with adding cluster pipelines and ability to refer pipelines in pipelines, because there was real need to reuse same pipeline(let's say "deploy to one environment") in other pipelines in different namespaces.
Without this features we are forced to consolidate more functions in one clustertask, and use it instead of pipelines, so task reusability drops, and reusability moved to code level.

[zhangtbj]

Understand if it complex to add cluster scope pipeline from technical perspective.

We don't have this requirement now because we already downgrade to a single Task in our side.

But two cents from me or maybe from user perspective:

• if the pipeline is also a kind of template like Task, I think it should be able to extracted and shared at the cluster level.
• If Tekton is also a CI/CD tool, like Jenkins or Concourse, I think it is better to share the pipeline for the complex scenarios.

We are doing build, but just do build as single step, but for other complex cases like:

• Do build -> unit test -> deploy as a common process
• Do build -> onboard to stage -> onboard to production env
• Or some multiple cases by using condition, etc...

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[dbazhal]

Still love this idea, so I'd
/remove-lifecycle stale
/lifecycle frozen

[krishnaraman001]

Can anyone tell me what/how to achieve using OCI bundles, can we have any sample to know it better?

[AlanGreene]

@krishnaraman001 you can find information about Tekton Bundles in the docs. The feature must be enabled first, the docs include details of how to do this.

For example:

• referencing a Pipeline: https://tekton.dev/docs/pipelines/pipelineruns/#tekton-bundles
• creating a bundle using tkn: https://github.com/tektoncd/cli/blob/main/docs/cmd/tkn_bundle_push.md

[lbernick]

@dbazhal @zhangtbj @jsimao71 and others on this thread-- sounds like you might have opinions on the future of ClusterTasks, please feel free to chime in on #4476!

[lbernick]

We've now implemented remote pipelines, and we have a cluster-scoped resolver that can resolve pipelines from other namespaces. I think that meets the needs described in this issue, so I'm going to close. Feel free to reopen if any use cases are unaddressed!

/close

[tekton-robot]

@lbernick: Closing this issue.

In response to this:

We've now implemented remote pipelines, and we have a cluster-scoped resolver that can resolve pipelines from other namespaces. I think that meets the needs described in this issue, so I'm going to close. Feel free to reopen if any use cases are unaddressed!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.