Any plan to integrate with Tekton pipeline

[gyliu513]

Tekton pipeline plan to be a k8s-native piepline resource, so any plan for kubeflow pipeline integrate with it?

/cc @jlewi @jinchihe @hougangliu

[vicaire]

Thanks @gyliu513. This is a very interesting proposal. We need to compare the feature set with Argo. I am not sure that it would make sense to support two pipeline resources if the feature set if similar. What do you think?

[gyliu513]

@vicaire Seems the goal of https://github.com/tektoncd/pipeline is to build a K8s-native Pipeline resource, tekton pipeline is also a project of CDF https://cd.foundation/ , so the tekton pipeline seems deserve to check more ;-)

[vicaire]

Sounds good. We will have a look at Tekton.

[Ark-kun]

Tekton is focusing on CI/CD scenarios while Argo and KF Pipelines have bigger scope and are more general-purpose.

[missedone]

@Ark-kun , would you like to share a little more of the bigger scope of KF/Argo?, i'm evaluating tekton/KF/Argo for our next continuous deployment tool.

[animeshsingh]

@vicaire Tekton is becoming the CI/CD backbone behind our cloud and even in OpenShift. Would like to see if you have given this more thought, and this might also be an opportunity to decouple KFP from Argo to some extent cc @paveldournov @IronPan

[axsaucedo]

@animeshsingh we were discussing Tekton PIpelines today - primarily as they are the backend to Jenkins X, and part of the CD.Foundation.

It's still early stages but I would be interested to learn more about how you are using Tekton in your cloud.

Furthermore, I personally see there is may be some opportunity within KFP, as currently I feel there is quite a lot of ambiguity around CI/CD vs ETL logic within individual KF Pipelines.

I think it would be interesting to explore even a conceptual separation between the use of KF Pipelines for ETL (ML Training / Batch processing) vs the CI/CD of trained pipelines.

I agree that having yet another service added to the KF Package is not desired, but exploring Tekton pipelines itself and even decoupling KFP from Argo could provide interesting insights on this conceptual separation of ETL vs CI/CD.

Perhaps even having something that would remove the CI/CD from the KFP all together would even help on this separation of concerns alltogether, which means that KFP could be left untouched.

What are your thoughts on this?

[jlewi]

@axsaucedo @animeshsingh I thought someone at the MLOps Sig meeting someone mentioned that they were working on a KFP DSL compiler that would execute on Tekton?

Does KFP have a well defined resource model? Would Tekton make a good resource model for pipelines?

[Ark-kun]

There were some plans to split the DSL compiler in two parts - the first creates a logical pipeline (a pipeline/graph component) and the second stage that converts the graph component to Argo workflow. This would make it easier to support more runners and orchestrators as they only need to implement the second part.

What are the key differences between Argo and Tekton?
Does Tekton help you pass data between tasks? (this is a core feature that Argo supports).

[jlewi]

Here's my limited understanding of Tekton's model.

• a Task corresponds to a sequence of steps executed sequentially in a pod
• Data can be passed between steps in a pod using the pod volume /workspace
• To pass data between tasks you use PipelineResource which provides a mechanism for moving data into/out of /workspace
  • For example, the git resource basically adds a container to a task which syncs a repo to /workspace so it is available to the steps in your task
  • There is a PipelineResource for GCS
  • And an open issue PipelineResource for S3 buckets (for IBM COS, AWS S3, Minio ...) tektoncd/pipeline#323 to support S3

@bobcatfish and other tekton experts might be able to shed more light on Tekton

[geota]

found a good discussion on tektoncd vs argo
tektoncd/pipeline#644 (comment)

If you are interested we created this spreadsheet a while ago which compares some of the Tekton Pipeline goals with existing CI/CD tools (though it compares Argo, not Argo CI or CD). (note this is shared with members of https://groups.google.com/forum/#!forum/knative-dev - since we used to be part of knative! - you'll have to join the group to see the doc)

Some notes.... Tekton seems more focused to source-based pipelines. Lots of similarities, the below were my takeaways:

• Sources as a first class concept - Explicit goal for Tekton. In Argo, they are a kind of artifact/input but it doesn't go further than that, doesn't seem to have awareness of branches, commit IDs, etc. unless explicitly provided as parameters
• Looping - Tekton does not want the Pipeline CRD to support this
• Dynamic execution - Tekton does not want the Pipeline CRD to support this

There is also this announcement:
https://www.weave.works/blog/argo-flux-join-forces

This space seems rapidly developing.

[animeshsingh]

@Ark-kun there is some work going on for kfp-tekton compiler here, curently being driven by us, but we are hoping we can others join in the effort.

Design doc and specifications: http://bit.ly/kfp-tekton
KFP-Tekton-Argo comparision: https://bit.ly/kfp-tekton-argo-comparision
Github repo: https://github.com/kubeflow/kfp-tekton
CD Foundation MLOps Sig:
https://cd.foundation/blog/2020/02/11/announcing-the-cd-foundation-mlops-sig/

The work is being driven under the MLOps Sig there. Its a biweekly 30 min sync, so would be great if you can subscribe and join

There were some plans to split the DSL compiler in two parts - the first creates a logical pipeline (a pipeline/graph component) and the second stage that converts the graph component to Argo workflow. This would make it easier to support more runners and orchestrators as they only need to implement the second part.

Any further work on your side? If you want, we should start some internal discussions

cc @jlewi

[animeshsingh]

@Ark-kun coming back to it. Would be great to syncup on the above cc @jessiezcc @paveldournov

[jlewi]

Duplicate of #3647