Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added pipeline root doc for kubeflow pipelines #2909

Merged
merged 10 commits into from
Sep 14, 2021
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions content/en/docs/components/pipelines/pipeline-root.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
+++
title = "Pipeline Root"
description = "Getting started with Kubeflow Pipelines pipeline root"
weight = 50

+++
{{% beta-status
feedbacklink="https://github.com/kubeflow/pipelines/issues" %}}

Starting from [Kubeflow Pipelines SDK v2](https://www.kubeflow.org/docs/components/pipelines/sdk/v2/) and Kubeflow Pipelines 1.7.0, Kubeflow Pipelines supports a new intermediate artifact repository feature: pipeline root in both [standalone deployment](https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/) and [AI platform Pipelines](https://cloud.google.com/ai-platform/pipelines/docs).

## Before you start
This guide tells you the basic concepts of Kubeflow Pipelines pipeline root and how to use it.
This guide assumes that you already have Kubeflow Pipelines installed, or want to use standalone or AI platform Pipelines options in the [Kubeflow Pipelines deployment
guide](/docs/components/pipelines/installation/) to deploy Kubeflow Pipelines.

## What is pipeline root?

Pipeline root represents an artifact repository where Kubeflow Pipelines stores a pipeline's artifacts.
This feature supports MinIO, S3, GCS natively using [Go CDK](https://github.com/google/go-cloud). Artifacts can be more accessible in S3 and GCS when integrating Kubeflow Pipelines with other systems.

**Note:** For MinIO, you can't change the MinIO instance. Kubeflow Pipelines can only use the Minio instance deployed with itself.
capri-xiyue marked this conversation as resolved.
Show resolved Hide resolved
capri-xiyue marked this conversation as resolved.
Show resolved Hide resolved

## How to configure pipeline root authentication
#### MinIO
You don't need to pass the authentication for MinIO
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add period at end

capri-xiyue marked this conversation as resolved.
Show resolved Hide resolved

#### GCS
If you want to specify the `pipeline root` to GCS :

* Set up pipeline root authentication in pipeline level via sdk:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/sdk/SDK

(all instances)

`dsl.get_pipeline_conf().add_op_transformer(aws.use_gcp_secret('xxx', ‘xxx’, ‘xxx’))`

* Set up pipeline root authentication via service account key:
check [authentication-pipelines](https://www.kubeflow.org/docs/distributions/gke/pipelines/authentication-pipelines/)
capri-xiyue marked this conversation as resolved.
Show resolved Hide resolved

#### S3
If you want to specify the `pipeline root` to S3:
capri-xiyue marked this conversation as resolved.
Show resolved Hide resolved

* Set up pipeline root authentication in pipeline level via sdk:
`dsl.get_pipeline_conf().add_op_transformer(aws.use_aws_secret('xxx', ‘xxx’, ‘xxx’))`
capri-xiyue marked this conversation as resolved.
Show resolved Hide resolved

**references**:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

References:

* [add-op-transformer](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.dsl.html#kfp.dsl.PipelineConf.add_op_transformer)
* [use-gcp-secret](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.extensions.html#kfp.aws.use_aws_secret)

## How to configure pipeline root

#### Via ConfigMaps in Kubernetes Cluster

You can configure default pipeline root for Kubeflow Pipelines via changing the `defaultPipelineRoot` entry of ConfigMaps `kfp-launcher` in Kubernetes Cluster.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"You can configure a default pipeline root for Kubeflow Pipelines by changing..."


```shell
kubectl edit configMap kfp-launcher -n ${namespace}
```
This pipeline root will be the default pipeline root for all pipelines running in the Kubernetes Cluster unless you override it through the following methods:
capri-xiyue marked this conversation as resolved.
Show resolved Hide resolved

#### Via Building Pipelines
capri-xiyue marked this conversation as resolved.
Show resolved Hide resolved
You can configure a pipeline root through the `kfp.dsl.pipeline` annotation when [building pipelines](https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/#build-your-pipeline)

#### Via Submitting a Pipeline through SDK
You can configure pipeline root via `pipeline_root` argument when you submit a Pipeline using one of the following:
* [create_run_from_pipeline_func](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client.create_run_from_pipeline_func)
* [create_run_from_pipeline_package](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client.create_run_from_pipeline_package)
* [run_pipeline](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client.run_pipeline).

#### Via Submitting a Pipeline Run through UI
You can configure a pipeline root via the `pipeline_root` run parameters when you submit a pipeline run in the UI
<img src="/docs/images/pipelines/v2/pipelines-ui-pipelineroot.png"
alt="Configure pipeline root on the pipelines UI"
class="mt-3 mb-3 border border-info rounded">
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.