Update imagepullsecret and update to v0.13.0

README.md

@@ -14,7 +14,7 @@ has Steps. Please look for more details in [Tekton repo](https://github.com/tekt
 ## Kubeflow Pipeline DSL to Tekton Compiler
 
 We are currently using [Kubeflow Pipelines 0.5.0](https://github.com/kubeflow/pipelines/releases/tag/0.5.0) and
-[Tekton 0.11.3](https://github.com/tektoncd/pipeline/releases/tag/v0.11.3) for this project.
+[Tekton 0.13.0](https://github.com/tektoncd/pipeline/releases/tag/v0.13.0) for this project.
 
 ![kfp-tekton](images/kfp-tekton-phase-one.png)
 

sdk/FEATURES.md

@@ -11,6 +11,7 @@ Below are the list of features that are currently available in the KFP Tekton co
  + [Input Parameters](#input-parameters)
  + [ContainerOp](#containerop)
  + [Affinity, Node Selector, and Tolerations](#affinity-node-selector-and-tolerations)
+ + [ImagePullSecrets](#imagepullsecrets)
 - [Pipeline DSL features with custom Tekton implementation](#pipeline-dsl-features-with-custom-tekton-implementation)
  * [Features with same behavior as Argo](#features-with-same-behavior-as-argo)
  + [InitContainers](#initcontainers)
@@ -22,7 +23,6 @@ Below are the list of features that are currently available in the KFP Tekton co
  * [Features with limitations](#features-with-limitations)
  + [ParallelFor](#parallelfor) - [Tracking issue][ParallelFor]
  + [Variable Substitutions](#variable-substitutions) - [Tracking issue][VarSub]
- + [ImagePullSecrets](#imagepullsecrets) - [Tracking issue][ImagePullSecrets]
  * [Features with different behavior than Argo](#features-with-different-behavior-than-argo)
  + [Sidecars](#sidecars) - [Tracking issue][Sidecars]
 - [Pipeline features that are unavailable on Tekton](#pipeline-features-that-are-unavailable-on-tekton)
@@ -82,7 +82,16 @@ The [affinity](/sdk/python/tests/compiler/testdata/affinity.py),
 [node_selector](/sdk/python/tests/compiler/testdata/node_selector.py), and
 [tolerations](/sdk/python/tests/compiler/testdata/tolerations.py) python tests are examples of how to use these features.
 
-This feature is recently implemented in Tekton and is available on the Tekton master branch and Tekton 0.13.0+.
+This feature is recently implemented in Tekton and is available on Tekton v0.13.0 onwards.
+
+### ImagePullSecrets
+
+ImagePullSecret is a feature for the components to know which secret to use when pulling container images from private registries. It is implemented
+with Tekton's [podTemplate](https://github.com/tektoncd/pipeline/blob/master/docs/podtemplates.md) field under Tekton
+PipelineRun. The [imagepullsecrets](/sdk/python/tests/compiler/testdata/imagepullsecrets.py) python test is an example of how to use this
+feature.
+
+This feature is recently implemented in Tekton and is available on Tekton v0.13.0 onwards.
 
 # Pipeline DSL features with custom Tekton implementation
 ## Features with same behavior as Argo
@@ -158,17 +167,6 @@ argo -> tekton
 [parallel_join_with_argo_vars](/sdk/python/tests/compiler/testdata/parallel_join_with_argo_vars.py) is an example of how Argo variables are
 used and it can still be converted to Tekton variables with our Tekton compiler. However, other Argo variables will throw out an error because those Argo variables are very unique to Argo's pipeline system. 
 
-### ImagePullSecrets
-
-[Tracking issue #1779][ImagePullSecrets]
-
-ImagePullSecret is a feature for the component to know which secret to use when pulling container images from private registries. It is implemented with Kubernetes `ServiceAccount` and bound to the Tekton's [ServiceAccount](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#specifying-custom-serviceaccount-credentials) field under Tekton
-PipelineRun. The [imagepullsecrets](/sdk/python/tests/compiler/testdata/imagepullsecrets.py) python test is an example of how to use this
-feature.
-
-However, this approach is not an ideal way to solve this problem because it adds extra work to the server to maintain the `ServiceAccount`. The better approach will be using `podTemplate` to store the image pull `secrets`. The Tekton pipeline doesn't support this feature yet, but we opened a
-[tracking issue](https://github.com/tektoncd/pipeline/issues/2339) in the Tekton pipeline.
-
 ## Features with different behavior than Argo
 Below are the KFP-Tekton features with different behavior than Argo.
 
@@ -199,7 +197,6 @@ An exit handler is a component that always executes, irrespective of success or
 <!-- Issue and PR links-->
 
 [ParallelFor]: https://github.com/tektoncd/pipeline/issues/2050
-[VarSub]: https://github.com/tektoncd/pipeline/issues/2322
-[ImagePullSecrets]: https://github.com/tektoncd/pipeline/issues/1779
+[VarSub]: https://github.com/tektoncd/pipeline/issues/1522
 [Sidecars]: https://github.com/tektoncd/pipeline/issues/1347
 [exitHandler]: https://github.com/tektoncd/pipeline/pull/2437

sdk/README.md

@@ -29,7 +29,7 @@ to ensure you are set up properly to use the KFP-Tekton compiler.
 
  - Python: `3.7.5`
  - Kubeflow Pipelines: [`0.5.0`](https://github.com/kubeflow/pipelines/releases/tag/0.5.0)
- - Tekton: [`0.11.3`](https://github.com/tektoncd/pipeline/releases/tag/v0.11.3)
+ - Tekton: [`0.13.0`](https://github.com/tektoncd/pipeline/releases/tag/v0.13.0)
  - Tekton CLI: [`0.8.0`](https://github.com/tektoncd/cli/releases/tag/v0.8.0)
 
 Follow the instructions for [installing project prerequisites](/sdk/python/README.md#development-prerequisites)
@@ -103,8 +103,6 @@ In order to utilize the latest features and functions of the `kfp-tekton` compil
 nightly built or build it from the [master](https://github.com/tektoncd/pipeline/blob/master/DEVELOPMENT.md#install-pipeline) branch. 
 Features that require special builds different from the 'Tested Version' will be listed below.
 
-- [Affinity, Node Selector, and Tolerations](/sdk/FEATURES.md#affinity-node-selector-and-tolerations)
-
 ## Additional Features
 
 ### 1. Compile Kubeflow Pipelines as a Tekton PipelineRun

sdk/python/README.md

@@ -36,7 +36,7 @@ the [SDK README](/sdk/README.md)
 1. [`Python`](https://www.python.org/downloads/): version `3.5` or later
 2. [`Kubernetes` Cluster](https://v1-15.docs.kubernetes.io/docs/setup/): version `1.15` ([required by Kubeflow](https://www.kubeflow.org/docs/started/k8s/overview/) and Tekton 0.11)
 3. [`kubectl` CLI](https://kubernetes.io/docs/tasks/tools/install-kubectl/): required to deploy Tekton pipelines to Kubernetes cluster
-4. [`Tekton` Deployment](https://github.com/tektoncd/pipeline/releases/tag/v0.11.3/): version `0.11.3` (or greater to support Tekton API version `v1beta1`), required for end-to-end testing
+4. [`Tekton` Deployment](https://github.com/tektoncd/pipeline/releases/tag/v0.13.0/): version `0.13.0` (or greater to support Tekton API version `v1beta1`), required for end-to-end testing
 5. [`tkn` CLI](https://github.com/tektoncd/cli#installing-tkn): required to work with Tekton pipelines
 6. [`Kubeflow Pipelines` Deployment](https://www.kubeflow.org/docs/pipelines/installation/overview/): required for some end-to-end tests
 
@@ -48,10 +48,10 @@ A working Tekton cluster deployment is required to perform end-to-end tests of t
 
 #### Tekton Cluster
 
-Follow the instructions listed [here](https://github.com/tektoncd/pipeline/blob/v0.11.3/docs/install.md#installing-tekton-pipelines-on-kubernetes)
+Follow the instructions listed [here](https://github.com/tektoncd/pipeline/blob/v0.13.0/docs/install.md#installing-tekton-pipelines-on-kubernetes)
 or simply run:
 
- kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.11.3/release.yaml
+ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.13.0/release.yaml
 
 **Note**, if your container runtime does not support image-reference:tag@digest (like cri-o used in OpenShift 4.x),
 use `release.notags.yaml` instead.

sdk/python/kfp_tekton/compiler/compiler.py

@@ -332,7 +332,7 @@ def _workflow_with_pipelinerun(self, task_refs, pipeline, pipeline_template, wor
  }
  }
 
- # Generate TaskRunSpec PodTemplate:s
+ # Generate TaskRunSpecs PodTemplate:s
  task_run_spec = []
  for task in task_refs:
  op = pipeline.ops.get(task['name'])
@@ -347,26 +347,16 @@ def _workflow_with_pipelinerun(self, task_refs, pipeline, pipeline_template, wor
  if bool(task_spec["taskPodTemplate"]):
  task_run_spec.append(task_spec)
  if len(task_run_spec) > 0:
- pipelinerun['spec']['taskRunSpec'] = task_run_spec
+ pipelinerun['spec']['taskRunSpecs'] = task_run_spec
 
  # add workflow level timeout to pipeline run
  if pipeline.conf.timeout:
  pipelinerun['spec']['timeout'] = '%ds' % pipeline.conf.timeout
 
- # generate the Tekton service account template for image pull secret
- service_template = {}
+ # generate the Tekton podTemplate for image pull secret
  if len(pipeline.conf.image_pull_secrets) > 0:
- service_template = {
- 'apiVersion': 'v1',
- 'kind': 'ServiceAccount',
- 'metadata': {'name': sanitize_k8s_name(pipelinerun['metadata']['name'], suffix_space=3) + '-sa'}
- }
- for image_pull_secret in pipeline.conf.image_pull_secrets:
- service_template['imagePullSecrets'] = [{'name': image_pull_secret.name}]
-
- if service_template:
- workflow = workflow + [service_template]
- pipelinerun['spec']['serviceAccountName'] = service_template['metadata']['name']
+ pipelinerun['spec']['podTemplate'] = pipelinerun['spec'].get('podTemplate', {})
+ pipelinerun['spec']['podTemplate']['imagePullSecrets'] = [{"name": s.name} for s in pipeline.conf.image_pull_secrets]
 
  workflow = workflow + [pipelinerun]
 

sdk/python/tests/compiler/testdata/affinity.yaml

@@ -47,7 +47,7 @@ metadata:
 spec:
  pipelineRef:
  name: affinity
- taskRunSpec:
+ taskRunSpecs:
  - pipelineTaskName: sleep
  taskPodTemplate:
  affinity:

sdk/python/tests/compiler/testdata/big_data_passing.yaml

@@ -523,6 +523,9 @@ spec:
 apiVersion: tekton.dev/v1beta1
 kind: PipelineRun
 metadata:
+ annotation:
+ tekton.dev/input_artifacts: '{}'
+ tekton.dev/output_artifacts: '{}'
  name: file-passing-pipelines-run
 spec:
  pipelineRef:

sdk/python/tests/compiler/testdata/imagepullsecrets.yaml

@@ -53,13 +53,6 @@ spec:
  taskRef:
  name: get-frequent
 ---
-apiVersion: v1
-imagePullSecrets:
-- name: secretA
-kind: ServiceAccount
-metadata:
- name: save-most-frequent-run-sa
----
 apiVersion: tekton.dev/v1beta1
 kind: PipelineRun
 metadata:
@@ -73,4 +66,6 @@ spec:
  value: This is a test
  pipelineRef:
  name: save-most-frequent
- serviceAccountName: save-most-frequent-run-sa
+ podTemplate:
+ imagePullSecrets:
+ - name: secretA

sdk/python/tests/compiler/testdata/node_selector.yaml

@@ -47,7 +47,7 @@ metadata:
 spec:
  pipelineRef:
  name: node-selector
- taskRunSpec:
+ taskRunSpecs:
  - pipelineTaskName: sleep
  taskPodTemplate:
  nodeSelector:

sdk/python/tests/compiler/testdata/tolerations.yaml

@@ -53,7 +53,7 @@ metadata:
 spec:
  pipelineRef:
  name: tolerations
- taskRunSpec:
+ taskRunSpecs:
  - pipelineTaskName: download
  taskPodTemplate:
  tolerations: