Create a run of the newest pipeline version by default using Python SDK

[jiyongjung0]

What steps did you take:

All actions were done using Python SDK (with TFX cli).

1. Created a pipeline 'A' with version name 'A' using client.upload_pipeline() API.
2. Created a pipeline version 'B' of pipeline 'A' using client.pipeline_uploads.upload_pipeline_version() API.
3. Created a run with client.run_pipeline() API. I used pipeline_id which was returned from 1 because pipeline_id is not changed even when a new version is uploaded.

What happened:

1. A run of pipeline version 'A' of pipeline 'A' was created.

What did you expect to happen:

1. A run of the most recent pipeline version('B') of pipeline 'A' is being created

Environment:

GCP - Cloud AI Platform Pipelines. Installed from marketplace.

KFP version: 0.2.5
KFP SDK version: 0.3.0

Anything else you would like to add:

When we select a pipeline from the web frontend UI, the most recent version is selected by default. So I thought that the most recent version of pipeline will be selected when I create a run using CLI.

/kind bug

[rmgogogo]

#3339
Jing, would you review

[rmgogogo]

I thought it default should run latest version, in such case, Jiyong should not hit the issue, right?

[jiyongjung0]

Right. I expected that it will run latest version, but it didn't.

[rmgogogo]

I assume without the version_ID in the create_run, it should pick up latest version. Jing may didn't aware this ticket. Let me ping her via hangout.

Meanwhile, I think #3339 is OK to go as no conflict to allow execute a specific version.

[jingzhang36]

While creating run from pipeline version, we can use client.runs.create_run (instead of client.run_pipeline). The former is automatically synced to the latest backend api interface (while the latter requires manual update to keep up with backend api interface)

When using client.runs.create_run, we use resource_references field to specify the pipeline version we want to use to create the run. Let me make an example for you and report back

[jingzhang36]

In addition to @pselden's PR of adding version support to run_pipeline method in SDK (#3339) . We also make our backend more accommodating by translating the pipeline id to its default version if users still want to use pipeline id instead of version to create run #3437.

[Bobgy]

/close
looks like fixed by #3437

[k8s-ci-robot]

@Bobgy: Closing this issue.

In response to this:

/close
looks like fixed by #3437

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[techjacker]

If passing pipeline_version is the recommended way (rather than pipeline_id) to trigger pipeline runs then surely pipeline_version has to be unique? However, I do not see this documented anywhere.

I was planning on using the same pipeline_version for multiple pipelines so would be good to get clarification on this.

I was planning on setting pipeline_version="${PIPELINE_NAME}-${GIT_SHA}". Good approach do you think?