[sdk] parameters are ignored for pipelines generated with kfp-sdk 1.8.22

[luminoso]

Environment

• KFP version: 2.0.0
• KFP SDK version: 2.4.0
• All dependencies version:

kfp                       2.4.0
kfp-pipeline-spec         0.2.2
kfp-server-api            2.0.5

Steps to reproduce

1. Create a pipeline using kfp-sdk 1.8.22 and upload it. For example, producer_consumer_param.py is a good example.

To make it compatible with kfp-sdk 1.8.22 we just need to generate a yaml instead of json file

compiler.Compiler().compile(
        pipeline_func=producer_consumer_param_pipeline,
        package_path='producer_consumer_param_pipeline.yaml')

2. Upload your pipeline to kfp 2 with kfp-sdk 1.8.22

3. Run the pipeline using kfp-sdk 1.8.22 with params defined

client.run_pipeline(
    experiment_id="0a2707a0-a252-444e-8fd8-b82398ffcc3b",
    job_name="producer-consumer-1",
    pipeline_id="89309c2a-68de-41fd-873c-0cb732b45b95",
    version_id="4e2f67dd-a6d9-4861-b9d2-8d544325e882",
    params={"text": "olá mundo1"},
)

Everything works as expected.

Expected result

1. Update to kfp-sdk 2.4.0
2. Run the same code:

client.run_pipeline(
    experiment_id="0a2707a0-a252-444e-8fd8-b82398ffcc3b",
    job_name="producer-consumer-1",
    pipeline_id="89309c2a-68de-41fd-873c-0cb732b45b95",
    version_id="4e2f67dd-a6d9-4861-b9d2-8d544325e882",
    params={"text": "olá mundo2"},
)

3. Pipeline runs, but fails to pass params. The default Hello world argument is used to run the pipeline.

Which means that kfp-sdk 1.8.22 generated-pipeline arguments are ignored when invoked with kfp-sdk 2.x, even though parameters work as expected if webfronted is used.

Materials and Reference

I have to compile pipelines with kfp sdk 1.x due to taints and nodeselectors are not yet implemented in kfp-sdk 2.x: #9682 #9913

I could also use kfp.deprecated.Client but code is now half-broken

I can't also keep using kfp-sdk 1.x because package requirements are now do outdated that creates a mess for the rest of the project using other packages.

---

Impacted by this bug? Give it a 👍.

[jasonkit]

I experience the same issue.

I observed in frontend, when it is a v1 pipeline, Run is created using v1beta1 api, and it works.

I suspected this might be bug on backend when using v2beta1 api to create run with v1 pipeline instead of bug on sdk v2.

When I check KFP database's run_details table, I found that params I pass to sdk v2's client.run_pipeline will be appear under the RuntimeParameters column and Parameters column is empty

However if I using sdk v1's client.run_pipeline (or using frontend which is calling v1beta1 api) to create run, RuntimeParameters column is empty, and Parameters column will contain parameters from client.run_pipeline's params argument AS WELL AS the default parameter specified on the v1 pipeline.

[zijianjoy]

Currently we do not support authoring and uploading with a combination of v1 and v2 SDKs. Please choose one of the version for your authoring and uploading experience.

[zijianjoy]

/cc @chensun

[jasonkit]

@zijianjoy

Currently we do not support authoring and uploading with a combination of v1 and v2 SDKs. Please choose one of the version for your authoring and uploading experience.

In that case, could the team either help

• fixing the code under kfp.deprecated.Client in sdk v2, such that we can call v1 api on sdk v2 ?
• or upgrading the package dependency sdk v1 ?

As the outdated package dependencies in sdk v1 is blocking us updating other dependencies

[zijianjoy]

@jasonkit Feel free to create issues which identify the outdated package that needs to be updated. So it can be upgraded. Contributions are welcomed as well.

[luminoso]

- fixing the code under kfp.deprecated.Client in sdk v2, such that we can call v1 api on sdk v2 ?

I did this as a workaround. I simplified kfp-sdk 1.8.x to trigger v1 pipelines with parameters (this means using v1beta1 kfp endpoints, the same kfp REST endpoints that kfp-sdk 1.x uses)

The obvious requirement is to know which pipelines are generated with kfp-sdk 1.x and which ones aren't so that the correct kfp REST endpoint is triggered.

This means I'm simultaneously using kfp-sdk 2.x and a few bits from kfp-sdk 1.x.

The end result looks like this:

client = kfp.Client("http://...")

# new method, run_pipeline_v1beta1_sdk1() added with `patch_kfp_sdk2_client(), by porting and containizing the sdk1 
client = patch_kfp_sdk2_client(client)

# invoke a pipeline compiled with kfp-sdk 1.x, using kfp-sdk 2.x, that is running in kfp 2.x
client.run_pipeline_v1beta1_sdk1(
            "experiment-id",
            pipeline_id="pipeline-id",
            params=params,  #  params will work
        )

Since I did this I can try to do a PR fixing (and isolating) kfp.deprecated code, or I can share what I did and create a GitHub repository so that you can have patch_kfp_sdk2_client() / run_pipeline_v1beta1_sdk1() along with kfp-sdk 2.x

[zijianjoy]

@luminoso Regarding this question, please contact @connor-mccarthy to confirm if this change is acceptable.