[bug] After upgrade to kfp 2.0(latest), not able to run old pipelines created using kfp v1. "Start" button disabled

[mChowdhury-91]

Environment

• How do you deploy Kubeflow Pipelines (KFP)? - Standalone deployment
• KFP version:
  kfp 2.2.0
  kfp-pipeline-spec 0.2.2
  kfp-server-api 2.0.2

Steps to reproduce

Recently we have upgraded kubeflow pipelines to support KFP V2, but noticed that backward compatibility is breaking. The pipelines that were created earlier using V1 is no more executable.
The "Start" button is disabled as the error says:"A pipeline must be selected" and under Run parameters section the parameters are not visible "This pipeline has no parameters".

Expected result

The v1 executions should work as before, without having to re-upload a new yaml version.
Note: If we upload the a new version with same yaml content, it works(shows the run parameters)

Impacted by this bug? Give it a 👍.

[mChowdhury-91]

@jlyaoyuli I did some more analysis on the above issue and here are my findings:
After upgrade, The PipelineSpec column is added new to pipeline_versions table which holds the data of pipeline yaml. Any new version upload, creates a new entry in the PipelineSpec column and the pipeline works.
Also, if we manually edit the database column to upload a yaml entry to old pipelines, they seem to work too..

Before upgrade, the data for pipeline yaml was being read/fetched from minio directly.
The backward compatibility seems to be broken.

In my opinion, this is the possible reason why old pipelines created before upgrade aren't working. Let me know your thoughts and analysis and how this can be fixed.

[jlyaoyuli]

Hello @mChowdhury-91 , thanks for your update! Would you mind providing the pipeline you used? V1 pipeline works well from my side.

[mChowdhury-91]

@jlyaoyuli None of the pipelines which were uploaded before the upgrade works, but as I mentioned, if we upload a new version of the same pipeline, it works well(V1 as well)

[thesuperzapper]

@mChowdhury-91 can you share a minimal pipeline that does not work, because I have a number of pipelines that do successfully work.

[subasathees]

@thesuperzapper @mChowdhury-91 @jlyaoyuli , I am also facing the same issue. None of the default pipeline running after upgrade to kubeflow v1.8.
I am using hello-world pipeline and it givens below error.
Error:

Hello world pipeline reference: https://www.kubeflow.org/docs/components/pipelines/v2/hello-world/

[Davidnet]

@subasathees Hi that is not an error, is a warning, that at the moment, the component runs in a python3.7 enviroment and that in the future it will use 3.8.

[DnPlas]

Hi folks, I am actually hitting a similar issue to what @mChowdhury-91 reports. After upgrading pipelines from 2.0.0-alpha7 (kubeflow 1.7) to 2.0.3, I cannot longer create runs from my already uploaded pipelines as the "Start" button is greyed out, check the image:

Please note that this is not the case when creating a run using the older Python SDK; so kfp==1.8.22 and a v1 code works just fine.

Also note that cloning an existing run that was created using the Python SDK also works, but trying to clone a run from an uploaded v1 pipeline does not work.

This is what I have observed so far:

SDK	Backend	Compiled pipeline	Create runs from UI	Create runs using SDK
1.8.22	2.0.3	v1	No	Yes
2.4.0	2.0.3	v2	Yes	Yes
2.4.0	2.0.3	v1	No	No

I wonder if this should be considered an issue for releasing kubeflow 1.8.1, do you mind confirming @chensun ?

[chensun]

@jlyaoyuli is going to look into it.

[DnPlas]

Hey @chensun @jlyaoyuli - are there any updates on this one? We think it would be a good idea to include it in a potential patch release of 1.8, but we need confirmation this is going to be worked on before EOW.

[jlyaoyuli]

Thank you everyone for all of the discussion. This issue should be fixed after #10276 is merged.

[chensun]

Hey @chensun @jlyaoyuli - are there any updates on this one? We think it would be a good idea to include it in a potential patch release of 1.8, but we need confirmation this is going to be worked on before EOW.

@DnPlas we released KFP 2.0.5 today, which contains this fix and another frequent ask: #10229

Please feel free to patch Kubeflow 1.8 release if these two issues would have high impact.