[backend] persistence agent don't update pipeline run status after workflow deleted

[algs]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?

• KFP version: 1.5.0

• KFP SDK version:

Steps to reproduce

1. launch a pipeline run, and observe the status of nodes in the run as expected from UI.
2. use command kubectl delete workflow xxx to delete the workflow corresponding to the pipeline run, then the unfinished nodes in the pipeline run stay running, and never been updated.
3. the delete workflow may happen when we delete the scheduledworkflow, and then the workflow without the parent scheduledworkflow got GC

Expected result

The running nodes in the pipeline run been marked as terminated

Materials and Reference

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

cc @Jeffwan

[Bobgy]

Thank you @algs for flagging this bug report!
This is a real problem from the persistence agent design.

My rough idea to improve would be that, persistence agent should also periodically list all runs from KFP API that are still in running state, if any of the runs do not have a corresponding argo workflow. We should update the DB record to maybe a special state called "WORKFLOW_DELETED" etc.

What do you think?

[DaleSin]

Maybe just list runs which have not been updated/reported for a certain time, it should be a few runs.

[algs]

Maybe just list runs which have not been updated/reported for a certain time, it should be a few runs.

would this because of service error rather than the corresponding workflow deleted?

[Jeffwan]

My rough idea to improve would be that, persistence agent should also periodically list all runs from KFP API that are still in running state, if any of the runs do not have a corresponding argo workflow. We should update the DB record to maybe a special state called "WORKFLOW_DELETED" etc.

An alternative could be persistent agent watches workflow changes and if there's a delete workflow event, check running state of associated run and update DB. I prefer to choose the way which can reduce apiserver pressure. :D

[Bobgy]

@Jeffwan great idea! That feels like the most efficient solution.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

/lifecycle frozen
Welcome anyone to contribute this!