Better Error Messages - When Pod Creating

[aronchick]

I ran several pipelines serially, and the problem was that the containers needed to bind a PVC for each. The PVCs were read only one - so subsequent containers sat around pending, waiting for the PVC to release. Would be great if there was a better error message here than.

[elikatsis]

Hello David.

That's weird. First of all, PVCs can be one of the following (info by the official documentation):

• ReadWriteOnce (the volume can be mounted as read-write by a single node)
• ReadOnlyMany (the volume can be mounted read-only by many nodes)
• ReadWriteMany (means the volume can be mounted as read-write by many nodes)

You probably mean that your PVCs where RWO, right?
If yes, then there shouldn't be any pods pending because of that. Instead, they should just be scheduled on the same node (adding load on that node).

On the other hand, if you mean that the access mode was ROM, that would allow pods to be scheduled to any node.

[aronchick]

The PVC is RWO - and the second pod that tried to start (both were TFJobs) would fail and crash because the PVC was not mountable. Then it would continue to do so over many minutes, with a CrashLoopBackoff every time, until, eventually, the first job finished, released the PVC, and the second PVC (eventually) restarted and picked it up.

[jessiezcc]

@IronPan, should this error handling happen at component level or pipeline level?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

It's now possible checking pod yaml and events directly from KFP UI: #3304