Support disabling node output caching for custom KFP components

[ptitzler]

Pipeline nodes produce output, such as files. Some runtime environments support caching of these outputs, eliminating the need to re-execute nodes, which can improve performance and reduce resource usage. If a node does not produce output in a deterministic way - that is given the same inputs the generated output is different - re-using the output from previous executions might lead to unexpected results.

This PR enables users to optionally disable caching for nodes for Kubeflow Pipelines pipelines. Caching can be disabled by defining a pipeline default or by explicitly disabling it for individual nodes.

Example: Kubeflow Central Dashboard: cached output was re-used

Example: Kubeflow Central Dashboard: cached output was not re-used

Closes #2894

What changes were proposed in this pull request?

• Add new pipeline default property in new section 'custom node defaults'

  

• Add new node property for custom KFP components

• Updated relevant documentation in pipelines documentation topic

Known limitations:

• User cannot explicitly choose "use runtime environment default behavior". Once a user checks the 'disable' box they are locked in and can only choose between enable and disable (Add support for three-valued BooleanControl #2906)
• Pipeline default property is displayed in generic pipeline editor and airflow pipeline editor even though it is not supported in those runtime environments

How was this pull request tested?

• Manual testing:
  • caching not disabled (pipeline default and node property)
  • caching disabled (pipeline default) and not overridden for individual nodes
  • caching disabled (pipeline default) and but overridden for individual nodes
  • caching not disabled (pipeline default) but disabled for individual nodes
• As part of each test the pod metadata and the task status was reviewed
• Updated existing tests
• Reviewed output of make docs (user_guide/pipelines.html)

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[ptitzler]

The property labels and descriptions still require editing for clarity.

[akchinSTC]

verified cache and annotations set correctly, some small nits =)

[kiersten-stokes]

LGTM! All scenarios testing as expected (with the known limitation re: the Boolean handling in mind)

Pipeline default property is displayed in generic pipeline editor and airflow pipeline editor even though it is not supported in those runtime environments

An upcoming PR of mine should address that - this will be a good test of that PR's functionality 🙂

[ptitzler]

Final doc updates are done.