Add support for Kubernetes annotations

[ptitzler]

This PR adds support for Kubernetes annotations for Kubeflow Pipelines and Apache Airflow runtime environments.

Annotations are supported for generic components and custom components and can be defined as pipeline defaults or for individual nodes.

Input needs to be entered as a string using the annotation_key=annotation_value format until proper UI support (#2780) is released.

Annotation keys are validated according to this spec.

Closes #2849

Pipeline default settings in VPE
Displayed in section "Node defaults" (Settings in this section apply to all nodes):

Generic node setting in VPE

Custom node setting in VPE
Displayed in section "Additional properties"

Applied annotations (KFP)

Example describe pod output (KFP)

$ kubectl describe pod lambda-... -n kubeflow-user-example-com
Name:         lambda-6sj47-3256227022
Namespace:    kubeflow-user-example-com
Priority:     0
Node:         cloning1.fyre.ibm.com/9.30.189.14
Start Time:   Sun, 07 Aug 2022 22:57:16 -0700
Labels:       pipeline/runid=e0b80cae-9c4b-4e89-a594-0b7c7eb55c16
             pipelines.kubeflow.org/cache_enabled=true
             pipelines.kubeflow.org/cache_id=
             pipelines.kubeflow.org/enable_caching=true
             pipelines.kubeflow.org/kfp_sdk_version=1.8.13
             pipelines.kubeflow.org/metadata_context_id=1
             pipelines.kubeflow.org/metadata_execution_id=1
             pipelines.kubeflow.org/metadata_written=true
             pipelines.kubeflow.org/pipeline-sdk-type=kfp
             workflows.argoproj.io/completed=true
             workflows.argoproj.io/workflow=lambda-6sj47
Annotations:  a-to-Z: soup to nuts
             annotationA: annotation default  A
             pipelines.kubeflow.org/arguments.parameters: {"URL": "http://www.bing.com"}
             pipelines.kubeflow.org/component_ref: {"digest": "8e4384f422a088e4814024df7955e952c1488bd091fa0d4873d5f611d741ceb4"}

Applied annotations (Airflow)

The applied annotations can be inspected using the kubectl describe pod command or in the DAG task instance details page in the Airflow UI:

Notes:

• Status (if checked, functionality is implemented and was verified):
  • Pipeline editor: validation
  • KFP generic node property
  • KFP custom node property
  • KFP pipeline default
  • Airflow generic node property
  • Airflow custom node property
  • Airflow pipeline default

What changes were proposed in this pull request?

• Added new optional 'Kubernetes pod annotations' pipeline default property
• Added new optional 'Kubernetes pod annotations' node property
• Updated the user_guide/pipelines.html documentation

How was this pull request tested?

• Added new validation tests
• Manual testing (see scope and status in previous section)
• Reviewed output of make docs

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[kevin-bates]

Thanks @ptitzler - this will be extremely useful!

[kiersten-stokes]

All my scenarios are testing as expected!

(Also, 20 files changed, wow! Hoping to reduce that for future params with some upcoming re-factoring 🤞 )