Make multi-valued input fields more user-friendly

docs/source/user_guide/pipelines.md

@@ -103,7 +103,7 @@ Each pipeline node is configured using properties. Default node properties are a
 
 **Default properties that apply only to custom nodes**
 
- - [Disallow cached output](#disallow-cached-output)
+ - [Disable node caching](#disable-node-caching)
 
 #### Adding nodes
 
@@ -163,7 +163,7 @@ Nodes that are implemented using [custom components](pipeline-components.html#cu
    - [Data volumes](#data-volumes)
    - [Kubernetes tolerations](#kubernetes-tolerations)
    - [Kubernetes pod annotations](#kubernetes-pod-annotations)
-   - [Disallow cached output](#disallow-cached-output)
+   - [Disable node caching](#disable-node-caching)
 
 #### Defining dependencies between nodes
 
@@ -203,19 +203,26 @@ The following alphabetically sorted list identifies the node properties that are
 
 ##### Data volumes
    - A list of [Persistent Volume Claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PVC) to be mounted into the container that executes the component. 
-   - Format: `/mnt/path=existing-pvc-name`. Entries that are empty (`/mnt/path=`) or malformed are ignored. Entries with a PVC name considered to be an [invalid Kubernetes resource name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) will raise a validation error after pipeline submission or export.
-   - The referenced PVCs must exist in the Kubernetes namespace where the pipeline nodes are executed.
+   - Format: 
+     - _Mount path_: the path where the PVC shall be mounted in the container. Example: `/mnt/datavol/`
+     - _Persistent volume claim name_: a valid [Kubernetes resource name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) identifying a PVC that exists in the Kubernetes namespace where the pipeline nodes are executed. Example: `my-data-pvc`
    - Data volumes are not mounted when the pipeline is executed locally.
 
-##### Disallow cached output
+##### Disable node caching
    - Pipeline nodes produce output, such as files. Some runtime environments support caching of these outputs, eliminating the need to re-execute nodes, which can improve performance and reduce resource usage. If a node does not produce output in a deterministic way - that is, when given the same inputs, the generated output is different - re-using the output from previous executions might lead to unexpected results.
+   - Format:
+     - `True` node output is not cached
+     - `False` node output is cached
+     - If no behavior is specified, the runtime environment's default caching behavior is applied.
    - Caching can only be disabled for pipelines that are executed on Kubeflow Pipelines.
 
 ##### Environment Variables
    - This property applies only to generic components.
-   - A list of environment variables to be set inside in the container.  Specify one variable/value pair per line, separated by `=`.
+   - A list of environment variables to be set inside in the container.
+   - Format:
+     - _Environment variable_: name of the variable to be set. Example: `optimize`
+     - _Value_: the value to be assigned to said variable. Example: `true`
    - A set of default environment variables can also be set in the pipeline properties tab. If any default environment variables are set, the **Environment Variables** property in the node properties tab will include these variables and their values with a note that each is a pipeline default. Pipeline default environment variables are not editable from the node properties tab. Individual nodes can override a pipeline default value for a given variable by re-defining the variable/value pair in its own node properties. 
-   - Example: `TOKEN=value`
 
 ##### File Dependencies
    - This property applies only to generic components.
@@ -230,20 +237,28 @@ The following alphabetically sorted list identifies the node properties that are
 
 ##### Kubernetes Pod Annotations
    - A list of [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#attaching-metadata-to-objects) to be attached to the pod that executes the node.
-   - Format: `annotation-key=annotation-value`. Entries that are empty (`annotation-key=`) are ignored. Entries with a key considered to be [invalid](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set) will raise a validation error after pipeline submission or export.
+   - Format:
+     - _Key_: a [valid Kubernetes annotation key](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). Example: `project`
+     - _Value_: value to be assigned to said annotation key. Example: `abandoned basket analysis`
    - Annotations are ignored when the pipeline is executed locally.
-   - Example: `project=abandoned basket analysis`  
 
 ##### Kubernetes Secrets
-   - A list of [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) to be accessed as environment variables during Jupyter notebook or script execution. Format: `ENV_VAR=secret-name:secret-key`. Entries that are empty (`ENV_VAR=`) or malformed are ignored. Entries with a secret name considered to be an [invalid Kubernetes resource name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) or with [an invalid secret key](https://kubernetes.io/docs/concepts/configuration/secret/#restriction-names-data) will raise a validation error after pipeline submission or export. The referenced secrets must exist in the Kubernetes namespace where the generic pipeline nodes are executed.
+   - A list of [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) to be accessed as environment variables during Jupyter notebook or script execution.
+   - Format:
+     - _Environment variable_: name of the variable to be set. Example: `optimize`
+     - _Secret Name_: a valid [Kubernetes resource name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) identifying a [secret](https://kubernetes.io/docs/concepts/configuration/secret/#restriction-names-data) that exists in the Kubernetes namespace where the pipeline nodes are executed. Example: `database-credentials`
+     - _Secret Key_: key that is defined in said secret. Example: `uid`
    - Secrets are ignored when the pipeline is executed locally. For remote execution, if an environment variable was assigned both a static value (via the 'Environment Variables' property) and a Kubernetes secret value, the secret's value is used.
-   - Example: `ENV_VAR=secret-name:secret-key`
 
 ##### Kubernetes Tolerations
    - A list of [Kubernetes tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to be applied to the pod where the component is executed.
-   - Format: `TOL_ID=key:operator:value:effect`. Refer to [the toleration specification](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#toleration-v1-core) for a description of the values for `key`, `operator`, `value`, and `effect`.
+   - Format: 
+     - _Key_: taint key the toleration applies to
+     - _Operator_: represents the key's relationship to the value. Must be `Equal` or `Exists`.
+     - _Value_: taint value the toleration matches to
+     - _Effect_: indicates the taint effect to match. If specified, must be `NoExecute`, `NoSchedule`, or `PreferNoSchedule`.
+   - Refer to [the toleration specification](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#toleration-v1-core) for a description of each property.
    - Tolerations are ignored when the pipeline is executed locally.
-   - Example: `TOL_1=my-key:Exists::NoExecute`
 
 ##### Label
    - Specify a label to replace the default node name. For generic components the default label is the file name. For custom components the default name is the component name.

elyra/cli/pipeline_app.py

@@ -35,6 +35,7 @@
 from elyra.metadata.schemaspaces import Runtimes
 from elyra.pipeline import pipeline_constants
 from elyra.pipeline.component_catalog import ComponentCache
+from elyra.pipeline.component_parameter import VolumeMount
 from elyra.pipeline.kfp.kfp_authentication import AuthenticationError
 from elyra.pipeline.kfp.kfp_authentication import KFPAuthenticator
 from elyra.pipeline.parser import PipelineParser
@@ -584,7 +585,11 @@ def describe(json_option, pipeline_path):
         # (... there are none today)
         # volumes
         for vm in node.get_component_parameter(pipeline_constants.MOUNTED_VOLUMES, []):
-            describe_dict["volume_dependencies"]["value"].add(vm.pvc_name)
+            # The below is a workaround until https://github.com/elyra-ai/elyra/issues/2919 is fixed
+            if not isinstance(vm, (VolumeMount, dict)):
+                continue
+            pvc_name = vm.pvc_name if isinstance(vm, VolumeMount) else vm.get("pvc_name")
+            describe_dict["volume_dependencies"]["value"].add(pvc_name)
 
         if Operation.is_generic_operation(node.op):
             # update stats that are specific to generic components

elyra/kfp/operator.py

@@ -26,17 +26,10 @@
 from kubernetes.client.models import V1EnvVar
 from kubernetes.client.models import V1EnvVarSource
 from kubernetes.client.models import V1ObjectFieldSelector
-from kubernetes.client.models import V1PersistentVolumeClaimVolumeSource
-from kubernetes.client.models import V1SecretKeySelector
-from kubernetes.client.models import V1Toleration
 from kubernetes.client.models import V1Volume
 from kubernetes.client.models import V1VolumeMount
 
 from elyra._version import __version__
-from elyra.pipeline.pipeline import KubernetesAnnotation
-from elyra.pipeline.pipeline import KubernetesSecret
-from elyra.pipeline.pipeline import KubernetesToleration
-from elyra.pipeline.pipeline import VolumeMount
 
 """
 The ExecuteFileOp uses a python script to bootstrap the user supplied image with the required dependencies.
@@ -92,10 +85,6 @@ def __init__(
         mem_request: Optional[str] = None,
         gpu_limit: Optional[str] = None,
         workflow_engine: Optional[str] = "argo",
-        volume_mounts: Optional[List[VolumeMount]] = None,
-        kubernetes_secrets: Optional[List[KubernetesSecret]] = None,
-        kubernetes_tolerations: Optional[List[KubernetesToleration]] = None,
-        kubernetes_pod_annotations: Optional[List[KubernetesAnnotation]] = None,
         **kwargs,
     ):
         """Create a new instance of ContainerOp.
@@ -119,10 +108,6 @@ def __init__(
           mem_request: memory requested for the operation (in Gi)
           gpu_limit: maximum number of GPUs allowed for the operation
           workflow_engine: Kubeflow workflow engine, defaults to 'argo'
-          volume_mounts: data volumes to be mounted
-          kubernetes_secrets: secrets to be made available as environment variables
-          kubernetes_tolerations: Kubernetes tolerations to be added to the pod
-          kubernetes_pod_annotations: annotations to be applied to the pod
           kwargs: additional key value pairs to pass e.g. name, image, sidecars & is_exit_handler.
                   See Kubeflow pipelines ContainerOp definition for more parameters or how to use
                   https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.dsl.html#kfp.dsl.ContainerOp
@@ -149,12 +134,6 @@ def __init__(
         self.cpu_request = cpu_request
         self.mem_request = mem_request
         self.gpu_limit = gpu_limit
-        self.volume_mounts = volume_mounts  # optional data volumes to be mounted to the pod
-        self.kubernetes_secrets = kubernetes_secrets  # optional secrets to be made available as env vars
-        self.kubernetes_tolerations = (
-            kubernetes_tolerations  # optional Kubernetes tolerations to be attached to the pod
-        )
-        self.kubernetes_pod_annotations = kubernetes_pod_annotations  # optional annotations
 
         argument_list = []
 
@@ -246,55 +225,12 @@ def __init__(
 
         super().__init__(**kwargs)
 
-        # add user-specified volume mounts: the referenced PVCs must exist
-        # or this generic operation will fail
-        if self.volume_mounts:
-            unique_pvcs = []
-            for volume_mount in self.volume_mounts:
-                if volume_mount.pvc_name not in unique_pvcs:
-                    self.add_volume(
-                        V1Volume(
-                            name=volume_mount.pvc_name,
-                            persistent_volume_claim=V1PersistentVolumeClaimVolumeSource(
-                                claim_name=volume_mount.pvc_name
-                            ),
-                        )
-                    )
-                    unique_pvcs.append(volume_mount.pvc_name)
-                self.container.add_volume_mount(V1VolumeMount(mount_path=volume_mount.path, name=volume_mount.pvc_name))
-
         # We must deal with the envs after the superclass initialization since these amend the
         # container attribute that isn't available until now.
         if self.pipeline_envs:
             for key, value in self.pipeline_envs.items():  # Convert dict entries to format kfp needs
                 self.container.add_env_variable(V1EnvVar(name=key, value=value))
 
-        if self.kubernetes_secrets:
-            for secret in self.kubernetes_secrets:  # Convert tuple entries to format kfp needs
-                self.container.add_env_variable(
-                    V1EnvVar(
-                        name=secret.env_var,
-                        value_from=V1EnvVarSource(secret_key_ref=V1SecretKeySelector(name=secret.name, key=secret.key)),
-                    )
-                )
-
-        # add user-provided tolerations
-        if self.kubernetes_tolerations:
-            for toleration in self.kubernetes_tolerations:
-                self.add_toleration(
-                    V1Toleration(
-                        effect=toleration.effect,
-                        key=toleration.key,
-                        operator=toleration.operator,
-                        value=toleration.value,
-                    )
-                )
-
-        # add user-provided annotations to pod
-        if self.kubernetes_pod_annotations:
-            for annotation in self.kubernetes_pod_annotations:
-                self.add_pod_annotation(annotation.key, annotation.value)
-
         # If crio volume size is found then assume kubeflow pipelines environment is using CRI-o as
         # its container runtime
         if self.emptydir_volume_size:

elyra/pipeline/airflow/component_parser_airflow.py

@@ -22,8 +22,8 @@
 
 from elyra.pipeline.catalog_connector import CatalogEntry
 from elyra.pipeline.component import Component
-from elyra.pipeline.component import ComponentParameter
 from elyra.pipeline.component import ComponentParser
+from elyra.pipeline.component_parameter import ComponentParameter
 from elyra.pipeline.runtime_type import RuntimeProcessorType
 
 CONTROL_ID = "OneOfControl"
@@ -361,14 +361,28 @@ def _get_init_arguments(self, init_function: ast.FunctionDef) -> Dict[str, Dict]
                         and isinstance(arg.annotation.slice.value.value, ast.Name)
                     ):
                         # arg is of the form `<arg>: Optional[<multi-valued_type>]`
-                        # e.g. `env = Optional[Dict[str, str]]` or `env = Optional[List[int]]`
+                        # [<multi-valued_type> or `env = Optional[List[int]]`
                         # In Python 3.7 and lower
                         data_type = arg.annotation.slice.value.value.id
 
                     if isinstance(arg.annotation.value, ast.Name) and arg.annotation.value.id == "Optional":
                         # arg typehint includes the phrase 'Optional'
                         required = False
 
+                elif isinstance(arg.annotation, ast.BinOp):
+                    if (
+                        isinstance(arg.annotation.left, ast.Subscript)
+                        and isinstance(arg.annotation.left.value, ast.Name)
+                        and isinstance(arg.annotation.left.value.id, str)
+                    ):
+                        # arg is of the form `<arg>: [<type>] | [<other_type>]`
+                        # e.g. `env = dict[str, str] | None`
+                        data_type = arg.annotation.left.value.id
+
+                    if isinstance(arg.annotation.right, ast.Constant) and arg.annotation.right.value is None:
+                        # arg typehint includes None, making it optional
+                        required = False
+
             # Insert AST-parsed (or default) values into dictionary
             init_arg_dict[arg_name] = {"data_type": data_type, "default_value": default_value, "required": required}