diff --git a/docs/source/user_guide/pipelines.md b/docs/source/user_guide/pipelines.md index 836efcdc5..273580fff 100644 --- a/docs/source/user_guide/pipelines.md +++ b/docs/source/user_guide/pipelines.md @@ -103,7 +103,7 @@ Each pipeline node is configured using properties. Default node properties are a **Default properties that apply only to custom nodes** - - [Disallow cached output](#disallow-cached-output) + - [Disable node caching](#disable-node-caching) #### Adding nodes @@ -163,7 +163,7 @@ Nodes that are implemented using [custom components](pipeline-components.html#cu - [Data volumes](#data-volumes) - [Kubernetes tolerations](#kubernetes-tolerations) - [Kubernetes pod annotations](#kubernetes-pod-annotations) - - [Disallow cached output](#disallow-cached-output) + - [Disable node caching](#disable-node-caching) #### Defining dependencies between nodes @@ -203,19 +203,26 @@ The following alphabetically sorted list identifies the node properties that are ##### Data volumes - A list of [Persistent Volume Claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PVC) to be mounted into the container that executes the component. - - Format: `/mnt/path=existing-pvc-name`. Entries that are empty (`/mnt/path=`) or malformed are ignored. Entries with a PVC name considered to be an [invalid Kubernetes resource name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) will raise a validation error after pipeline submission or export. - - The referenced PVCs must exist in the Kubernetes namespace where the pipeline nodes are executed. + - Format: + - _Mount path_: the path where the PVC shall be mounted in the container. Example: `/mnt/datavol/` + - _Persistent volume claim name_: a valid [Kubernetes resource name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) identifying a PVC that exists in the Kubernetes namespace where the pipeline nodes are executed. Example: `my-data-pvc` - Data volumes are not mounted when the pipeline is executed locally. -##### Disallow cached output +##### Disable node caching - Pipeline nodes produce output, such as files. Some runtime environments support caching of these outputs, eliminating the need to re-execute nodes, which can improve performance and reduce resource usage. If a node does not produce output in a deterministic way - that is, when given the same inputs, the generated output is different - re-using the output from previous executions might lead to unexpected results. + - Format: + - `True` node output is not cached + - `False` node output is cached + - If no behavior is specified, the runtime environment's default caching behavior is applied. - Caching can only be disabled for pipelines that are executed on Kubeflow Pipelines. ##### Environment Variables - This property applies only to generic components. - - A list of environment variables to be set inside in the container. Specify one variable/value pair per line, separated by `=`. + - A list of environment variables to be set inside in the container. + - Format: + - _Environment variable_: name of the variable to be set. Example: `optimize` + - _Value_: the value to be assigned to said variable. Example: `true` - A set of default environment variables can also be set in the pipeline properties tab. If any default environment variables are set, the **Environment Variables** property in the node properties tab will include these variables and their values with a note that each is a pipeline default. Pipeline default environment variables are not editable from the node properties tab. Individual nodes can override a pipeline default value for a given variable by re-defining the variable/value pair in its own node properties. - - Example: `TOKEN=value` ##### File Dependencies - This property applies only to generic components. @@ -230,20 +237,28 @@ The following alphabetically sorted list identifies the node properties that are ##### Kubernetes Pod Annotations - A list of [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#attaching-metadata-to-objects) to be attached to the pod that executes the node. - - Format: `annotation-key=annotation-value`. Entries that are empty (`annotation-key=`) are ignored. Entries with a key considered to be [invalid](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set) will raise a validation error after pipeline submission or export. + - Format: + - _Key_: a [valid Kubernetes annotation key](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). Example: `project` + - _Value_: value to be assigned to said annotation key. Example: `abandoned basket analysis` - Annotations are ignored when the pipeline is executed locally. - - Example: `project=abandoned basket analysis` ##### Kubernetes Secrets - - A list of [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) to be accessed as environment variables during Jupyter notebook or script execution. Format: `ENV_VAR=secret-name:secret-key`. Entries that are empty (`ENV_VAR=`) or malformed are ignored. Entries with a secret name considered to be an [invalid Kubernetes resource name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) or with [an invalid secret key](https://kubernetes.io/docs/concepts/configuration/secret/#restriction-names-data) will raise a validation error after pipeline submission or export. The referenced secrets must exist in the Kubernetes namespace where the generic pipeline nodes are executed. + - A list of [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) to be accessed as environment variables during Jupyter notebook or script execution. + - Format: + - _Environment variable_: name of the variable to be set. Example: `optimize` + - _Secret Name_: a valid [Kubernetes resource name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) identifying a [secret](https://kubernetes.io/docs/concepts/configuration/secret/#restriction-names-data) that exists in the Kubernetes namespace where the pipeline nodes are executed. Example: `database-credentials` + - _Secret Key_: key that is defined in said secret. Example: `uid` - Secrets are ignored when the pipeline is executed locally. For remote execution, if an environment variable was assigned both a static value (via the 'Environment Variables' property) and a Kubernetes secret value, the secret's value is used. - - Example: `ENV_VAR=secret-name:secret-key` ##### Kubernetes Tolerations - A list of [Kubernetes tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to be applied to the pod where the component is executed. - - Format: `TOL_ID=key:operator:value:effect`. Refer to [the toleration specification](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#toleration-v1-core) for a description of the values for `key`, `operator`, `value`, and `effect`. + - Format: + - _Key_: taint key the toleration applies to + - _Operator_: represents the key's relationship to the value. Must be `Equal` or `Exists`. + - _Value_: taint value the toleration matches to + - _Effect_: indicates the taint effect to match. If specified, must be `NoExecute`, `NoSchedule`, or `PreferNoSchedule`. + - Refer to [the toleration specification](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#toleration-v1-core) for a description of each property. - Tolerations are ignored when the pipeline is executed locally. - - Example: `TOL_1=my-key:Exists::NoExecute` ##### Label - Specify a label to replace the default node name. For generic components the default label is the file name. For custom components the default name is the component name. diff --git a/elyra/cli/pipeline_app.py b/elyra/cli/pipeline_app.py index 1e54ea2c3..b3db6f318 100644 --- a/elyra/cli/pipeline_app.py +++ b/elyra/cli/pipeline_app.py @@ -35,6 +35,7 @@ from elyra.metadata.schemaspaces import Runtimes from elyra.pipeline import pipeline_constants from elyra.pipeline.component_catalog import ComponentCache +from elyra.pipeline.component_parameter import VolumeMount from elyra.pipeline.kfp.kfp_authentication import AuthenticationError from elyra.pipeline.kfp.kfp_authentication import KFPAuthenticator from elyra.pipeline.parser import PipelineParser @@ -584,7 +585,11 @@ def describe(json_option, pipeline_path): # (... there are none today) # volumes for vm in node.get_component_parameter(pipeline_constants.MOUNTED_VOLUMES, []): - describe_dict["volume_dependencies"]["value"].add(vm.pvc_name) + # The below is a workaround until https://github.com/elyra-ai/elyra/issues/2919 is fixed + if not isinstance(vm, (VolumeMount, dict)): + continue + pvc_name = vm.pvc_name if isinstance(vm, VolumeMount) else vm.get("pvc_name") + describe_dict["volume_dependencies"]["value"].add(pvc_name) if Operation.is_generic_operation(node.op): # update stats that are specific to generic components diff --git a/elyra/kfp/operator.py b/elyra/kfp/operator.py index 4aaeedca3..36b26165b 100644 --- a/elyra/kfp/operator.py +++ b/elyra/kfp/operator.py @@ -26,17 +26,10 @@ from kubernetes.client.models import V1EnvVar from kubernetes.client.models import V1EnvVarSource from kubernetes.client.models import V1ObjectFieldSelector -from kubernetes.client.models import V1PersistentVolumeClaimVolumeSource -from kubernetes.client.models import V1SecretKeySelector -from kubernetes.client.models import V1Toleration from kubernetes.client.models import V1Volume from kubernetes.client.models import V1VolumeMount from elyra._version import __version__ -from elyra.pipeline.pipeline import KubernetesAnnotation -from elyra.pipeline.pipeline import KubernetesSecret -from elyra.pipeline.pipeline import KubernetesToleration -from elyra.pipeline.pipeline import VolumeMount """ The ExecuteFileOp uses a python script to bootstrap the user supplied image with the required dependencies. @@ -92,10 +85,6 @@ def __init__( mem_request: Optional[str] = None, gpu_limit: Optional[str] = None, workflow_engine: Optional[str] = "argo", - volume_mounts: Optional[List[VolumeMount]] = None, - kubernetes_secrets: Optional[List[KubernetesSecret]] = None, - kubernetes_tolerations: Optional[List[KubernetesToleration]] = None, - kubernetes_pod_annotations: Optional[List[KubernetesAnnotation]] = None, **kwargs, ): """Create a new instance of ContainerOp. @@ -119,10 +108,6 @@ def __init__( mem_request: memory requested for the operation (in Gi) gpu_limit: maximum number of GPUs allowed for the operation workflow_engine: Kubeflow workflow engine, defaults to 'argo' - volume_mounts: data volumes to be mounted - kubernetes_secrets: secrets to be made available as environment variables - kubernetes_tolerations: Kubernetes tolerations to be added to the pod - kubernetes_pod_annotations: annotations to be applied to the pod kwargs: additional key value pairs to pass e.g. name, image, sidecars & is_exit_handler. See Kubeflow pipelines ContainerOp definition for more parameters or how to use https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.dsl.html#kfp.dsl.ContainerOp @@ -149,12 +134,6 @@ def __init__( self.cpu_request = cpu_request self.mem_request = mem_request self.gpu_limit = gpu_limit - self.volume_mounts = volume_mounts # optional data volumes to be mounted to the pod - self.kubernetes_secrets = kubernetes_secrets # optional secrets to be made available as env vars - self.kubernetes_tolerations = ( - kubernetes_tolerations # optional Kubernetes tolerations to be attached to the pod - ) - self.kubernetes_pod_annotations = kubernetes_pod_annotations # optional annotations argument_list = [] @@ -246,55 +225,12 @@ def __init__( super().__init__(**kwargs) - # add user-specified volume mounts: the referenced PVCs must exist - # or this generic operation will fail - if self.volume_mounts: - unique_pvcs = [] - for volume_mount in self.volume_mounts: - if volume_mount.pvc_name not in unique_pvcs: - self.add_volume( - V1Volume( - name=volume_mount.pvc_name, - persistent_volume_claim=V1PersistentVolumeClaimVolumeSource( - claim_name=volume_mount.pvc_name - ), - ) - ) - unique_pvcs.append(volume_mount.pvc_name) - self.container.add_volume_mount(V1VolumeMount(mount_path=volume_mount.path, name=volume_mount.pvc_name)) - # We must deal with the envs after the superclass initialization since these amend the # container attribute that isn't available until now. if self.pipeline_envs: for key, value in self.pipeline_envs.items(): # Convert dict entries to format kfp needs self.container.add_env_variable(V1EnvVar(name=key, value=value)) - if self.kubernetes_secrets: - for secret in self.kubernetes_secrets: # Convert tuple entries to format kfp needs - self.container.add_env_variable( - V1EnvVar( - name=secret.env_var, - value_from=V1EnvVarSource(secret_key_ref=V1SecretKeySelector(name=secret.name, key=secret.key)), - ) - ) - - # add user-provided tolerations - if self.kubernetes_tolerations: - for toleration in self.kubernetes_tolerations: - self.add_toleration( - V1Toleration( - effect=toleration.effect, - key=toleration.key, - operator=toleration.operator, - value=toleration.value, - ) - ) - - # add user-provided annotations to pod - if self.kubernetes_pod_annotations: - for annotation in self.kubernetes_pod_annotations: - self.add_pod_annotation(annotation.key, annotation.value) - # If crio volume size is found then assume kubeflow pipelines environment is using CRI-o as # its container runtime if self.emptydir_volume_size: diff --git a/elyra/pipeline/airflow/component_parser_airflow.py b/elyra/pipeline/airflow/component_parser_airflow.py index cc0704ffe..6c89db18e 100644 --- a/elyra/pipeline/airflow/component_parser_airflow.py +++ b/elyra/pipeline/airflow/component_parser_airflow.py @@ -22,8 +22,8 @@ from elyra.pipeline.catalog_connector import CatalogEntry from elyra.pipeline.component import Component -from elyra.pipeline.component import ComponentParameter from elyra.pipeline.component import ComponentParser +from elyra.pipeline.component_parameter import ComponentParameter from elyra.pipeline.runtime_type import RuntimeProcessorType CONTROL_ID = "OneOfControl" @@ -361,7 +361,7 @@ def _get_init_arguments(self, init_function: ast.FunctionDef) -> Dict[str, Dict] and isinstance(arg.annotation.slice.value.value, ast.Name) ): # arg is of the form `: Optional[]` - # e.g. `env = Optional[Dict[str, str]]` or `env = Optional[List[int]]` + # [ or `env = Optional[List[int]]` # In Python 3.7 and lower data_type = arg.annotation.slice.value.value.id @@ -369,6 +369,20 @@ def _get_init_arguments(self, init_function: ast.FunctionDef) -> Dict[str, Dict] # arg typehint includes the phrase 'Optional' required = False + elif isinstance(arg.annotation, ast.BinOp): + if ( + isinstance(arg.annotation.left, ast.Subscript) + and isinstance(arg.annotation.left.value, ast.Name) + and isinstance(arg.annotation.left.value.id, str) + ): + # arg is of the form `: [] | []` + # e.g. `env = dict[str, str] | None` + data_type = arg.annotation.left.value.id + + if isinstance(arg.annotation.right, ast.Constant) and arg.annotation.right.value is None: + # arg typehint includes None, making it optional + required = False + # Insert AST-parsed (or default) values into dictionary init_arg_dict[arg_name] = {"data_type": data_type, "default_value": default_value, "required": required} diff --git a/elyra/pipeline/airflow/processor_airflow.py b/elyra/pipeline/airflow/processor_airflow.py index 79eacd28a..b94dcd3de 100644 --- a/elyra/pipeline/airflow/processor_airflow.py +++ b/elyra/pipeline/airflow/processor_airflow.py @@ -22,9 +22,11 @@ import tempfile from textwrap import dedent import time +from typing import Any from typing import Dict from typing import List from typing import Optional +from typing import Set from typing import Union import autopep8 @@ -37,11 +39,16 @@ from elyra.airflow.operator import BootscriptBuilder from elyra.metadata.schemaspaces import RuntimeImages from elyra.metadata.schemaspaces import Runtimes +from elyra.pipeline import pipeline_constants from elyra.pipeline.component_catalog import ComponentCache +from elyra.pipeline.component_parameter import ElyraProperty +from elyra.pipeline.component_parameter import ElyraPropertyList +from elyra.pipeline.component_parameter import KubernetesAnnotation +from elyra.pipeline.component_parameter import KubernetesToleration +from elyra.pipeline.component_parameter import VolumeMount from elyra.pipeline.pipeline import GenericOperation from elyra.pipeline.pipeline import Operation from elyra.pipeline.pipeline import Pipeline -from elyra.pipeline.pipeline_constants import COS_OBJECT_PREFIX from elyra.pipeline.processor import PipelineProcessor from elyra.pipeline.processor import RuntimePipelineProcessor from elyra.pipeline.processor import RuntimePipelineProcessorResponse @@ -178,7 +185,9 @@ def process(self, pipeline: Pipeline) -> None: if pipeline.contains_generic_operations(): object_storage_url = f"{cos_endpoint}" - os_path = join_paths(pipeline.pipeline_parameters.get(COS_OBJECT_PREFIX), pipeline_instance_id) + os_path = join_paths( + pipeline.pipeline_parameters.get(pipeline_constants.COS_OBJECT_PREFIX), pipeline_instance_id + ) object_storage_path = f"/{cos_bucket}/{os_path}" else: object_storage_url = None @@ -241,7 +250,9 @@ def _cc_pipeline(self, pipeline: Pipeline, pipeline_name: str, pipeline_instance cos_bucket = runtime_configuration.metadata.get("cos_bucket") pipeline_instance_id = pipeline_instance_id or pipeline_name - artifact_object_prefix = join_paths(pipeline.pipeline_parameters.get(COS_OBJECT_PREFIX), pipeline_instance_id) + artifact_object_prefix = join_paths( + pipeline.pipeline_parameters.get(pipeline_constants.COS_OBJECT_PREFIX), pipeline_instance_id + ) self.log_pipeline_info( pipeline_name, @@ -332,19 +343,11 @@ def _cc_pipeline(self, pipeline: Pipeline, pipeline_name: str, pipeline_instance "mem_request": operation.memory, "gpu_limit": operation.gpu, "operator_source": operation.filename, - "is_generic_operator": operation.is_generic, - "doc": operation.doc, - "volumes": operation.mounted_volumes, - "secrets": operation.kubernetes_secrets, - "kubernetes_tolerations": operation.kubernetes_tolerations, - "kubernetes_pod_annotations": operation.kubernetes_pod_annotations, } if runtime_image_pull_secret is not None: target_op["runtime_image_pull_secret"] = runtime_image_pull_secret - target_ops.append(target_op) - self.log_pipeline_info( pipeline_name, f"processing operation dependencies for id '{operation.id}'", @@ -442,14 +445,16 @@ def _cc_pipeline(self, pipeline: Pipeline, pipeline_name: str, pipeline_instance "parent_operation_ids": operation.parent_operation_ids, "component_params": operation.component_params_as_dict, "operator_source": component.component_source, + } + + target_op.update( + { "is_generic_operator": operation.is_generic, "doc": operation.doc, - "volumes": operation.mounted_volumes, - "kubernetes_tolerations": operation.kubernetes_tolerations, - "kubernetes_pod_annotations": operation.kubernetes_pod_annotations, + "elyra_params": operation.elyra_params, } - - target_ops.append(target_op) + ) + target_ops.append(target_op) ordered_target_ops = OrderedDict() @@ -493,16 +498,6 @@ def create_pipeline_file( template_env.filters["regex_replace"] = lambda x: AirflowPipelineProcessor.scrub_invalid_characters(x) template = template_env.get_template("airflow_template.jinja2") - # Pass functions used to render data volumes and secrets to the template env - rendering_functions = { - "render_volumes_for_generic_op": AirflowPipelineProcessor.render_volumes_for_generic_op, - "render_executor_config_for_custom_op": AirflowPipelineProcessor.render_executor_config_for_custom_op, - "render_secrets_for_generic_op": AirflowPipelineProcessor.render_secrets_for_generic_op, - "render_secrets_for_cos": AirflowPipelineProcessor.render_secrets_for_cos, - "render_executor_config_for_generic_op": AirflowPipelineProcessor.render_executor_config_for_generic_op, - } - template.globals.update(rendering_functions) - ordered_ops = self._cc_pipeline(pipeline, pipeline_name, pipeline_instance_id) runtime_configuration = self._get_metadata_configuration( schemaspace=Runtimes.RUNTIMES_SCHEMASPACE_ID, name=pipeline.runtime_config @@ -518,11 +513,12 @@ def create_pipeline_file( operations_list=ordered_ops, pipeline_name=pipeline_instance_id, user_namespace=user_namespace, - cos_secret=cos_secret, + cos_secret=cos_secret if any(op.get("is_generic_operator") for op in ordered_ops.values()) else None, kube_config_path=None, is_paused_upon_creation="False", in_cluster="True", pipeline_description=pipeline_description, + processor=self, ) # Write to python file and fix formatting @@ -591,187 +587,122 @@ def _get_node_name(self, operations_list: list, node_id: str) -> Optional[str]: return operation["notebook"] return None - @staticmethod - def render_volumes_for_generic_op(op: Dict) -> str: + def render_volumes(self, elyra_parameters: Dict[str, ElyraProperty]) -> str: """ - Render any data volumes defined for the specified generic op for use in - the Airflow DAG template - + Render volumes defined for an operation for use in the Airflow DAG template :returns: a string literal containing the python code to be rendered in the DAG """ - if not op.get("volumes"): - return "" - op["volume_vars"] = [] # store variable names in op's dict for template to access - - # Include import statements and comment - str_to_render = f""" - from airflow.contrib.kubernetes.volume import Volume - from airflow.contrib.kubernetes.volume_mount import VolumeMount - # Volumes and mounts for operation '{op['id']}'""" - for idx, volume in enumerate(op.get("volumes", [])): - var_name = AirflowPipelineProcessor.scrub_invalid_characters(f"volume_{op['id']}_{idx}") - - # Define VolumeMount and Volume objects + str_to_render = "" + for v in elyra_parameters.get(pipeline_constants.MOUNTED_VOLUMES, []): str_to_render += f""" - mount_{var_name} = VolumeMount( - name='{volume.pvc_name}', - mount_path='{volume.path}', - sub_path=None, - read_only=False - ) - {var_name} = Volume( - name='{volume.pvc_name}', configs={{"persistentVolumeClaim": {{"claimName": "{volume.pvc_name}"}}}} - ) - """ - - op["volume_vars"].append(var_name) - - op["volume_mount_vars"] = [f"mount_{volume_var}" for volume_var in op["volume_vars"]] + Volume(name="{v.pvc_name}", configs={{"persistentVolumeClaim": {{"claimName": "{v.pvc_name}"}}}}),""" return dedent(str_to_render) - @staticmethod - def render_executor_config_for_custom_op(op: Dict) -> Dict[str, Dict[str, List]]: + def render_mounts(self, elyra_parameters: Dict[str, ElyraProperty]) -> str: """ - Render any data volumes or tolerations defined for the specified custom op - for use in the Airflow DAG template - - :returns: a dict defining the volumes and mounts to be rendered in the DAG + Render volume mounts defined for an operation for use in the Airflow DAG template + :returns: a string literal containing the python code to be rendered in the DAG """ - executor_config = {"KubernetesExecutor": {}} - - # Handle volume mounts - if op.get("volumes"): - executor_config["KubernetesExecutor"]["volumes"] = [] - executor_config["KubernetesExecutor"]["volume_mounts"] = [] - for volume in op.get("volumes", []): - # Add volume and volume mount entry - executor_config["KubernetesExecutor"]["volumes"].append( - { - "name": volume.pvc_name, - "persistentVolumeClaim": {"claimName": volume.pvc_name}, - } - ) - executor_config["KubernetesExecutor"]["volume_mounts"].append( - {"mountPath": volume.path, "name": volume.pvc_name, "read_only": False} - ) - - # Handle tolerations - if op.get("kubernetes_tolerations"): - executor_config["KubernetesExecutor"]["tolerations"] = [] - for toleration in op.get("kubernetes_tolerations", []): - # Add Kubernetes toleration entry - executor_config["KubernetesExecutor"]["tolerations"].append( - { - "key": toleration.key, - "operator": toleration.operator, - "value": toleration.value, - "effect": toleration.effect, - } - ) - - # Handle annotations - if op.get("kubernetes_pod_annotations"): - executor_config["KubernetesExecutor"]["annotations"] = {} - for annotation in op.get("kubernetes_pod_annotations", []): - # Add Kubernetes annotation entry - executor_config["KubernetesExecutor"]["annotations"][annotation.key] = annotation.value - - return executor_config + str_to_render = "" + for v in elyra_parameters.get(pipeline_constants.MOUNTED_VOLUMES, []): + str_to_render += f""" + VolumeMount(name="{v.pvc_name}", mount_path="{v.path}", sub_path=None, read_only=False),""" + return dedent(str_to_render) - @staticmethod - def render_executor_config_for_generic_op(op: Dict) -> Dict[str, Dict[str, List]]: + def render_secrets(self, elyra_parameters: Dict[str, ElyraProperty], cos_secret: Optional[str]) -> str: """ - Render tolerations and annotations defined for the specified generic op - for use in the Airflow DAG template - - :returns: a dict defining the tolerations and annotations to be rendered in the DAG + Render Kubernetes secrets defined for an operation for use in the Airflow DAG template + :returns: a string literal containing the python code to be rendered in the DAG """ - executor_config = {"KubernetesExecutor": {}} - - # Handle tolerations - if op.get("kubernetes_tolerations"): - executor_config["KubernetesExecutor"]["tolerations"] = [] - for toleration in op.get("kubernetes_tolerations", []): - # Add Kubernetes toleration entry - executor_config["KubernetesExecutor"]["tolerations"].append( - { - "key": toleration.key, - "operator": toleration.operator, - "value": toleration.value, - "effect": toleration.effect, - } - ) - - # Handle annotations - if op.get("kubernetes_pod_annotations"): - executor_config["KubernetesExecutor"]["annotations"] = {} - for annotation in op.get("kubernetes_pod_annotations", []): - # Add Kubernetes annotation entry - executor_config["KubernetesExecutor"]["annotations"][annotation.key] = annotation.value - - return executor_config + str_to_render = "" + if cos_secret: + str_to_render += f""" + Secret("env", "AWS_ACCESS_KEY_ID", "{cos_secret}", "AWS_ACCESS_KEY_ID"), + Secret("env", "AWS_SECRET_ACCESS_KEY", "{cos_secret}", "AWS_SECRET_ACCESS_KEY"),""" + for s in elyra_parameters.get(pipeline_constants.KUBERNETES_SECRETS, []): + str_to_render += f""" + Secret("env", "{s.env_var}", "{s.name}", "{s.key}"),""" + return dedent(str_to_render) - @staticmethod - def render_secrets_for_cos(cos_secret: str): + def render_annotations(self, elyra_parameters: Dict[str, ElyraProperty]) -> Dict: """ - Render the Kubernetes secrets required for COS - - :returns: a string literal of the python code to be rendered in the DAG + Render Kubernetes annotations defined for an operation for use in the Airflow DAG template + :returns: a string literal containing the python code to be rendered in the DAG """ - if not cos_secret: - return "" - - return dedent( - f""" - from airflow.kubernetes.secret import Secret - ## Ensure that the secret '{cos_secret}' is defined in the Kubernetes namespace where the pipeline is run - env_var_secret_id = Secret( - deploy_type="env", - deploy_target="AWS_ACCESS_KEY_ID", - secret="{cos_secret}", - key="AWS_ACCESS_KEY_ID", - ) - env_var_secret_key = Secret( - deploy_type="env", - deploy_target="AWS_SECRET_ACCESS_KEY", - secret="{cos_secret}", - key="AWS_SECRET_ACCESS_KEY", - ) - """ - ) + annotations = elyra_parameters.get(pipeline_constants.KUBERNETES_POD_ANNOTATIONS, ElyraPropertyList([])) + return annotations.to_dict() - @staticmethod - def render_secrets_for_generic_op(op: Dict) -> str: + def render_tolerations(self, elyra_parameters: Dict[str, ElyraProperty]): """ - Render any Kubernetes secrets defined for the specified op for use in - the Airflow DAG template - + Render any Kubernetes tolerations defined for an operation for use in the Airflow DAG template :returns: a string literal containing the python code to be rendered in the DAG """ - if not op.get("secrets"): - return "" - op["secret_vars"] = [] # store variable names in op's dict for template to access - - # Include import statement and comment - str_to_render = f""" - from airflow.kubernetes.secret import Secret - # Secrets for operation '{op['id']}'""" - for idx, secret in enumerate(op.get("secrets", [])): - var_name = AirflowPipelineProcessor.scrub_invalid_characters(f"secret_{op['id']}_{idx}") - - # Define Secret object + str_to_render = "" + for t in elyra_parameters.get(pipeline_constants.KUBERNETES_TOLERATIONS, []): + key = f'"{t.key}"' if t.key is not None else t.key + value = f'"{t.value}"' if t.value is not None else t.value + effect = f'"{t.effect}"' if t.value is not None else t.effect str_to_render += f""" - {var_name} = Secret( - deploy_type='env', - deploy_target="{secret.env_var}", - secret="{secret.name}", - key="{secret.key}", - ) - """ - - op["secret_vars"].append(var_name) + {{"key": {key}, "operator": "{t.operator}", "value": {value}, "effect": {effect}}},""" return dedent(str_to_render) + def render_elyra_owned_properties(self, elyra_parameters: Dict[str, ElyraProperty]): + """ + Build the KubernetesExecutor object for the given operation for use in the DAG. + """ + kubernetes_executor = {} + for value in elyra_parameters.values(): + if isinstance(value, (ElyraProperty, ElyraPropertyList)): + value.add_to_execution_object(runtime_processor=self, execution_object=kubernetes_executor) + + return {"KubernetesExecutor": kubernetes_executor} if kubernetes_executor else {} + + def add_mounted_volume(self, instance: VolumeMount, execution_object: Any, **kwargs) -> None: + """Add VolumeMount instance to the execution object for the given runtime processor""" + if "volumes" not in execution_object: + execution_object["volumes"] = [] + if "volume_mounts" not in execution_object: + execution_object["volume_mounts"] = [] + execution_object["volumes"].append( + { + "name": instance.pvc_name, + "persistentVolumeClaim": {"claimName": instance.pvc_name}, + } + ) + execution_object["volume_mounts"].append( + {"mountPath": instance.path, "name": instance.pvc_name, "read_only": False} + ) + + def add_kubernetes_pod_annotation(self, instance: KubernetesAnnotation, execution_object: Any, **kwargs) -> None: + """Add KubernetesAnnotation instance to the execution object for the given runtime processor""" + if "annotations" not in execution_object: + execution_object["annotations"] = {} + execution_object["annotations"][instance.key] = instance.value + + def add_kubernetes_toleration(self, instance: KubernetesToleration, execution_object: Any, **kwargs) -> None: + """Add KubernetesToleration instance to the execution object for the given runtime processor""" + if "tolerations" not in execution_object: + execution_object["tolerations"] = [] + execution_object["tolerations"].append( + { + "key": instance.key, + "operator": instance.operator, + "value": instance.value, + "effect": instance.effect, + } + ) + + @property + def supported_properties(self) -> Set[str]: + """A list of Elyra-owned properties supported by this runtime processor.""" + return [ + pipeline_constants.ENV_VARIABLES, + pipeline_constants.KUBERNETES_SECRETS, + pipeline_constants.MOUNTED_VOLUMES, + pipeline_constants.KUBERNETES_POD_ANNOTATIONS, + pipeline_constants.KUBERNETES_TOLERATIONS, + ] + class AirflowPipelineProcessorResponse(RuntimePipelineProcessorResponse): diff --git a/elyra/pipeline/catalog_connector.py b/elyra/pipeline/catalog_connector.py index 21dce3270..d0a85a229 100644 --- a/elyra/pipeline/catalog_connector.py +++ b/elyra/pipeline/catalog_connector.py @@ -13,6 +13,8 @@ # See the License for the specific language governing permissions and # limitations under the License. # +from __future__ import annotations + from abc import abstractmethod from copy import deepcopy import hashlib @@ -39,7 +41,7 @@ from elyra._version import __version__ from elyra.metadata.metadata import Metadata from elyra.pipeline.component import Component -from elyra.pipeline.component import ComponentParameter +from elyra.pipeline.component_parameter import ComponentParameter from elyra.pipeline.runtime_type import RuntimeProcessorType from elyra.util.url import FileTransportAdapter from elyra.util.url import get_verify_parm diff --git a/elyra/pipeline/component.py b/elyra/pipeline/component.py index 02ebaf429..743534e2e 100644 --- a/elyra/pipeline/component.py +++ b/elyra/pipeline/component.py @@ -13,9 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. # +from __future__ import annotations + from abc import abstractmethod from dataclasses import dataclass -from enum import Enum from importlib import import_module import json from logging import Logger @@ -26,6 +27,10 @@ from traitlets.config import LoggingConfigurable +from elyra.pipeline.component_parameter import ComponentParameter +from elyra.pipeline.component_parameter import ElyraProperty +from elyra.pipeline.runtime_type import RuntimeProcessorType + # Rather than importing only the CatalogEntry class needed in the Component parse # type hint below, the catalog_connector module must be imported in its # entirety in order to avoid a circular reference issue @@ -34,203 +39,7 @@ except ImportError: import sys - catalog_connector = sys.modules[__package__ + ".catalog_connector"] -from elyra.pipeline.runtime_type import RuntimeProcessorType - - -class ComponentParameter(object): - """ - Represents a single property for a pipeline component - """ - - def __init__( - self, - id: str, - name: str, - json_data_type: str, - value: Any, - description: str, - allowed_input_types: List[Optional[str]] = None, - required: Optional[bool] = False, - allow_no_options: Optional[bool] = False, - items: Optional[List[str]] = None, - ): - """ - :param id: Unique identifier for a property - :param name: The name of the property for display - :param json_data_type: The JSON data type that represents this parameters value - :param allowed_input_types: The input types that the property can accept, including those for custom rendering - :param value: The default value of the property - :param description: A description of the property for display - :param items: For properties with a control of 'EnumControl', the items making up the enum - :param required: Whether the property is required - :param allow_no_options: Specifies whether to allow parent nodes that don't specifically - define output properties to be selected as input to this node parameter - """ - - if not id: - raise ValueError("Invalid component: Missing field 'id'.") - if not name: - raise ValueError("Invalid component: Missing field 'name'.") - - self._ref = id - self._name = name - self._json_data_type = json_data_type - - # The JSON type that the value entered for this property will be rendered in. - # E.g., array types are entered by users and processed by the backend as - # strings whereas boolean types are entered and processed as booleans - self._value_entry_type = json_data_type - if json_data_type in ["array", "object"]: - self._value_entry_type = "string" - - if json_data_type == "boolean" and isinstance(value, str): - value = bool(value in ["True", "true"]) - elif json_data_type == "number" and isinstance(value, str): - try: - # Attempt to coerce string to integer value - value = int(value) - except ValueError: - # Value could not be coerced to integer, assume float - value = float(value) - if json_data_type in ["array", "object"] and not isinstance(value, str): - value = str(value) - self._value = value - - self._description = description - - if not allowed_input_types: - allowed_input_types = ["inputvalue", "inputpath", "file"] - self._allowed_input_types = allowed_input_types - - self._items = items or [] - - # Check description for information about 'required' parameter - if "not optional" in description.lower() or ( - "required" in description.lower() - and "not required" not in description.lower() - and "n't required" not in description.lower() - ): - required = True - - self._required = required - self._allow_no_options = allow_no_options - - @property - def ref(self) -> str: - return self._ref - - @property - def name(self) -> str: - return self._name - - @property - def allowed_input_types(self) -> List[Optional[str]]: - return self._allowed_input_types - - @property - def json_data_type(self) -> str: - return self._json_data_type - - @property - def value_entry_type(self) -> str: - return self._value_entry_type - - @property - def value(self) -> Any: - return self._value - - @property - def description(self) -> str: - return self._description - - @property - def items(self) -> List[str]: - return self._items - - @property - def required(self) -> bool: - return bool(self._required) - - @property - def allow_no_options(self) -> bool: - return self._allow_no_options - - @staticmethod - def render_parameter_details(param: "ComponentParameter") -> str: - """ - Render the parameter data type and UI hints needed for the specified param for - use in the custom component properties DAG template - - :returns: a string literal containing the JSON object to be rendered in the DAG - """ - json_dict = {"title": param.name, "description": param.description} - if len(param.allowed_input_types) == 1: - # Parameter only accepts a single type of input - input_type = param.allowed_input_types[0] - if not input_type: - # This is an output - json_dict["type"] = "string" - json_dict["uihints"] = {"ui:widget": "hidden", "outputpath": True} - elif input_type == "inputpath": - json_dict.update( - { - "type": "object", - "properties": { - "widget": {"type": "string", "default": input_type}, - "value": {"type": "string", "enum": []}, - }, - "uihints": {"widget": {"ui:field": "hidden"}, "value": {input_type: True}}, - } - ) - elif input_type == "file": - json_dict["type"] = "string" - json_dict["uihints"] = {"ui:widget": input_type} - else: - json_dict["type"] = param.value_entry_type - - # Render default value if it is not None - if param.value is not None: - json_dict["default"] = param.value - else: - # Parameter accepts multiple types of inputs; render a oneOf block - one_of = [] - for widget_type in param.allowed_input_types: - obj = { - "type": "object", - "properties": {"widget": {"type": "string"}, "value": {}}, - "uihints": {"widget": {"ui:widget": "hidden"}, "value": {}}, - } - if widget_type == "inputvalue": - obj["title"] = InputTypeDescriptionMap[param.value_entry_type].value - obj["properties"]["widget"]["default"] = param.value_entry_type - obj["properties"]["value"]["type"] = param.value_entry_type - if param.value_entry_type == "boolean": - obj["properties"]["value"]["title"] = " " - - # Render default value if it is not None - if param.value is not None: - obj["properties"]["value"]["default"] = param.value - else: # inputpath or file types - obj["title"] = InputTypeDescriptionMap[widget_type].value - obj["properties"]["widget"]["default"] = widget_type - if widget_type == "outputpath": - obj["uihints"]["value"] = {"ui:readonly": "true", widget_type: True} - obj["properties"]["value"]["type"] = "string" - elif widget_type == "inputpath": - obj["uihints"]["value"] = {widget_type: True} - obj["properties"]["value"]["type"] = "string" - obj["properties"]["value"]["enum"] = [] - if param.allow_no_options: - obj["uihints"]["allownooptions"] = param.allow_no_options - else: - obj["uihints"]["value"] = {"ui:widget": widget_type} - obj["properties"]["value"]["type"] = "string" - - one_of.append(obj) - json_dict["oneOf"] = one_of - - return json.dumps(json_dict) + catalog_connector = sys.modules[f"{__package__}.catalog_connector"] class Component(object): @@ -345,10 +154,7 @@ def runtime_type(self) -> Optional[RuntimeProcessorType]: @property def op(self) -> Optional[str]: - if self._op: - return self._op - else: - return self._id + return self._op or self._id @property def categories(self) -> List[str]: @@ -368,9 +174,7 @@ def parameter_refs(self) -> dict: @property def import_statement(self) -> Optional[str]: - if not self._package_name: - return None - return f"from {self._package_name} import {self._name}" + return f"from {self._package_name} import {self._name}" if self._package_name else None @property def input_properties(self) -> List[ComponentParameter]: @@ -399,6 +203,20 @@ def _log_warning(msg: str, logger: Optional[Logger] = None): else: print(f"WARNING: {msg}") + def get_elyra_parameters(self) -> List[ComponentParameter]: + """ + Retrieve the list of Elyra-owned ComponentParameters that apply to this + component, removing any whose id collides with a property parsed from + the component definition. + """ + op_type = "generic" if self.component_reference == "elyra" else "custom" + elyra_params = ElyraProperty.get_classes_for_component_type(op_type, self.runtime_type) + if self.properties: + # Remove certain Elyra-owned parameters if a parameter of the same id is already present + parsed_property_ids = [param.ref for param in self.properties] + elyra_params = [param for param in elyra_params if param.property_id not in parsed_property_ids] + return elyra_params + class ComponentParser(LoggingConfigurable): # ABC component_platform: RuntimeProcessorType = None @@ -409,7 +227,7 @@ class ComponentParser(LoggingConfigurable): # ABC } @classmethod - def create_instance(cls, platform: RuntimeProcessorType) -> "ComponentParser": + def create_instance(cls, platform: RuntimeProcessorType) -> ComponentParser: """ Class method that creates the appropriate instance of ComponentParser based on platform type name. """ @@ -425,7 +243,7 @@ def file_types(self) -> List[str]: return self._file_types @abstractmethod - def parse(self, catalog_entry: "catalog_connector.CatalogEntry") -> Optional[List[Component]]: + def parse(self, catalog_entry: catalog_connector.CatalogEntry) -> Optional[List[Component]]: """ Parse a component definition given in the catalog entry and return a list of fully-qualified Component objects @@ -492,17 +310,6 @@ def determine_type_information(self, parsed_type: str) -> "ParameterTypeInfo": return data_type_info -class InputTypeDescriptionMap(Enum): - """A mapping of input types to the description that will appear in the UI""" - - string = "Please enter a string value:" - number = "Please enter a number value:" - boolean = "Please select or deselect the checkbox:" - file = "Please select a file to use as input:" - inputpath = "Please select an output from a parent:" - outputpath = None # outputs are read-only and don't require a description - - @dataclass class ParameterTypeInfo: """ diff --git a/elyra/pipeline/component_catalog.py b/elyra/pipeline/component_catalog.py index ccb511b56..4ecdde630 100644 --- a/elyra/pipeline/component_catalog.py +++ b/elyra/pipeline/component_catalog.py @@ -41,10 +41,9 @@ from elyra.metadata.schemaspaces import ComponentCatalogs from elyra.pipeline.catalog_connector import ComponentCatalogConnector from elyra.pipeline.component import Component -from elyra.pipeline.component import ComponentParameter from elyra.pipeline.component import ComponentParser from elyra.pipeline.component_metadata import ComponentCatalogMetadata -from elyra.pipeline.pipeline_constants import ELYRA_COMPONENT_PROPERTIES +from elyra.pipeline.component_parameter import ComponentParameter from elyra.pipeline.runtime_type import RuntimeProcessorType BLOCKING_TIMEOUT = 0.5 @@ -614,6 +613,13 @@ def get_generic_components() -> List[Component]: def get_generic_component(component_id: str) -> Optional[Component]: return ComponentCache._generic_components.get(component_id) + @staticmethod + def get_generic_component_from_op(component_op: str) -> Optional[Component]: + for component in ComponentCache.get_generic_components(): + if component.op == component_op: + return component + return None + @staticmethod def get_generic_component_ops() -> List[str]: return [component.op for component in ComponentCache.get_generic_components()] @@ -626,6 +632,7 @@ def load_jinja_template(template_name: str) -> Template: """ loader = PackageLoader("elyra", "templates/components") template_env = Environment(loader=loader) + template_env.policies["json.dumps_kwargs"] = {"sort_keys": False} # prevent automatic key sort on 'tojson' return template_env.get_template(template_name) @@ -664,31 +671,20 @@ def to_canvas_palette(components: List[Component]) -> Dict: @staticmethod def to_canvas_properties(component: Component) -> Dict: """ - Converts catalog components into appropriate canvas properties format + Converts catalog components into appropriate canvas properties format. If component_id is one of the generic set, generic template is rendered, - otherwise, the runtime-specific property template is rendered + otherwise, the runtime-specific property template is rendered. """ - template_vars = {} if ComponentCache.get_generic_component(component.id) is not None: template = ComponentCache.load_jinja_template("generic_properties_template.jinja2") else: - # Determine which component properties parsed from the definition - # collide with Elyra-defined properties (in the case of a collision, - # only the parsed property will be displayed) - property_collisions_list = [] - for param in component.properties: - if param.ref in ELYRA_COMPONENT_PROPERTIES: - property_collisions_list.append(param.ref) - - template_vars["elyra_property_collisions_list"] = property_collisions_list - - if len(property_collisions_list) != len(ELYRA_COMPONENT_PROPERTIES): - template_vars["additional_properties_apply"] = True - - template_vars["render_parameter_details"] = ComponentParameter.render_parameter_details template = ComponentCache.load_jinja_template("canvas_properties_template.jinja2") + template_vars = { + "elyra_owned_parameters": component.get_elyra_parameters(), + "render_parameter_details": ComponentParameter.render_parameter_details, + } template.globals.update(template_vars) canvas_properties = template.render(component=component) return json.loads(canvas_properties) diff --git a/elyra/pipeline/component_parameter.py b/elyra/pipeline/component_parameter.py new file mode 100644 index 000000000..295c6832f --- /dev/null +++ b/elyra/pipeline/component_parameter.py @@ -0,0 +1,823 @@ +# +# Copyright 2018-2022 Elyra Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +from __future__ import annotations + +from enum import Enum +import json +from textwrap import dedent +from typing import Any +from typing import Dict +from typing import List +from typing import Optional +from typing import Set +from typing import TYPE_CHECKING + +# Prevent a circular reference by importing RuntimePipelineProcessor only during type-checking +if TYPE_CHECKING: + from elyra.pipeline.processor import RuntimePipelineProcessor + +from elyra.pipeline.pipeline_constants import DISABLE_NODE_CACHING +from elyra.pipeline.pipeline_constants import ENV_VARIABLES +from elyra.pipeline.pipeline_constants import KUBERNETES_POD_ANNOTATIONS +from elyra.pipeline.pipeline_constants import KUBERNETES_SECRETS +from elyra.pipeline.pipeline_constants import KUBERNETES_TOLERATIONS +from elyra.pipeline.pipeline_constants import MOUNTED_VOLUMES +from elyra.util.kubernetes import is_valid_annotation_key +from elyra.util.kubernetes import is_valid_kubernetes_key +from elyra.util.kubernetes import is_valid_kubernetes_resource_name + + +class ElyraProperty: + """ + A component property that is defined and processed by Elyra. + """ + + property_id: str + generic: bool + custom: bool + _display_name: str + _json_data_type: str + _required: bool = False + _ui_details_map: Dict[str, Dict] = {} + + _subclass_property_map: Dict[str, type] = {} + _json_type_to_default: Dict[str, Any] = {"boolean": False, "number": 0, "array": "[]", "object": "{}", "string": ""} + + @classmethod + def all_subclasses(cls): + """Get all nested subclasses for a class.""" + return set(cls.__subclasses__()).union([s for c in cls.__subclasses__() for s in c.all_subclasses()]) + + @classmethod + def build_property_map(cls) -> None: + """Build the map of property subclasses.""" + cls._subclass_property_map = {sc.property_id: sc for sc in cls.all_subclasses() if hasattr(sc, "property_id")} + + @classmethod + def create_instance(cls, prop_id: str, value: Optional[Any]) -> ElyraProperty | ElyraPropertyList | None: + """Create an instance of a class with the given property id using the user-entered values.""" + if not cls._subclass_property_map: + cls.build_property_map() + + sc = cls._subclass_property_map.get(prop_id) + if not sc: + return None + + if issubclass(sc, ElyraPropertyListItem): + if not isinstance(value, list): + return None + # Create instance for each list element and convert to ElyraPropertyList + instances = ElyraPropertyList([sc.create_instance(prop_id, item) for item in value]) + return instances.deduplicate() + + return sc.create_instance(prop_id, value) + + @classmethod + def get_classes_for_component_type(cls, component_type: str, runtime_type: Optional[str] = "") -> Set[type]: + """ + Retrieve property subclasses that apply to the given component type + (e.g., custom or generic) and to the given runtime type. + """ + from elyra.pipeline.processor import PipelineProcessorManager # placed here to avoid circular reference + + processor_props = set() + for processor in PipelineProcessorManager.instance().get_all_processors(): + props = getattr(processor, "supported_properties", set()) + if processor.type.name == runtime_type and props: + processor_props = props # correct processor is found, and it explicitly specifies its properties + break + processor_props.update(props) + + all_subclasses = set() + for sc in cls.all_subclasses(): + sc_id = getattr(sc, "property_id", "") + if sc_id in processor_props and getattr(sc, component_type, False): + all_subclasses.add(sc) + + return all_subclasses + + @classmethod + def get_schema(cls) -> Dict[str, Any]: + """Build the JSON schema for an Elyra-owned component property""" + class_description = dedent(cls.__doc__).replace("\n", " ") + schema = {"title": cls._display_name, "description": class_description, "type": cls._json_data_type} + if cls._json_data_type not in ["array", "object"]: # property is a scalar value + return schema + + properties, uihints, required_list = {}, {}, [] + for attr, ui_info in cls._ui_details_map.items(): + attr_type = ui_info.get("json_type", "string") + attr_default = cls._json_type_to_default.get(attr_type, "") + attr_title = cls._ui_details_map[attr].get("display_name", attr) + properties[attr] = {"type": attr_type, "title": attr_title, "default": attr_default} + if cls._ui_details_map[attr].get("placeholder"): + uihints[attr] = {"ui:placeholder": cls._ui_details_map[attr].get("placeholder")} + + if ui_info.get("required"): + required_list.append(attr) + if cls._json_data_type == "array": + schema["items"] = {"type": "object", "properties": properties, "required": required_list} + schema["uihints"] = {"items": uihints} + elif cls._json_data_type == "object": + schema.update({"properties": properties, "required": required_list, "uihints": uihints}) + + return schema + + @staticmethod + def strip_if_string(var: Any) -> Any: + """Strip surrounding whitespace from variable if it is a string""" + return var.strip() if isinstance(var, str) else var + + @staticmethod + def unpack(value_dict: dict, *variables): + """Get the values corresponding to the given keys in the provided dict.""" + if not isinstance(value_dict, dict): + value_dict = {} + for var_name in variables: + value = value_dict.get(var_name) + yield ElyraProperty.strip_if_string(value) if value is not None else None + + def get_value_for_display(self) -> Dict[str, Any]: + """ + Get a representation of the instance to display in UI error messages. + Should be implemented in any subclass that has validation criteria. + """ + pass + + def add_to_execution_object(self, runtime_processor: RuntimePipelineProcessor, execution_object: Any, **kwargs): + """ + Add a property instance to the execution object for the given runtime processor. + Calls the runtime processor's implementation of add_{property_type}, e.g. + runtime_processor.add_kubernetes_secret(self, execution_object, **kwargs). + """ + pass + + def get_all_validation_errors(self) -> List[str]: + """Perform custom validation on an instance.""" + return [] + + def is_empty_instance(self) -> bool: + """Returns a boolean indicating whether this instance is considered a no-op.""" + return False + + +class DisableNodeCaching(ElyraProperty): + """Disable caching to force node re-execution in the target runtime environment.""" + + property_id = DISABLE_NODE_CACHING + generic = False + custom = True + _display_name = "Disable node caching" + _json_data_type = "string" + + def __init__(self, selection, **kwargs): + self.selection = selection == "True" + + @classmethod + def create_instance(cls, prop_id: str, value: Optional[Any]) -> DisableNodeCaching: + return DisableNodeCaching(selection=value) + + @classmethod + def get_schema(cls) -> Dict[str, Any]: + """Build the JSON schema for an Elyra-owned component property""" + schema = super().get_schema() + schema["enum"] = ["True", "False"] + return schema + + def add_to_execution_object(self, runtime_processor: RuntimePipelineProcessor, execution_object: Any, **kwargs): + """Add DisableNodeCaching info to the execution object for the given runtime processor""" + runtime_processor.add_disable_node_caching(instance=self, execution_object=execution_object, **kwargs) + + +class ElyraPropertyListItem(ElyraProperty): + """ + An Elyra-owned property that is meant to be a member of an ElyraOwnedPropertyList. + """ + + _keys: List[str] + + @classmethod + def get_schema(cls) -> Dict[str, Any]: + """Build the JSON schema for an Elyra-owned component property""" + schema = super().get_schema() + schema["default"] = [] + return schema + + def to_dict(self) -> Dict[str, Any]: + """Convert instance to a dict with relevant class attributes.""" + dict_repr = {attr: getattr(self, attr, None) for attr in self._ui_details_map} + return dict_repr + + def get_value_for_display(self) -> Dict[str, Any]: + """Get a representation of the instance to display in UI error messages.""" + return self.to_dict() + + def get_key_for_dict_entry(self) -> str: + """ + Given the attribute names in the 'key' property, construct a key + based on the attribute values of the instance. + """ + prop_key = "" + for key_attr in self._keys: + key_part = getattr(self, key_attr) + if key_part: + prop_key += f"{key_part}:" if key_attr != self._keys[-1] else key_part + return prop_key + + def get_value_for_dict_entry(self) -> str: + """Returns the value to be used when constructing a dict from a list of classes.""" + return self.to_dict() + + +class EnvironmentVariable(ElyraPropertyListItem): + """ + Environment variables to be set on the execution environment. + """ + + property_id = ENV_VARIABLES + generic = True + custom = False + _display_name = "Environment Variables" + _json_data_type = "array" + _keys = ["env_var"] + _ui_details_map = { + "env_var": { + "display_name": "Environment Variable", + "placeholder": "ENV_VAR", + "json_type": "string", + "required": True, + }, + "value": {"display_name": "Value", "placeholder": "value", "json_type": "string", "required": False}, + } + + def __init__(self, env_var, value, **kwargs): + self.env_var = env_var + self.value = value + + @classmethod + def create_instance(cls, prop_id: str, value: Optional[Any]) -> EnvironmentVariable | None: + env_var, env_value = cls.unpack(value, "env_var", "value") + if env_value is None or env_value == "": + return None # skip inclusion and continue + + return EnvironmentVariable(env_var=env_var, value=env_value) + + @classmethod + def get_schema(cls) -> Dict[str, Any]: + """Build the JSON schema for an Elyra-owned component property""" + schema = super().get_schema() + schema["uihints"] = {"canRefresh": True} + return schema + + def get_value_for_dict_entry(self) -> str: + """Returns the value to be used when constructing a dict from a list of classes.""" + return self.value + + def get_all_validation_errors(self) -> List[str]: + """Perform custom validation on an instance.""" + validation_errors = [] + if not self.env_var: + validation_errors.append("Required environment variable was not specified.") + elif " " in self.env_var: + validation_errors.append(f"Environment variable '{self.env_var}' includes invalid space character(s).") + + return validation_errors + + def add_to_execution_object(self, runtime_processor: RuntimePipelineProcessor, execution_object: Any, **kwargs): + """Add EnvironmentVariable instance to the execution object for the given runtime processor""" + runtime_processor.add_env_var(instance=self, execution_object=execution_object, **kwargs) + + +class KubernetesSecret(ElyraPropertyListItem): + """ + Kubernetes secrets to make available as environment variables to this node. + The secret name and key given must be present in the Kubernetes namespace + where the node is executed or this node will not run. + """ + + property_id = KUBERNETES_SECRETS + generic = True + custom = False + _display_name = "Kubernetes Secrets" + _json_data_type = "array" + _keys = ["env_var"] + _ui_details_map = { + "env_var": { + "display_name": "Environment Variable", + "placeholder": "ENV_VAR", + "json_type": "string", + "required": True, + }, + "name": {"display_name": "Secret Name", "placeholder": "secret-name", "json_type": "string", "required": True}, + "key": {"display_name": "Secret Key", "placeholder": "secret-key", "json_type": "string", "required": True}, + } + + def __init__(self, env_var, name, key, **kwargs): + self.env_var = env_var + self.name = name + self.key = key + + @classmethod + def create_instance(cls, prop_id: str, value: Optional[Any]) -> KubernetesSecret | None: + env_var, name, key = cls.unpack(value, "env_var", "name", "key") + return KubernetesSecret(env_var=env_var, name=name, key=key) + + def get_all_validation_errors(self) -> List[str]: + """Perform custom validation on an instance.""" + validation_errors = [] + if not self.env_var: + validation_errors.append("Required environment variable was not specified.") + if not self.name: + validation_errors.append("Required secret name was not specified.") + elif not is_valid_kubernetes_resource_name(self.name): + validation_errors.append( + f"Secret name '{self.name}' is not a valid Kubernetes resource name.", + ) + if not self.key: + validation_errors.append("Required secret key was not specified.") + elif not is_valid_kubernetes_key(self.key): + validation_errors.append( + f"Key '{self.key}' is not a valid Kubernetes secret key.", + ) + + return validation_errors + + def add_to_execution_object(self, runtime_processor: RuntimePipelineProcessor, execution_object: Any, **kwargs): + """Add KubernetesSecret instance to the execution object for the given runtime processor""" + runtime_processor.add_kubernetes_secret(instance=self, execution_object=execution_object, **kwargs) + + +class VolumeMount(ElyraPropertyListItem): + """ + Volumes to be mounted in this node. The specified Persistent Volume Claims must exist in the + Kubernetes namespace where the node is executed or this node will not run. + """ + + property_id = MOUNTED_VOLUMES + generic = True + custom = True + _display_name = "Data Volumes" + _json_data_type = "array" + _keys = ["path"] + _ui_details_map = { + "path": {"display_name": "Mount Path", "placeholder": "/mount/path", "json_type": "string", "required": True}, + "pvc_name": { + "display_name": "Persistent Volume Claim Name", + "placeholder": "pvc-name", + "json_type": "string", + "required": True, + }, + } + + def __init__(self, path, pvc_name, **kwargs): + self.path = path + self.pvc_name = pvc_name + + @classmethod + def create_instance(cls, prop_id: str, value: Optional[Any]) -> VolumeMount | None: + path, pvc_name = cls.unpack(value, "path", "pvc_name") + return VolumeMount(path=path, pvc_name=pvc_name) + + def get_all_validation_errors(self) -> List[str]: + """Identify configuration issues for this instance""" + validation_errors = [] + if not self.path: + validation_errors.append("Required mount path was not specified.") + if not self.pvc_name: + validation_errors.append("Required persistent volume claim name was not specified.") + elif not is_valid_kubernetes_resource_name(self.pvc_name): + validation_errors.append(f"PVC name '{self.pvc_name}' is not a valid Kubernetes resource name.") + + return validation_errors + + def add_to_execution_object(self, runtime_processor: RuntimePipelineProcessor, execution_object: Any, **kwargs): + """Add VolumeMount instance to the execution object for the given runtime processor""" + self.path = f"/{self.path.strip('/')}" # normalize path + runtime_processor.add_mounted_volume(instance=self, execution_object=execution_object, **kwargs) + + +class KubernetesAnnotation(ElyraPropertyListItem): + """ + Metadata to be added to this node. The metadata is exposed as annotation + in the Kubernetes pod that executes this node. + """ + + property_id = KUBERNETES_POD_ANNOTATIONS + generic = True + custom = True + _display_name = "Kubernetes Pod Annotations" + _json_data_type = "array" + _keys = ["key"] + _ui_details_map = { + "key": {"display_name": "Key", "placeholder": "annotation_key", "json_type": "string", "required": True}, + "value": {"display_name": "Value", "placeholder": "annotation_value", "json_type": "string", "required": True}, + } + + def __init__(self, key, value, **kwargs): + self.key = key + self.value = value + + @classmethod + def create_instance(cls, prop_id: str, value: Optional[Any]) -> KubernetesAnnotation | None: + key, value = cls.unpack(value, "key", "value") + return KubernetesAnnotation(key=key, value=value) + + def get_all_validation_errors(self) -> List[str]: + """Perform custom validation on an instance.""" + validation_errors = [] + if not self.key: + validation_errors.append("Required annotation key was not specified.") + elif not is_valid_annotation_key(self.key): + validation_errors.append(f"'{self.key}' is not a valid Kubernetes annotation key.") + if not self.value: + validation_errors.append("Required annotation value was not specified.") + + return validation_errors + + def get_value_for_dict_entry(self) -> str: + """Returns the value to be used when constructing a dict from a list of classes.""" + return self.value + + def add_to_execution_object(self, runtime_processor: RuntimePipelineProcessor, execution_object: Any, **kwargs): + """Add KubernetesAnnotation instance to the execution object for the given runtime processor""" + runtime_processor.add_kubernetes_pod_annotation(instance=self, execution_object=execution_object, **kwargs) + + +class KubernetesToleration(ElyraPropertyListItem): + """ + Kubernetes tolerations to apply to the pod where the node is executed. + """ + + property_id = KUBERNETES_TOLERATIONS + generic = True + custom = True + _display_name = "Kubernetes Tolerations" + _json_data_type = "array" + _keys = ["key", "operator", "value", "effect"] + _ui_details_map = { + "key": {"display_name": "Key", "placeholder": "key", "json_type": "string", "required": False}, + "operator": {"display_name": "Operator", "json_type": "string", "required": True}, + "value": {"display_name": "Value", "placeholder": "value", "json_type": "string", "required": False}, + "effect": {"display_name": "Effect", "placeholder": "NoSchedule", "json_type": "string", "required": False}, + } + + def __init__(self, key, operator, value, effect, **kwargs): + self.key = key + self.operator = operator + self.value = value + self.effect = effect + + @classmethod + def create_instance(cls, prop_id: str, value: Optional[Any]) -> KubernetesToleration | None: + key, operator, value, effect = cls.unpack(value, "key", "operator", "value", "effect") + return KubernetesToleration(key=key, operator=operator, value=value, effect=effect) + + @classmethod + def get_schema(cls) -> Dict[str, Any]: + """Build the JSON schema for an Elyra-owned component property""" + schema = super().get_schema() + op_enum = ["Equal", "Exists"] + schema["items"]["properties"]["operator"]["enum"] = op_enum + schema["items"]["properties"]["operator"]["default"] = op_enum[0] + schema["items"]["properties"]["effect"]["enum"] = ["", "NoExecute", "NoSchedule", "PreferNoSchedule"] + return schema + + def get_all_validation_errors(self) -> List[str]: + """ + Perform custom validation on an instance using the constraints documented in + https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ + """ + validation_errors = [] + + # Ensure the operator is valid + if self.operator not in ["Exists", "Equal"]: + validation_errors.append( + f"'{self.operator}' is not a valid operator: the value must be one of 'Exists' or 'Equal'." + ) + + if self.operator == "Equal" and not self.key: + validation_errors.append( + f"'{self.operator}' is not a valid operator: operator must be 'Exists' if no key is specified." + ) + + if ( + self.effect is not None + and len(self.effect) > 0 + and self.effect not in ["NoExecute", "NoSchedule", "PreferNoSchedule"] + ): + validation_errors.append( + f"'{self.effect}' is not a valid effect: effect must be one " + f"of 'NoExecute', 'NoSchedule', or 'PreferNoSchedule'." + ) + + if self.operator == "Exists" and self.value: + validation_errors.append( + f"'{self.value}' is not a valid value: value should be empty if operator is 'Exists'." + ) + return validation_errors + + def add_to_execution_object(self, runtime_processor: RuntimePipelineProcessor, execution_object: Any, **kwargs): + """Add KubernetesToleration instance to the execution object for the given runtime processor""" + runtime_processor.add_kubernetes_toleration(instance=self, execution_object=execution_object, **kwargs) + + +class ElyraPropertyList(list): + """ + A list class that exposes functionality specific to lists whose entries are + of the class ElyraOwnedPropertyListItem. + """ + + def to_dict(self: List[ElyraPropertyListItem], use_prop_as_value: bool = False) -> Dict[str, str]: + """ + Each Elyra-owned property consists of a set of attributes, some subset of which represents + a unique key. Lists of these properties, however, often need converted to dictionary + form for processing - so we must convert. + """ + prop_dict = {} + for prop in self: + if prop is None or not isinstance(prop, ElyraPropertyListItem): + continue # invalid entry; skip inclusion and continue + prop_key = prop.get_key_for_dict_entry() + if prop_key is None: + continue # invalid entry; skip inclusion and continue + + prop_value = prop.get_value_for_dict_entry() + if use_prop_as_value: + prop_value = prop # use of the property object itself as the value + prop_dict[prop_key] = prop_value + + return prop_dict + + def deduplicate(self: ElyraPropertyList) -> ElyraPropertyList: + """Remove duplicates from the given list""" + instance_dict = self.to_dict(use_prop_as_value=True) + return ElyraPropertyList({**instance_dict}.values()) + + @staticmethod + def merge(primary: ElyraPropertyList, secondary: ElyraPropertyList) -> ElyraPropertyList: + """ + Merge two lists of Elyra-owned properties, preferring the values given in the + primary parameter in the case of a matching key between the two lists. + """ + primary_dict = primary.to_dict(use_prop_as_value=True) + secondary_dict = secondary.to_dict(use_prop_as_value=True) + + merged_list = list({**secondary_dict, **primary_dict}.values()) + return ElyraPropertyList(merged_list) + + @staticmethod + def difference(minuend: ElyraPropertyList, subtrahend: ElyraPropertyList) -> ElyraPropertyList: + """ + Given two lists of Elyra-owned properties, remove any duplicate instances + found in the second (subtrahend) from the first (minuend), if present. + + :param minuend: list to be subtracted from + :param subtrahend: list from which duplicates will be determined and given preference + + :returns: the difference of the two lists + """ + subtract_dict = minuend.to_dict(use_prop_as_value=True) + for key in subtrahend.to_dict().keys(): + if key in subtract_dict: + subtract_dict.pop(key) + + return ElyraPropertyList(subtract_dict.values()) + + def add_to_execution_object(self, runtime_processor: RuntimePipelineProcessor, execution_object: Any): + """ + Add a property instance to the execution object for the given runtime processor + for each list item. + """ + for item in self: + if isinstance(item, ElyraPropertyListItem): + item.add_to_execution_object(runtime_processor=runtime_processor, execution_object=execution_object) + + +class ElyraPropertyJSONEncoder(json.JSONEncoder): + """ + A JSON Encoder class to prevent errors during serialization of Elyra-owned property classes. + """ + + def default(self, o): + """ + Render dataclass content as dict + """ + return o.__dict__ if isinstance(o, ElyraProperty) else super().default(o) + + +class ComponentParameter(object): + """ + Represents a single property for a pipeline component + """ + + def __init__( + self, + id: str, + name: str, + json_data_type: str, + description: str, + value: Optional[Any] = "", + allowed_input_types: Optional[List[Optional[str]]] = None, + required: Optional[bool] = False, + allow_no_options: Optional[bool] = False, + items: Optional[List[str]] = None, + ): + """ + :param id: Unique identifier for a property + :param name: The name of the property for display + :param json_data_type: The JSON data type that represents this parameters value + :param allowed_input_types: The input types that the property can accept, including those for custom rendering + :param value: The default value of the property + :param description: A description of the property for display + :param required: Whether the property is required + :param allow_no_options: Specifies whether to allow parent nodes that don't specifically + define output properties to be selected as input to this node parameter + :param items: For properties with a control of 'EnumControl', the items making up the enum + """ + + if not id: + raise ValueError("Invalid component: Missing field 'id'.") + if not name: + raise ValueError("Invalid component: Missing field 'name'.") + + self._ref = id + self._name = name + self._json_data_type = json_data_type + + # The JSON type that the value entered for this property will be rendered in. + # E.g., array types are entered by users and processed by the backend as + # strings whereas boolean types are entered and processed as booleans + self._value_entry_type = json_data_type + if json_data_type in {"array", "object"}: + self._value_entry_type = "string" + + if json_data_type == "boolean" and isinstance(value, str): + value = value in ["True", "true"] + elif json_data_type == "number" and isinstance(value, str): + try: + # Attempt to coerce string to integer value + value = int(value) + except ValueError: + # Value could not be coerced to integer, assume float + value = float(value) + if json_data_type in {"array", "object"} and not isinstance(value, str): + value = str(value) + self._value = value + + self._description = description + + if not allowed_input_types: + allowed_input_types = ["inputvalue", "inputpath", "file"] + self._allowed_input_types = allowed_input_types + + self._items = items or [] + + # Check description for information about 'required' parameter + if "not optional" in description.lower() or ( + "required" in description.lower() + and "not required" not in description.lower() + and "n't required" not in description.lower() + ): + required = True + + self._required = required + self._allow_no_options = allow_no_options + + @property + def ref(self) -> str: + return self._ref + + @property + def name(self) -> str: + return self._name + + @property + def allowed_input_types(self) -> List[Optional[str]]: + return self._allowed_input_types + + @property + def json_data_type(self) -> str: + return self._json_data_type + + @property + def value_entry_type(self) -> str: + return self._value_entry_type + + @property + def value(self) -> Any: + return self._value + + @property + def description(self) -> str: + return self._description + + @property + def items(self) -> List[str]: + return self._items + + @property + def required(self) -> bool: + return bool(self._required) + + @property + def allow_no_options(self) -> bool: + return self._allow_no_options + + @staticmethod + def render_parameter_details(param: ComponentParameter) -> str: + """ + Render the parameter data type and UI hints needed for the specified param for + use in the custom component properties DAG template + :returns: a string literal containing the JSON object to be rendered in the DAG + """ + json_dict = {"title": param.name, "description": param.description} + if len(param.allowed_input_types) == 1: + # Parameter only accepts a single type of input + input_type = param.allowed_input_types[0] + if not input_type: + # This is an output + json_dict["type"] = "string" + json_dict["uihints"] = {"ui:widget": "hidden", "outputpath": True} + elif input_type == "inputpath": + json_dict.update( + { + "type": "object", + "properties": { + "widget": {"type": "string", "default": input_type}, + "value": {"type": "string", "enum": []}, + }, + "uihints": {"widget": {"ui:field": "hidden"}, "value": {input_type: True}}, + } + ) + elif input_type == "file": + json_dict["type"] = "string" + json_dict["uihints"] = {"ui:widget": input_type} + else: + json_dict["type"] = param.value_entry_type + + # Render default value if it is not None + if param.value is not None: + json_dict["default"] = param.value + else: + # Parameter accepts multiple types of inputs; render a oneOf block + one_of = [] + for widget_type in param.allowed_input_types: + obj = { + "type": "object", + "properties": {"widget": {"type": "string"}, "value": {}}, + "uihints": {"widget": {"ui:widget": "hidden"}, "value": {}}, + } + if widget_type == "inputvalue": + obj["title"] = InputTypeDescriptionMap[param.value_entry_type].value + obj["properties"]["widget"]["default"] = param.value_entry_type + obj["properties"]["value"]["type"] = param.value_entry_type + if param.value_entry_type == "boolean": + obj["properties"]["value"]["title"] = " " + + # Render default value if it is not None + if param.value is not None: + obj["properties"]["value"]["default"] = param.value + else: # inputpath or file types + obj["title"] = InputTypeDescriptionMap[widget_type].value + obj["properties"]["widget"]["default"] = widget_type + if widget_type == "outputpath": + obj["uihints"]["value"] = {"ui:readonly": "true", widget_type: True} + obj["properties"]["value"]["type"] = "string" + elif widget_type == "inputpath": + obj["uihints"]["value"] = {widget_type: True} + obj["properties"]["value"]["type"] = "string" + obj["properties"]["value"]["enum"] = [] + if param.allow_no_options: + obj["uihints"]["allownooptions"] = param.allow_no_options + else: + obj["uihints"]["value"] = {"ui:widget": widget_type} + obj["properties"]["value"]["type"] = "string" + + one_of.append(obj) + json_dict["oneOf"] = one_of + + return json.dumps(json_dict) + + +class InputTypeDescriptionMap(Enum): + """A mapping of input types to the description that will appear in the UI""" + + string = "Please enter a string value:" + number = "Please enter a number value:" + boolean = "Please select or deselect the checkbox:" + file = "Please select a file to use as input:" + inputpath = "Please select an output from a parent:" + outputpath = None # outputs are read-only and don't require a description diff --git a/elyra/pipeline/handlers.py b/elyra/pipeline/handlers.py index 1ea8989bb..fe61fbfbd 100644 --- a/elyra/pipeline/handlers.py +++ b/elyra/pipeline/handlers.py @@ -208,7 +208,9 @@ async def get(self, runtime_type): # Get pipeline properties json pipeline_properties_json = PipelineDefinition.get_canvas_properties_from_template( - package_name="templates/pipeline", template_name="pipeline_properties_template.jinja2" + package_name="templates/pipeline", + template_name="pipeline_properties_template.jinja2", + runtime_type=runtime_processor_type.name, ) self.set_status(200) diff --git a/elyra/pipeline/kfp/component_parser_kfp.py b/elyra/pipeline/kfp/component_parser_kfp.py index 9a3b01dd7..11797cdcb 100644 --- a/elyra/pipeline/kfp/component_parser_kfp.py +++ b/elyra/pipeline/kfp/component_parser_kfp.py @@ -25,8 +25,8 @@ from elyra.pipeline.catalog_connector import CatalogEntry from elyra.pipeline.component import Component -from elyra.pipeline.component import ComponentParameter from elyra.pipeline.component import ComponentParser +from elyra.pipeline.component_parameter import ComponentParameter from elyra.pipeline.kfp.kfp_component_utils import component_yaml_schema from elyra.pipeline.runtime_type import RuntimeProcessorType diff --git a/elyra/pipeline/kfp/processor_kfp.py b/elyra/pipeline/kfp/processor_kfp.py index 3e8ac7245..583d826c0 100644 --- a/elyra/pipeline/kfp/processor_kfp.py +++ b/elyra/pipeline/kfp/processor_kfp.py @@ -18,7 +18,9 @@ import re import tempfile import time +from typing import Any from typing import Dict +from typing import Set from urllib.parse import urlsplit from kfp import Client as ArgoClient @@ -27,7 +29,10 @@ from kfp.dsl import PipelineConf from kfp.aws import use_aws_secret # noqa H306 from kubernetes import client as k8s_client +from kubernetes.client import V1EnvVar +from kubernetes.client import V1EnvVarSource from kubernetes.client import V1PersistentVolumeClaimVolumeSource +from kubernetes.client import V1SecretKeySelector from kubernetes.client import V1Toleration from kubernetes.client import V1Volume from kubernetes.client import V1VolumeMount @@ -44,13 +49,20 @@ from elyra.kfp.operator import ExecuteFileOp from elyra.metadata.schemaspaces import RuntimeImages from elyra.metadata.schemaspaces import Runtimes +from elyra.pipeline import pipeline_constants from elyra.pipeline.component_catalog import ComponentCache +from elyra.pipeline.component_parameter import DisableNodeCaching +from elyra.pipeline.component_parameter import ElyraProperty +from elyra.pipeline.component_parameter import ElyraPropertyList +from elyra.pipeline.component_parameter import KubernetesAnnotation +from elyra.pipeline.component_parameter import KubernetesSecret +from elyra.pipeline.component_parameter import KubernetesToleration +from elyra.pipeline.component_parameter import VolumeMount from elyra.pipeline.kfp.kfp_authentication import AuthenticationError from elyra.pipeline.kfp.kfp_authentication import KFPAuthenticator from elyra.pipeline.pipeline import GenericOperation from elyra.pipeline.pipeline import Operation from elyra.pipeline.pipeline import Pipeline -from elyra.pipeline.pipeline_constants import COS_OBJECT_PREFIX from elyra.pipeline.processor import PipelineProcessor from elyra.pipeline.processor import RuntimePipelineProcessor from elyra.pipeline.processor import RuntimePipelineProcessorResponse @@ -364,7 +376,9 @@ def process(self, pipeline): if pipeline.contains_generic_operations(): object_storage_url = f"{cos_public_endpoint}" - os_path = join_paths(pipeline.pipeline_parameters.get(COS_OBJECT_PREFIX), pipeline_instance_id) + os_path = join_paths( + pipeline.pipeline_parameters.get(pipeline_constants.COS_OBJECT_PREFIX), pipeline_instance_id + ) object_storage_path = f"/{cos_bucket}/{os_path}" else: object_storage_url = None @@ -468,7 +482,9 @@ def _cc_pipeline( pipeline_instance_id = pipeline_instance_id or pipeline_name - artifact_object_prefix = join_paths(pipeline.pipeline_parameters.get(COS_OBJECT_PREFIX), pipeline_instance_id) + artifact_object_prefix = join_paths( + pipeline.pipeline_parameters.get(pipeline_constants.COS_OBJECT_PREFIX), pipeline_instance_id + ) self.log_pipeline_info( pipeline_name, @@ -510,6 +526,7 @@ def _cc_pipeline( # Create pipeline operation # If operation is one of the "generic" set of NBs or scripts, construct custom ExecuteFileOp if isinstance(operation, GenericOperation): + component = ComponentCache.get_generic_component_from_op(operation.classifier) # Collect env variables pipeline_envs = self._collect_envs( @@ -522,7 +539,7 @@ def _cc_pipeline( f"Creating pipeline component archive '{operation_artifact_archive}' for operation '{operation}'" ) - target_ops[operation.id] = ExecuteFileOp( + container_op = ExecuteFileOp( name=sanitized_operation_name, pipeline_name=pipeline_name, experiment_name=experiment_name, @@ -546,25 +563,17 @@ def _cc_pipeline( "mlpipeline-metrics": f"{pipeline_envs['ELYRA_WRITABLE_CONTAINER_DIR']}/mlpipeline-metrics.json", # noqa "mlpipeline-ui-metadata": f"{pipeline_envs['ELYRA_WRITABLE_CONTAINER_DIR']}/mlpipeline-ui-metadata.json", # noqa }, - volume_mounts=operation.mounted_volumes, - kubernetes_secrets=operation.kubernetes_secrets, - kubernetes_tolerations=operation.kubernetes_tolerations, - kubernetes_pod_annotations=operation.kubernetes_pod_annotations, ) - if operation.doc: - target_ops[operation.id].add_pod_annotation("elyra/node-user-doc", operation.doc) - - # TODO Can we move all of this to apply to non-standard components as well? Test when servers are up if cos_secret and not export: - target_ops[operation.id].apply(use_aws_secret(cos_secret)) + container_op.apply(use_aws_secret(cos_secret)) image_namespace = self._get_metadata_configuration(RuntimeImages.RUNTIME_IMAGES_SCHEMASPACE_ID) for image_instance in image_namespace: if image_instance.metadata["image_name"] == operation.runtime_image and image_instance.metadata.get( "pull_policy" ): - target_ops[operation.id].container.set_image_pull_policy(image_instance.metadata["pull_policy"]) + container_op.container.set_image_pull_policy(image_instance.metadata["pull_policy"]) self.log_pipeline_info( pipeline_name, @@ -656,64 +665,23 @@ def _cc_pipeline( container_op = factory_function(**sanitized_component_params) container_op.set_display_name(operation.name) - # Attach node comment - if operation.doc: - container_op.add_pod_annotation("elyra/node-user-doc", operation.doc) - - # Add user-specified volume mounts: the referenced PVCs must exist - # or this operation will fail - if operation.mounted_volumes: - unique_pvcs = [] - for volume_mount in operation.mounted_volumes: - if volume_mount.pvc_name not in unique_pvcs: - container_op.add_volume( - V1Volume( - name=volume_mount.pvc_name, - persistent_volume_claim=V1PersistentVolumeClaimVolumeSource( - claim_name=volume_mount.pvc_name - ), - ) - ) - unique_pvcs.append(volume_mount.pvc_name) - container_op.add_volume_mount( - V1VolumeMount(mount_path=volume_mount.path, name=volume_mount.pvc_name) - ) - - # Add user-specified Kubernetes tolerations - if operation.kubernetes_tolerations: - unique_tolerations = [] - for toleration in operation.kubernetes_tolerations: - if str(toleration) not in unique_tolerations: - container_op.add_toleration( - V1Toleration( - effect=toleration.effect, - key=toleration.key, - operator=toleration.operator, - value=toleration.value, - ) - ) - unique_tolerations.append(str(toleration)) - - # Add user-specified pod annotations - if operation.kubernetes_pod_annotations: - unique_annotations = [] - for annotation in operation.kubernetes_pod_annotations: - if annotation.key not in unique_annotations: - container_op.add_pod_annotation(annotation.key, annotation.value) - unique_annotations.append(annotation.key) - - # Force re-execution of the operation by setting staleness to zero days - # https://www.kubeflow.org/docs/components/pipelines/overview/caching/#managing-caching-staleness - if operation.disallow_cached_output: - container_op.set_caching_options(enable_caching=False) - container_op.execution_options.caching_strategy.max_cache_staleness = "P0D" - - target_ops[operation.id] = container_op except Exception as e: # TODO Fix error messaging and break exceptions down into categories self.log.error(f"Error constructing component {operation.name}: {str(e)}") raise RuntimeError(f"Error constructing component {operation.name}.") + # Attach node comment + if operation.doc: + container_op.add_pod_annotation("elyra/node-user-doc", operation.doc) + + # Process Elyra-owned properties as required for each type + for value in operation.elyra_params.values(): + if isinstance(value, (ElyraProperty, ElyraPropertyList)): + value.add_to_execution_object(runtime_processor=self, execution_object=container_op) + + # Add ContainerOp to target_ops dict + target_ops[operation.id] = container_op + # Process dependencies after all the operations have been created for operation in pipeline.operations.values(): op = target_ops[operation.id] @@ -799,6 +767,60 @@ def _sanitize_param_name(name: str) -> str: return normalized_name.replace(" ", "_") + def add_disable_node_caching(self, instance: DisableNodeCaching, execution_object: Any, **kwargs) -> None: + """Add DisableNodeCaching info to the execution object for the given runtime processor""" + # Force re-execution of the operation by setting staleness to zero days + # https://www.kubeflow.org/docs/components/pipelines/overview/caching/#managing-caching-staleness + if instance.selection: + execution_object.execution_options.caching_strategy.max_cache_staleness = "P0D" + + def add_kubernetes_secret(self, instance: KubernetesSecret, execution_object: Any, **kwargs) -> None: + """Add KubernetesSecret instance to the execution object for the given runtime processor""" + execution_object.container.add_env_variable( + V1EnvVar( + name=instance.env_var, + value_from=V1EnvVarSource(secret_key_ref=V1SecretKeySelector(name=instance.name, key=instance.key)), + ) + ) + + def add_mounted_volume(self, instance: VolumeMount, execution_object: Any, **kwargs) -> None: + """Add VolumeMount instance to the execution object for the given runtime processor""" + volume = V1Volume( + name=instance.pvc_name, + persistent_volume_claim=V1PersistentVolumeClaimVolumeSource(claim_name=instance.pvc_name), + ) + if volume not in execution_object.volumes: + execution_object.add_volume(volume) + execution_object.container.add_volume_mount(V1VolumeMount(mount_path=instance.path, name=instance.pvc_name)) + + def add_kubernetes_pod_annotation(self, instance: KubernetesAnnotation, execution_object: Any, **kwargs) -> None: + """Add KubernetesAnnotation instance to the execution object for the given runtime processor""" + if instance.key not in execution_object.pod_annotations: + execution_object.add_pod_annotation(instance.key, instance.value) + + def add_kubernetes_toleration(self, instance: KubernetesToleration, execution_object: Any, **kwargs) -> None: + """Add KubernetesToleration instance to the execution object for the given runtime processor""" + toleration = V1Toleration( + effect=instance.effect, + key=instance.key, + operator=instance.operator, + value=instance.value, + ) + if toleration not in execution_object.tolerations: + execution_object.add_toleration(toleration) + + @property + def supported_properties(self) -> Set[str]: + """A list of Elyra-owned properties supported by this runtime processor.""" + return [ + pipeline_constants.ENV_VARIABLES, + pipeline_constants.KUBERNETES_SECRETS, + pipeline_constants.MOUNTED_VOLUMES, + pipeline_constants.KUBERNETES_POD_ANNOTATIONS, + pipeline_constants.KUBERNETES_TOLERATIONS, + pipeline_constants.DISABLE_NODE_CACHING, + ] + class KfpPipelineProcessorResponse(RuntimePipelineProcessorResponse): _type = RuntimeProcessorType.KUBEFLOW_PIPELINES diff --git a/elyra/pipeline/parser.py b/elyra/pipeline/parser.py index 786584567..5b502f4ae 100644 --- a/elyra/pipeline/parser.py +++ b/elyra/pipeline/parser.py @@ -65,7 +65,12 @@ def parse(self, pipeline_json: Dict) -> Pipeline: description=description, pipeline_parameters=primary_pipeline.pipeline_parameters, ) - self._nodes_to_operations(pipeline_definition, pipeline_object, primary_pipeline.nodes) + + nodes = primary_pipeline.nodes + for pipeline in pipeline_definition.pipelines: + if pipeline.id == primary_pipeline.id: + nodes = pipeline.nodes + self._nodes_to_operations(pipeline_definition, pipeline_object, nodes) return pipeline_object def _nodes_to_operations( @@ -132,13 +137,20 @@ def _create_pipeline_operation(self, node: Node, super_node: Node = None) -> Ope if super_node: # gather parent-links tied to embedded nodes inputs parent_operations.extend(PipelineParser._get_parent_operation_links(super_node.to_dict(), node.id)) + # Split properties into component- and Elyra-owned + component_params, elyra_params = node.get("component_parameters", {}), {} + for param_id in list(component_params.keys()): + if param_id in node.elyra_owned_properties: + elyra_params[param_id] = node.pop_component_parameter(param_id) + return Operation.create_instance( id=node.id, type=node.type, classifier=node.op, name=node.label, parent_operation_ids=parent_operations, - component_params=node.get("component_parameters", {}), + component_params=component_params, + elyra_params=elyra_params, ) @staticmethod diff --git a/elyra/pipeline/pipeline.py b/elyra/pipeline/pipeline.py index 31e3a06a7..dd7390a19 100644 --- a/elyra/pipeline/pipeline.py +++ b/elyra/pipeline/pipeline.py @@ -13,10 +13,8 @@ # See the License for the specific language governing permissions and # limitations under the License. # -from dataclasses import asdict as dataclass_asdict -from dataclasses import dataclass -from dataclasses import is_dataclass -import json +from __future__ import annotations + import os import sys from typing import Any @@ -24,16 +22,14 @@ from typing import List from typing import Optional -from elyra.pipeline.pipeline_constants import DISALLOW_CACHED_OUTPUT +from elyra.pipeline.component_parameter import ElyraPropertyList +from elyra.pipeline.component_parameter import EnvironmentVariable from elyra.pipeline.pipeline_constants import ENV_VARIABLES -from elyra.pipeline.pipeline_constants import KUBERNETES_POD_ANNOTATIONS -from elyra.pipeline.pipeline_constants import KUBERNETES_SECRETS -from elyra.pipeline.pipeline_constants import KUBERNETES_TOLERATIONS -from elyra.pipeline.pipeline_constants import MOUNTED_VOLUMES +from elyra.pipeline.pipeline_constants import RUNTIME_IMAGE # TODO: Make pipeline version available more widely # as today is only available on the pipeline editor -PIPELINE_CURRENT_VERSION = 7.5 # TODO: Update to 8 with update to pipeline-editor v1.10 +PIPELINE_CURRENT_VERSION = 8 PIPELINE_CURRENT_SCHEMA = 3.0 @@ -53,16 +49,13 @@ def create_instance( classifier: str, parent_operation_ids: Optional[List[str]] = None, component_params: Optional[Dict[str, Any]] = None, - ) -> "Operation": + elyra_params: Optional[Dict[str, Any]] = None, + ) -> Operation: """Class method that creates the appropriate instance of Operation based on inputs.""" if Operation.is_generic_operation(classifier): - return GenericOperation( - id, type, name, classifier, parent_operation_ids=parent_operation_ids, component_params=component_params - ) - return Operation( - id, type, name, classifier, parent_operation_ids=parent_operation_ids, component_params=component_params - ) + return GenericOperation(id, type, name, classifier, parent_operation_ids, component_params, elyra_params) + return Operation(id, type, name, classifier, parent_operation_ids, component_params, elyra_params) def __init__( self, @@ -72,16 +65,17 @@ def __init__( classifier: str, parent_operation_ids: Optional[List[str]] = None, component_params: Optional[Dict[str, Any]] = None, + elyra_params: Optional[Dict[str, Any]] = None, ): """ - :param id: Generated UUID, 128 bit number used as a unique identifier - e.g. 123e4567-e89b-12d3-a456-426614174000 + :param id: Generated UUID, 128 bit number used as a unique identifier, e.g. 123e4567-e89b-12d3-a456-426614174000 :param type: The type of node e.g. execution_node :param classifier: indicates the operation's class :param name: The name of the operation :param parent_operation_ids: List of parent operation 'ids' required to execute prior to this operation :param component_params: dictionary of parameter key:value pairs that are used in the creation of a - a non-standard operation instance + non-Generic operation instance + :param elyra_params: dictionary of parameter key:value pairs that are owned by Elyra """ # Validate that the operation has all required properties @@ -100,46 +94,9 @@ def __init__( self._name = name self._parent_operation_ids = parent_operation_ids or [] self._component_params = component_params or {} + self._elyra_params = elyra_params or {} self._doc = None - self._mounted_volumes = [] - param_volumes = component_params.get(MOUNTED_VOLUMES) - if ( - param_volumes is not None - and isinstance(param_volumes, list) - and (len(param_volumes) == 0 or isinstance(param_volumes[0], VolumeMount)) - ): - # The mounted_volumes property is an Elyra system property (ie, not defined in the component - # spec) and must be removed from the component_params dict - self._mounted_volumes = self._component_params.pop(MOUNTED_VOLUMES, []) - - self._kubernetes_tolerations = [] - param_tolerations = component_params.get(KUBERNETES_TOLERATIONS) - if ( - param_tolerations is not None - and isinstance(param_tolerations, list) - and (len(param_tolerations) == 0 or isinstance(param_tolerations[0], KubernetesToleration)) - ): - # The kubernetes_tolerations property is the Elyra system property (ie, not defined in the component - # spec) and must be removed from the component_params dict - self._kubernetes_tolerations = self._component_params.pop(KUBERNETES_TOLERATIONS, []) - - self._kubernetes_pod_annotations = [] - param_annotations = component_params.get(KUBERNETES_POD_ANNOTATIONS) - if ( - param_annotations is not None - and isinstance(param_annotations, list) - and (len(param_annotations) == 0 or isinstance(param_annotations[0], KubernetesAnnotation)) - ): - # The kubernetes_pod_annotations property is an Elyra system property (ie, not defined in the component - # spec) and must be removed from the component_params dict - self._kubernetes_pod_annotations = self._component_params.pop(KUBERNETES_POD_ANNOTATIONS, []) - - # If disabled, this operation is requested to be re-executed in the - # target runtime environment, even if it was executed before. - param_disallow_cached_output = component_params.get(DISALLOW_CACHED_OUTPUT) - self._disallow_cached_output = param_disallow_cached_output - # Scrub the inputs and outputs lists self._component_params["inputs"] = Operation._scrub_list(component_params.get("inputs", [])) self._component_params["outputs"] = Operation._scrub_list(component_params.get("outputs", [])) @@ -185,25 +142,8 @@ def component_params_as_dict(self) -> Dict[str, Any]: return self._component_params or {} @property - def mounted_volumes(self) -> List["VolumeMount"]: - return self._mounted_volumes - - @property - def kubernetes_tolerations(self) -> List["KubernetesToleration"]: - return self._kubernetes_tolerations - - @property - def kubernetes_pod_annotations(self) -> List["KubernetesAnnotation"]: - return self._kubernetes_pod_annotations - - @property - def disallow_cached_output(self) -> Optional[bool]: - """ - Returns None if caching behavior is delegated to the runtime - Returns True if cached output may be used (instead of executing the op to produce it) - Returns False if cached output must not be used (instead of executing the op to produce it) - """ - return self._disallow_cached_output + def elyra_params(self) -> Optional[Dict[str, Any]]: + return self._elyra_params or {} @property def inputs(self) -> Optional[List[str]]: @@ -225,7 +165,7 @@ def is_generic(self) -> bool: def outputs(self, value: List[str]): self._component_params["outputs"] = value - def __eq__(self, other: "Operation") -> bool: + def __eq__(self, other: Operation) -> bool: if isinstance(self, other.__class__): return ( self.id == other.id @@ -278,6 +218,7 @@ def __init__( classifier: str, parent_operation_ids: Optional[List[str]] = None, component_params: Optional[Dict[str, Any]] = None, + elyra_params: Optional[Dict[str, Any]] = None, ): """ :param id: Generated UUID, 128 bit number used as a unique identifier @@ -299,19 +240,18 @@ def __init__( dependencies: List of local files/directories needed for the operation to run and packaged into each operation's dependency archive include_subdirectories: Include or Exclude subdirectories when packaging our 'dependencies' - env_vars: List of Environmental variables to set in the container image - e.g. FOO="BAR" + env_vars: List of Environmental variables to set in the container image, e.g. FOO="BAR" inputs: List of files to be consumed by this operation, produced by parent operation(s) outputs: List of files produced by this operation to be included in a child operation(s) cpu: number of cpus requested to run the operation memory: amount of memory requested to run the operation (in Gi) gpu: number of gpus requested to run the operation Entries for other (non-built-in) component types are a function of the respective component. + + :param elyra_params: dictionary of parameter key:value pairs that are owned by Elyra """ - super().__init__( - id, type, name, classifier, parent_operation_ids=parent_operation_ids, component_params=component_params - ) + super().__init__(id, type, name, classifier, parent_operation_ids, component_params, elyra_params) if not component_params.get("filename"): raise ValueError("Invalid pipeline operation: Missing field 'operation filename'.") @@ -329,11 +269,14 @@ def __init__( self._component_params["runtime_image"] = component_params.get("runtime_image") self._component_params["dependencies"] = Operation._scrub_list(component_params.get("dependencies", [])) self._component_params["include_subdirectories"] = component_params.get("include_subdirectories", False) - self._component_params["env_vars"] = KeyValueList(Operation._scrub_list(component_params.get("env_vars", []))) self._component_params["cpu"] = component_params.get("cpu") self._component_params["gpu"] = component_params.get("gpu") self._component_params["memory"] = component_params.get("memory") + if not elyra_params: + elyra_params = {} + self._elyra_params["env_vars"] = ElyraPropertyList(elyra_params.get(ENV_VARIABLES, [])) + @property def name(self) -> str: if self._name == os.path.basename(self.filename): @@ -350,7 +293,7 @@ def filename(self) -> str: @property def runtime_image(self) -> str: - return self._component_params.get("runtime_image") + return self._component_params.get(RUNTIME_IMAGE) @property def dependencies(self) -> Optional[List[str]]: @@ -361,8 +304,8 @@ def include_subdirectories(self) -> Optional[bool]: return self._component_params.get("include_subdirectories") @property - def env_vars(self) -> Optional["KeyValueList"]: - return self._component_params.get(ENV_VARIABLES) + def env_vars(self) -> ElyraPropertyList[EnvironmentVariable]: + return self._elyra_params.get(ENV_VARIABLES) @property def cpu(self) -> Optional[str]: @@ -376,11 +319,7 @@ def memory(self) -> Optional[str]: def gpu(self) -> Optional[str]: return self._component_params.get("gpu") - @property - def kubernetes_secrets(self) -> List["KubernetesSecret"]: - return self._component_params.get(KUBERNETES_SECRETS) - - def __eq__(self, other: "GenericOperation") -> bool: + def __eq__(self, other: GenericOperation) -> bool: if isinstance(self, other.__class__): return super().__eq__(other) return False @@ -539,86 +478,3 @@ def to_dict(self) -> Dict[str, str]: kv_dict[key] = value return kv_dict - - @classmethod - def to_str(cls, key: str, value: str) -> str: - return f"{key}{cls._key_value_separator}{value}" - - @classmethod - def from_dict(cls, kv_dict: Dict) -> "KeyValueList": - """ - Convert a set of key-value pairs stored in a dictionary to - a KeyValueList of strings with the defined separator. - """ - str_list = [KeyValueList.to_str(key, value) for key, value in kv_dict.items()] - return KeyValueList(str_list) - - @classmethod - def merge(cls, primary: "KeyValueList", secondary: "KeyValueList") -> "KeyValueList": - """ - Merge two key-value pair lists, preferring the values given in the - primary parameter in the case of a matching key between the two lists. - """ - primary_dict = primary.to_dict() - secondary_dict = secondary.to_dict() - - return KeyValueList.from_dict({**secondary_dict, **primary_dict}) - - @classmethod - def difference(cls, minuend: "KeyValueList", subtrahend: "KeyValueList") -> "KeyValueList": - """ - Given KeyValueLists, convert to dictionaries and remove any keys found in the - second (subtrahend) from the first (minuend), if present. - - :param minuend: list to be subtracted from - :param subtrahend: list whose keys will be removed from the minuend - - :returns: the difference of the two lists - """ - subtract_dict = minuend.to_dict() - for key in subtrahend.to_dict().keys(): - if key in subtract_dict: - subtract_dict.pop(key) - - return KeyValueList.from_dict(subtract_dict) - - -@dataclass -class VolumeMount: - path: str - pvc_name: str - - -@dataclass -class KubernetesSecret: - env_var: str - name: str - key: str - - -@dataclass -class KubernetesToleration: - key: str - operator: str - value: str - effect: str - - -@dataclass -class KubernetesAnnotation: - key: str - value: str - - -class DataClassJSONEncoder(json.JSONEncoder): - """ - A JSON Encoder class to prevent errors during serialization of dataclasses. - """ - - def default(self, o): - """ - Render dataclass content as dict - """ - if is_dataclass(o): - return dataclass_asdict(o) - return super().default(o) diff --git a/elyra/pipeline/pipeline_constants.py b/elyra/pipeline/pipeline_constants.py index 742adae59..c92ee9ff7 100644 --- a/elyra/pipeline/pipeline_constants.py +++ b/elyra/pipeline/pipeline_constants.py @@ -21,13 +21,7 @@ KUBERNETES_SECRETS = "kubernetes_secrets" KUBERNETES_TOLERATIONS = "kubernetes_tolerations" KUBERNETES_POD_ANNOTATIONS = "kubernetes_pod_annotations" -DISALLOW_CACHED_OUTPUT = "disallow_cached_output" +DISABLE_NODE_CACHING = "disable_node_caching" PIPELINE_META_PROPERTIES = ["name", "description", "runtime"] # optional static prefix to be used when generating an object name for object storage COS_OBJECT_PREFIX = "cos_object_prefix" -ELYRA_COMPONENT_PROPERTIES = [ - MOUNTED_VOLUMES, - KUBERNETES_TOLERATIONS, - KUBERNETES_POD_ANNOTATIONS, - DISALLOW_CACHED_OUTPUT, -] diff --git a/elyra/pipeline/pipeline_definition.py b/elyra/pipeline/pipeline_definition.py index ffe25e63a..dd4f5f906 100644 --- a/elyra/pipeline/pipeline_definition.py +++ b/elyra/pipeline/pipeline_definition.py @@ -27,18 +27,12 @@ from jinja2 import Undefined from elyra.pipeline.component_catalog import ComponentCache -from elyra.pipeline.pipeline import KeyValueList -from elyra.pipeline.pipeline import KubernetesAnnotation -from elyra.pipeline.pipeline import KubernetesSecret -from elyra.pipeline.pipeline import KubernetesToleration +from elyra.pipeline.component_parameter import ComponentParameter +from elyra.pipeline.component_parameter import ElyraProperty +from elyra.pipeline.component_parameter import ElyraPropertyList from elyra.pipeline.pipeline import Operation -from elyra.pipeline.pipeline import VolumeMount -from elyra.pipeline.pipeline_constants import ELYRA_COMPONENT_PROPERTIES -from elyra.pipeline.pipeline_constants import ENV_VARIABLES -from elyra.pipeline.pipeline_constants import KUBERNETES_POD_ANNOTATIONS +from elyra.pipeline.pipeline_constants import ENV_VARIABLES, RUNTIME_IMAGE from elyra.pipeline.pipeline_constants import KUBERNETES_SECRETS -from elyra.pipeline.pipeline_constants import KUBERNETES_TOLERATIONS -from elyra.pipeline.pipeline_constants import MOUNTED_VOLUMES from elyra.pipeline.pipeline_constants import PIPELINE_DEFAULTS from elyra.pipeline.pipeline_constants import PIPELINE_META_PROPERTIES from elyra.pipeline.runtime_type import RuntimeProcessorType @@ -117,7 +111,7 @@ def version(self) -> Union[int, float]: try: version = int(version) - except ValueError: # version is not an int + except ValueError: # version is not an int; this will only ever be the case in dev versions version = float(version) return version @@ -234,27 +228,29 @@ def set_property(self, key: str, value: Any): self._node["app_data"]["properties"][key] = value - def convert_kv_properties(self, kv_properties: Set[str]): + def convert_elyra_owned_properties(self) -> None: """ - Convert pipeline defaults-level list properties that have been identified - as sets of key-value pairs from a plain list type to the KeyValueList type. + Convert select pipeline-level properties to their corresponding dataclass + object type. No validation is performed. """ pipeline_defaults = self.get_property(PIPELINE_DEFAULTS, {}) - for property_name, value in pipeline_defaults.items(): - if property_name not in kv_properties: - continue + for param_id, param_value in list(pipeline_defaults.items()): + if isinstance(param_value, (ElyraProperty, ElyraPropertyList)) or param_value is None: + continue # property has already been properly converted or cannot be converted - # Replace plain list with KeyValueList - pipeline_defaults[property_name] = KeyValueList(value) - - if pipeline_defaults: - self.set_property(PIPELINE_DEFAULTS, pipeline_defaults) + converted_value = ElyraProperty.create_instance(param_id, param_value) + if converted_value is None: + continue + if isinstance(converted_value, ElyraProperty) and converted_value.is_empty_instance(): + del pipeline_defaults[param_id] + else: + pipeline_defaults[param_id] = converted_value class Node(AppDataBase): def __init__(self, node: Dict): super().__init__(node) - self._elyra_properties_to_skip = set() + self._elyra_owned_properties = set() @property def type(self) -> str: @@ -317,37 +313,49 @@ def component_source(self) -> Optional[str]: return None @property - def elyra_properties_to_skip(self) -> Set[str]: + def is_generic(self) -> True: """ - Elyra-defined node properties whose processing should be skipped - on the basis that their id collides with a property defined in - the component definition for this Node. + A property that denotes whether this node is a generic component """ - return self._elyra_properties_to_skip + if Operation.is_generic_operation(self.op): + return True + return False - def set_elyra_properties_to_skip(self, runtime_type_name: Optional[str]) -> None: + @property + def elyra_owned_properties(self) -> Set[str]: """ - Determine which Elyra-defined node-level properties to skip - on the basis that their id collides with a property defined in - the component definition for this Node. Then, set the node - property accordingly. + Elyra-defined node properties. In the case of a collision of ids + between Elyra-defined properties and properties parsed from the + component definition, the Elyra-defined property will be excluded. """ - if Operation.is_generic_operation(self.op): - # Generic operations will never have any collisions as all their properties are Elyra-owned - return + return self._elyra_owned_properties + + @elyra_owned_properties.setter + def elyra_owned_properties(self, value: Set) -> None: + self._elyra_owned_properties = value - if not runtime_type_name: - return + def set_elyra_owned_properties(self, runtime_type_name: Optional[str]) -> None: + """ + Determine which Elyra-defined node-level properties apply on the basis that their + id does not collide with a property defined in the component definition for this + Node. Then, set the Elyra-owned node properties accordingly. + """ + component = ComponentCache.get_generic_component_from_op(self.op) + if runtime_type_name and component is None: + runtime_type = RuntimeProcessorType.get_instance_by_name(runtime_type_name) + component = ComponentCache.instance().get_component(runtime_type, self.op) - runtime_type = RuntimeProcessorType.get_instance_by_name(runtime_type_name) - component = ComponentCache.instance().get_component(runtime_type, self.op) - if not component: - return + if component: + # Properties that have the same ref (id) as Elyra-owned node properties + # should be skipped during property propagation and conversion + self.elyra_owned_properties = {param.property_id for param in component.get_elyra_parameters()} + if self.is_generic: + self.elyra_owned_properties.add(RUNTIME_IMAGE) - # Properties that have the same ref (id) as Elyra-owned node properties - # should be skipped during property propagation and conversion - properties_to_skip = [prop.ref for prop in component.properties if prop.ref in ELYRA_COMPONENT_PROPERTIES] - self._elyra_properties_to_skip = set(properties_to_skip) + def unset_elyra_owned_properties(self) -> None: + """Remove properties that should be propagated but not persisted as an Elyra-owned property.""" + if self.is_generic: + self.elyra_owned_properties.remove(RUNTIME_IMAGE) def get_component_parameter(self, key: str, default_value=None) -> Any: """ @@ -373,38 +381,23 @@ def set_component_parameter(self, key: str, value: Any): if value is None: raise ValueError("Value is required") - if key not in self.elyra_properties_to_skip: - # This parameter has been parsed from a custom component definition and - # its value should not be manually set - self._node["app_data"]["component_parameters"][key] = value + self._node["app_data"]["component_parameters"][key] = value - def get_all_component_parameters(self) -> Dict[str, Any]: + def pop_component_parameter(self, key: str, default: Optional[Any] = None) -> Any: """ - Retrieve all component parameter key-value pairs. + Pop component parameter values for a given key + :param key: The parameter key to be retrieved + :param default: the value to be set if not found """ - return self._node["app_data"]["component_parameters"] + if not key: + raise ValueError("Key is required") + return self._node["app_data"]["component_parameters"].pop(key, default) - def convert_kv_properties(self, kv_properties: Set[str]): + def get_all_component_parameters(self) -> Dict[str, Any]: """ - Convert node-level list properties that have been identified as sets of - key-value pairs from a plain list type to the KeyValueList type. If any - k-v property has already been converted to a KeyValueList, all k-v - properties are assumed to have already been converted. + Retrieve all component parameter key-value pairs. """ - for kv_property in kv_properties: - value = self.get_component_parameter(kv_property) - if not value or not isinstance(value, list): # not list or KeyValueList - continue - - if isinstance(value, KeyValueList) or not isinstance(value[0], str): - # A KeyValueList instance implies all relevant properties have already been converted - # Similarly, if KeyValueList items aren't strings, this implies they have already been - # converted to the appropriate data class objects - return - - # Convert plain list to KeyValueList - if kv_property not in self.elyra_properties_to_skip: - self.set_component_parameter(kv_property, KeyValueList(value)) + return self._node["app_data"]["component_parameters"] def remove_env_vars_with_matching_secrets(self): """ @@ -413,63 +406,27 @@ def remove_env_vars_with_matching_secrets(self): """ env_vars = self.get_component_parameter(ENV_VARIABLES) secrets = self.get_component_parameter(KUBERNETES_SECRETS) - if isinstance(env_vars, KeyValueList) and isinstance(secrets, KeyValueList): - new_list = KeyValueList.difference(minuend=env_vars, subtrahend=secrets) + if isinstance(env_vars, ElyraPropertyList) and isinstance(secrets, ElyraPropertyList): + new_list = ElyraPropertyList.difference(minuend=env_vars, subtrahend=secrets) self.set_component_parameter(ENV_VARIABLES, new_list) - def convert_data_class_properties(self): + def convert_elyra_owned_properties(self) -> None: """ Convert select node-level list properties to their corresponding dataclass object type. No validation is performed. """ - volume_mounts = self.get_component_parameter(MOUNTED_VOLUMES) - if volume_mounts and isinstance(volume_mounts, KeyValueList): - volume_objects = [] - for mount_path, pvc_name in volume_mounts.to_dict().items(): - formatted_mount_path = f"/{mount_path.strip('/')}" - - # Create a VolumeMount class instance and add to list - volume_objects.append(VolumeMount(formatted_mount_path, pvc_name)) - - self.set_component_parameter(MOUNTED_VOLUMES, volume_objects) + for param_id in self.elyra_owned_properties: + param_value = self.get_component_parameter(param_id) + if isinstance(param_value, (ElyraProperty, ElyraPropertyList)) or param_value is None: + continue # property has already been properly converted or cannot be converted - secrets = self.get_component_parameter(KUBERNETES_SECRETS) - if secrets and isinstance(secrets, KeyValueList): - secret_objects = [] - for env_var_name, secret in secrets.to_dict().items(): - secret_name, *optional_key = secret.split(":", 1) - - secret_key = "" - if optional_key: - secret_key = optional_key[0].strip() - - # Create a KubernetesSecret class instance and add to list - secret_objects.append(KubernetesSecret(env_var_name, secret_name.strip(), secret_key)) - - self.set_component_parameter(KUBERNETES_SECRETS, secret_objects) - - kubernetes_tolerations = self.get_component_parameter(KUBERNETES_TOLERATIONS) - if kubernetes_tolerations and isinstance(kubernetes_tolerations, KeyValueList): - tolerations_objects = [] - for toleration, toleration_definition in kubernetes_tolerations.to_dict().items(): - # A definition comprises of ":::" - parts = toleration_definition.split(":") - key, operator, value, effect = (parts + [""] * 4)[:4] - # Create a KubernetesToleration class instance and add to list - # Note that the instance might be invalid. - tolerations_objects.append(KubernetesToleration(key, operator, value, effect)) - - self.set_component_parameter(KUBERNETES_TOLERATIONS, tolerations_objects) - - kubernetes_pod_annotations = self.get_component_parameter(KUBERNETES_POD_ANNOTATIONS) - if kubernetes_pod_annotations and isinstance(kubernetes_pod_annotations, KeyValueList): - annotations_objects = [] - for annotation_key, annotation_value in kubernetes_pod_annotations.to_dict().items(): - # Validation should have verified that the provided values are valid - # Create a KubernetesAnnotation class instance and add to list - annotations_objects.append(KubernetesAnnotation(annotation_key, annotation_value)) - - self.set_component_parameter(KUBERNETES_POD_ANNOTATIONS, annotations_objects) + converted_value = ElyraProperty.create_instance(param_id, param_value) + if converted_value is None: + continue + if isinstance(converted_value, ElyraProperty) and converted_value.is_empty_instance(): + self._node["app_data"]["component_parameters"].pop(param_id, None) + else: + self.set_component_parameter(param_id, converted_value) class PipelineDefinition(object): @@ -644,41 +601,33 @@ def propagate_pipeline_default_properties(self): For any default pipeline properties set (e.g. runtime image, volume), propagate the values to any nodes that do not set their own value for that property. """ - # Convert any key-value list pipeline default properties to the KeyValueList type - kv_properties = PipelineDefinition.get_kv_properties() - self.primary_pipeline.convert_kv_properties(kv_properties) + self.primary_pipeline.convert_elyra_owned_properties() pipeline_default_properties = self.primary_pipeline.get_property(PIPELINE_DEFAULTS, {}) for node in self.pipeline_nodes: - # Determine which Elyra-owned properties collide with parsed properties (and therefore must be skipped) - node.set_elyra_properties_to_skip(self.primary_pipeline.type) - - # Convert any key-value list node properties to the KeyValueList type if not done already - node.convert_kv_properties(kv_properties) + # Determine which Elyra-owned properties will require dataclass conversion, then convert + node.set_elyra_owned_properties(self.primary_pipeline.type) + node.convert_elyra_owned_properties() - for property_name, pipeline_default_value in pipeline_default_properties.items(): - if not pipeline_default_value: + for property_name, pipeline_value in pipeline_default_properties.items(): + if not pipeline_value: continue - if not Operation.is_generic_operation(node.op) and ( - property_name not in ELYRA_COMPONENT_PROPERTIES or property_name in node.elyra_properties_to_skip - ): - # Do not propagate default properties that do not apply to custom components, e.g. runtime image - continue + if property_name not in node.elyra_owned_properties: + continue # only Elyra-owned properties should be propagated node_value = node.get_component_parameter(property_name) if not node_value: - node.set_component_parameter(property_name, pipeline_default_value) + node.set_component_parameter(property_name, pipeline_value) continue - if isinstance(pipeline_default_value, KeyValueList) and isinstance(node_value, KeyValueList): - merged_list = KeyValueList.merge(node_value, pipeline_default_value) + if all(isinstance(value, ElyraPropertyList) for value in [pipeline_value, node_value]): + merged_list = ElyraPropertyList.merge(node_value, pipeline_value) node.set_component_parameter(property_name, merged_list) if self.primary_pipeline.runtime_config != "local": node.remove_env_vars_with_matching_secrets() - - node.convert_data_class_properties() + node.unset_elyra_owned_properties() def is_valid(self) -> bool: """ @@ -756,7 +705,7 @@ def get_supernodes(self) -> List[Node]: return supernode_list @staticmethod - def get_canvas_properties_from_template(package_name: str, template_name: str) -> Dict[str, Any]: + def get_canvas_properties_from_template(package_name: str, template_name: str, runtime_type: str) -> Dict[str, Any]: """ Retrieves the dict representation of the canvas-formatted properties associated with the given template and package names. Rendering does @@ -764,30 +713,26 @@ def get_canvas_properties_from_template(package_name: str, template_name: str) - SilentUndefined class. """ loader = PackageLoader("elyra", package_name) - template_env = Environment(loader=loader, undefined=SilentUndefined) - template = template_env.get_template(template_name) - output = template.render() - return json.loads(output) + params_custom = ElyraProperty.get_classes_for_component_type("custom", runtime_type) + params_generic = ElyraProperty.get_classes_for_component_type("generic", runtime_type) - @staticmethod - def get_kv_properties() -> Set[str]: - """ - Get pipeline properties in its canvas form and loop through to - find those that should consist of key/value pairs, as given in - the 'keyValueEntries' key. - """ - canvas_pipeline_properties = PipelineDefinition.get_canvas_properties_from_template( - package_name="templates/pipeline", template_name="pipeline_properties_template.jinja2" - ) + # Get intersection of parameter sets + params_both = params_custom & params_generic - kv_properties = set() - properties = canvas_pipeline_properties["properties"]["pipeline_defaults"]["properties"] - for prop_id, prop in properties.items(): - if prop.get("uihints", {}).get("keyValueEntries", False): - kv_properties.add(prop_id) + template_vars = { + "elyra_owned_custom_parameters": params_both ^ params_custom, + "elyra_owned_generic_parameters": params_generic ^ params_both, + "elyra_owned_parameters": params_both, + "render_parameter_details": ComponentParameter.render_parameter_details, + } + template_env = Environment(loader=loader, undefined=SilentUndefined) + template_env.policies["json.dumps_kwargs"] = {"sort_keys": False} # prevent automatic key sort on 'tojson' + template = template_env.get_template(template_name) + template.globals.update(template_vars) - return kv_properties + output = template.render() + return json.loads(output) class SilentUndefined(Undefined): diff --git a/elyra/pipeline/processor.py b/elyra/pipeline/processor.py index e4fef1bc1..866f3477d 100644 --- a/elyra/pipeline/processor.py +++ b/elyra/pipeline/processor.py @@ -13,6 +13,8 @@ # See the License for the specific language governing permissions and # limitations under the License. # +from __future__ import annotations + from abc import abstractmethod import ast import asyncio @@ -20,6 +22,7 @@ import os from pathlib import Path import time +from typing import Any from typing import Dict from typing import List from typing import Optional @@ -37,6 +40,12 @@ from elyra.metadata.manager import MetadataManager from elyra.pipeline.component import Component from elyra.pipeline.component_catalog import ComponentCache +from elyra.pipeline.component_parameter import DisableNodeCaching +from elyra.pipeline.component_parameter import EnvironmentVariable +from elyra.pipeline.component_parameter import KubernetesAnnotation +from elyra.pipeline.component_parameter import KubernetesSecret +from elyra.pipeline.component_parameter import KubernetesToleration +from elyra.pipeline.component_parameter import VolumeMount from elyra.pipeline.pipeline import GenericOperation from elyra.pipeline.pipeline import Operation from elyra.pipeline.pipeline import Pipeline @@ -50,7 +59,7 @@ class PipelineProcessorRegistry(SingletonConfigurable): - _processors: Dict[str, "PipelineProcessor"] = {} + _processors: Dict[str, PipelineProcessor] = {} def __init__(self, **kwargs): root_dir: Optional[str] = kwargs.pop("root_dir", None) @@ -82,6 +91,9 @@ def get_processor(self, processor_name: str): else: raise RuntimeError(f"Could not find pipeline processor '{processor_name}'") + def get_all_processors(self) -> List[PipelineProcessor]: + return list(self._processors.values()) + def is_valid_processor(self, processor_name: str) -> bool: return processor_name in self._processors.keys() @@ -119,6 +131,9 @@ def get_processor_for_runtime(self, runtime_name: str): processor = self._registry.get_processor(runtime_name) return processor + def get_all_processors(self) -> List[PipelineProcessor]: + return self._registry.get_all_processors() + def is_supported_runtime(self, runtime_name: str) -> bool: return self._registry.is_valid_processor(runtime_name) @@ -353,13 +368,13 @@ def _sort_operation_dependencies(operations_by_id: dict, ordered_operations: lis class RuntimePipelineProcessor(PipelineProcessor): - def _get_dependency_archive_name(self, operation: Operation) -> str: + def _get_dependency_archive_name(self, operation: GenericOperation) -> str: return f"{Path(operation.filename).stem}-{operation.id}.tar.gz" - def _get_dependency_source_dir(self, operation: Operation) -> str: + def _get_dependency_source_dir(self, operation: GenericOperation) -> str: return str(Path(self.root_dir) / Path(operation.filename).parent) - def _generate_dependency_archive(self, operation: Operation) -> Optional[str]: + def _generate_dependency_archive(self, operation: GenericOperation) -> Optional[str]: archive_artifact_name = self._get_dependency_archive_name(operation) archive_source_dir = self._get_dependency_source_dir(operation) @@ -377,7 +392,7 @@ def _generate_dependency_archive(self, operation: Operation) -> Optional[str]: return archive_artifact def _upload_dependencies_to_object_store( - self, runtime_configuration: str, pipeline_name: str, operation: Operation, prefix: str = "" + self, runtime_configuration: str, pipeline_name: str, operation: GenericOperation, prefix: str = "" ) -> None: """ Create dependency archive for the generic operation identified by operation @@ -581,3 +596,27 @@ def _process_list_value(self, value: str) -> Union[List, str]: return value return converted_list + + def add_disable_node_caching(self, instance: DisableNodeCaching, execution_object: Any, **kwargs) -> None: + """Add DisableNodeCaching info to the execution object for the given runtime processor""" + pass + + def add_env_var(self, instance: EnvironmentVariable, execution_object: Any, **kwargs) -> None: + """Add EnvironmentVariable instance to the execution object for the given runtime processor""" + pass + + def add_kubernetes_secret(self, instance: KubernetesSecret, execution_object: Any, **kwargs) -> None: + """Add KubernetesSecret instance to the execution object for the given runtime processor""" + pass + + def add_mounted_volume(self, instance: VolumeMount, execution_object: Any, **kwargs) -> None: + """Add VolumeMount instance to the execution object for the given runtime processor""" + pass + + def add_kubernetes_pod_annotation(self, instance: KubernetesAnnotation, execution_object: Any, **kwargs) -> None: + """Add KubernetesAnnotation instance to the execution object for the given runtime processor""" + pass + + def add_kubernetes_toleration(self, instance: KubernetesToleration, execution_object: Any, **kwargs) -> None: + """Add KubernetesToleration instance to the execution object for the given runtime processor""" + pass diff --git a/elyra/pipeline/validation.py b/elyra/pipeline/validation.py index a2304d7d0..b159d5c9e 100644 --- a/elyra/pipeline/validation.py +++ b/elyra/pipeline/validation.py @@ -28,31 +28,19 @@ from elyra.metadata.manager import MetadataManager from elyra.metadata.schema import SchemaManager from elyra.metadata.schemaspaces import Runtimes -from elyra.pipeline.component import Component from elyra.pipeline.component_catalog import ComponentCache -from elyra.pipeline.pipeline import DataClassJSONEncoder -from elyra.pipeline.pipeline import KeyValueList -from elyra.pipeline.pipeline import KubernetesAnnotation -from elyra.pipeline.pipeline import KubernetesSecret -from elyra.pipeline.pipeline import KubernetesToleration +from elyra.pipeline.component_parameter import ElyraProperty +from elyra.pipeline.component_parameter import ElyraPropertyJSONEncoder +from elyra.pipeline.component_parameter import ElyraPropertyList from elyra.pipeline.pipeline import Operation from elyra.pipeline.pipeline import PIPELINE_CURRENT_SCHEMA from elyra.pipeline.pipeline import PIPELINE_CURRENT_VERSION -from elyra.pipeline.pipeline import VolumeMount -from elyra.pipeline.pipeline_constants import ELYRA_COMPONENT_PROPERTIES from elyra.pipeline.pipeline_constants import ENV_VARIABLES -from elyra.pipeline.pipeline_constants import KUBERNETES_POD_ANNOTATIONS -from elyra.pipeline.pipeline_constants import KUBERNETES_SECRETS -from elyra.pipeline.pipeline_constants import KUBERNETES_TOLERATIONS -from elyra.pipeline.pipeline_constants import MOUNTED_VOLUMES from elyra.pipeline.pipeline_constants import RUNTIME_IMAGE from elyra.pipeline.pipeline_definition import Node from elyra.pipeline.pipeline_definition import PipelineDefinition from elyra.pipeline.processor import PipelineProcessorManager from elyra.pipeline.runtime_type import RuntimeProcessorType -from elyra.util.kubernetes import is_valid_annotation_key -from elyra.util.kubernetes import is_valid_kubernetes_key -from elyra.util.kubernetes import is_valid_kubernetes_resource_name from elyra.util.path import get_expanded_path @@ -113,7 +101,8 @@ def add_message( "message": message, "data": data, } - self._response["issues"].append(diagnostic) + if diagnostic not in self._response["issues"]: + self._response["issues"].append(diagnostic) if severity is ValidationSeverity.Error: self._has_fatal = True @@ -398,7 +387,7 @@ async def _validate_node_properties( for node in pipeline.nodes: if node.type == "execution_node": if Operation.is_generic_operation(node.op): - self._validate_generic_node_properties( + await self._validate_generic_node_properties( node=node, response=response, pipeline_runtime=pipeline_runtime ) # Validate runtime components against specific node properties in component registry @@ -410,7 +399,7 @@ async def _validate_node_properties( pipeline_definition=pipeline_definition, ) - def _validate_generic_node_properties(self, node: Node, response: ValidationResponse, pipeline_runtime: str): + async def _validate_generic_node_properties(self, node: Node, response: ValidationResponse, pipeline_runtime: str): """ Validate properties of a generic node :param node: the generic node to check @@ -422,11 +411,7 @@ def _validate_generic_node_properties(self, node: Node, response: ValidationResp image_name = node.get_component_parameter(RUNTIME_IMAGE) filename = node.get_component_parameter("filename") dependencies = node.get_component_parameter("dependencies") - env_vars = node.get_component_parameter(ENV_VARIABLES) - volumes = node.get_component_parameter(MOUNTED_VOLUMES) - secrets = node.get_component_parameter(KUBERNETES_SECRETS) - tolerations = node.get_component_parameter(KUBERNETES_TOLERATIONS) - annotations = node.get_component_parameter(KUBERNETES_POD_ANNOTATIONS) + component_props = await self._get_component_properties(node.op) self._validate_filepath( node_id=node.id, node_label=node_label, property_name="filename", filename=filename, response=response @@ -446,14 +431,12 @@ def _validate_generic_node_properties(self, node: Node, response: ValidationResp response=response, ) - if volumes: - self._validate_mounted_volumes(node.id, node_label, volumes, response=response) - if secrets: - self._validate_kubernetes_secrets(node.id, node_label, secrets, response=response) - if tolerations: - self._validate_kubernetes_tolerations(node.id, node_label, tolerations, response=response) - if annotations: - self._validate_kubernetes_pod_annotations(node.id, node_label, annotations, response=response) + for param in node.elyra_owned_properties: + required = self._is_required_property(component_props, param) + self._validate_elyra_owned_property(node.id, node.label, node, param, response, required) + else: + # Only env vars need to be validated for local runtime + self._validate_elyra_owned_property(node.id, node.label, node, ENV_VARIABLES, response) self._validate_label(node_id=node.id, node_label=node_label, response=response) if dependencies: @@ -467,9 +450,6 @@ def _validate_generic_node_properties(self, node: Node, response: ValidationResp filename=dependency, response=response, ) - if env_vars: - for env_var in env_vars: - self._validate_environmental_variables(node.id, node_label, env_var=env_var, response=response) async def _validate_custom_component_node_properties( self, node: Node, response: ValidationResponse, pipeline_definition: PipelineDefinition, pipeline_runtime: str @@ -482,32 +462,16 @@ async def _validate_custom_component_node_properties( :param pipeline_runtime: the pipeline runtime selected :return: """ - - component_list = await PipelineProcessorManager.instance().get_components(pipeline_runtime) - components = ComponentCache.to_canvas_palette(component_list) - # Full dict of properties for the operation e.g. current params, optionals etc - component_property_dict = await self._get_component_properties(pipeline_runtime, components, node.op) + component_property_dict = await self._get_component_properties(node.op, pipeline_runtime) current_parameters = component_property_dict["properties"]["component_parameters"]["properties"] - volumes = node.get_component_parameter(MOUNTED_VOLUMES) - if volumes and MOUNTED_VOLUMES not in node.elyra_properties_to_skip: - self._validate_mounted_volumes(node.id, node.label, volumes, response=response) - - tolerations = node.get_component_parameter(KUBERNETES_TOLERATIONS) - if tolerations and KUBERNETES_TOLERATIONS not in node.elyra_properties_to_skip: - self._validate_kubernetes_tolerations(node.id, node.label, tolerations, response=response) - - annotations = node.get_component_parameter(KUBERNETES_POD_ANNOTATIONS) - if annotations and KUBERNETES_POD_ANNOTATIONS not in node.elyra_properties_to_skip: - self._validate_kubernetes_pod_annotations(node.id, node.label, annotations, response=response) + for param in node.elyra_owned_properties: + param_required = self._is_required_property(component_property_dict, param) + self._validate_elyra_owned_property(node.id, node.label, node, param, response, param_required) - # List of just the parameters parsed from the component definition - parsed_parameters = [ - p - for p in current_parameters.keys() - if p not in ELYRA_COMPONENT_PROPERTIES or p in node.elyra_properties_to_skip - ] + # List of just the current parameters for the component + parsed_parameters = [p for p in current_parameters.keys() if p not in node.elyra_owned_properties] for default_parameter in parsed_parameters: node_param = node.get_component_parameter(default_parameter) if not node_param or node_param.get("value") is None: @@ -650,181 +614,52 @@ def _validate_resource_value( }, ) - def _validate_mounted_volumes( - self, node_id: str, node_label: str, volumes: List[VolumeMount], response: ValidationResponse + def _validate_elyra_owned_property( + self, + node_id: str, + node_label: str, + node: Node, + param_name: str, + response: ValidationResponse, + required: bool = False, ) -> None: """ Checks the format of mounted volumes to ensure they're in the correct form e.g. foo/path=pvc_name :param node_id: the unique ID of the node :param node_label: the given node name or user customized name/label of the node - :param volumes: a KeyValueList of volumes to check + :param param_name: the name of the parameter to check :param response: ValidationResponse containing the issue list to be updated """ - for volume in volumes: - # Ensure the PVC name is syntactically a valid Kubernetes resource name - if not is_valid_kubernetes_resource_name(volume.pvc_name): - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidVolumeMount", - message=f"PVC name '{volume.pvc_name}' is not a valid Kubernetes resource name.", - data={ - "nodeID": node_id, - "nodeName": node_label, - "propertyName": MOUNTED_VOLUMES, - "value": KeyValueList.to_str(volume.path, volume.pvc_name), - }, - ) - - def _validate_kubernetes_secrets( - self, node_id: str, node_label: str, secrets: List[KubernetesSecret], response: ValidationResponse - ) -> None: - """ - Checks the format of Kubernetes secrets to ensure they're in the correct form - e.g. FOO=SECRET_NAME:KEY - :param node_id: the unique ID of the node - :param node_label: the given node name or user customized name/label of the node - :param secrets: a KeyValueList of secrets to check - :param response: ValidationResponse containing the issue list to be updated - """ - for secret in secrets: - if not secret.name or not secret.key: - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidKubernetesSecret", - message=f"Environment variable '{secret.env_var}' has an improperly formatted representation of " - f"secret name and key.", - data={ - "nodeID": node_id, - "nodeName": node_label, - "propertyName": KUBERNETES_SECRETS, - "value": KeyValueList.to_str(secret.env_var, f"{(secret.name or '')}:{(secret.key or '')}"), - }, - ) - continue - # Ensure the secret name is syntactically a valid Kubernetes resource name - if not is_valid_kubernetes_resource_name(secret.name): - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidKubernetesSecret", - message=f"Secret name '{secret.name}' is not a valid Kubernetes resource name.", - data={ - "nodeID": node_id, - "nodeName": node_label, - "propertyName": KUBERNETES_SECRETS, - "value": KeyValueList.to_str(secret.env_var, f"{secret.name}:{secret.key}"), - }, - ) - # Ensure the secret key is a syntactically valid Kubernetes key - if not is_valid_kubernetes_key(secret.key): - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidKubernetesSecret", - message=f"Key '{secret.key}' is not a valid Kubernetes secret key.", - data={ - "nodeID": node_id, - "nodeName": node_label, - "propertyName": KUBERNETES_SECRETS, - "value": KeyValueList.to_str(secret.env_var, f"{secret.name}:{secret.key}"), - }, - ) - - def _validate_kubernetes_tolerations( - self, node_id: str, node_label: str, tolerations: List[KubernetesToleration], response: ValidationResponse - ) -> None: - """ - Checks the format of kubernetes tolerations to ensure they're in the correct form - e.g. key:operator:value:effect - :param node_id: the unique ID of the node - :param node_label: the given node name or user customized name/label of the node - :param tolerations: a KeyValueList of tolerations to check - :param response: ValidationResponse containing the issue list to be updated - """ - for toleration in tolerations: - # Verify key, operator, value, and effect according to the constraints defined in - # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#toleration-v1-core - if toleration.operator not in ["Exists", "Equal"]: + def validate_elyra_owned_property(elyra_property): + for msg in elyra_property.get_all_validation_errors(): response.add_message( severity=ValidationSeverity.Error, - message_type="invalidKubernetesToleration", - message=f"'{toleration.operator}' is not a valid operator. " - "The value must be one of 'Exists' or 'Equal'.", + message_type=f"invalid{elyra_property.__class__.__name__}", + message=msg, data={ "nodeID": node_id, "nodeName": node_label, - "propertyName": KUBERNETES_TOLERATIONS, - "value": f"{toleration.key}:{toleration.operator}:{toleration.value}:{toleration.effect}", - }, - ) - if len(toleration.key.strip()) == 0 and toleration.operator == "Equal": - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidKubernetesToleration", - message=f"'{toleration.operator}' is not a valid operator. " - "Operator must be 'Exists' if no key is specified.", - data={ - "nodeID": node_id, - "nodeName": node_label, - "propertyName": KUBERNETES_TOLERATIONS, - "value": f"{toleration.key}:{toleration.operator}:{toleration.value}:{toleration.effect}", - }, - ) - if len(toleration.effect.strip()) > 0 and toleration.effect not in [ - "NoExecute", - "NoSchedule", - "PreferNoSchedule", - ]: - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidKubernetesToleration", - message=f"'{toleration.effect}' is not a valid effect. Effect must be one of " - "'NoExecute', 'NoSchedule', or 'PreferNoSchedule'.", - data={ - "nodeID": node_id, - "nodeName": node_label, - "propertyName": KUBERNETES_TOLERATIONS, - "value": f"{toleration.key}:{toleration.operator}:{toleration.value}:{toleration.effect}", - }, - ) - if toleration.operator == "Exists" and len(toleration.value.strip()) > 0: - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidKubernetesToleration", - message=f"'{toleration.value}' is not a valid value. It should be empty if operator is 'Exists'.", - data={ - "nodeID": node_id, - "nodeName": node_label, - "propertyName": KUBERNETES_TOLERATIONS, - "value": f"{toleration.key}:{toleration.operator}:{toleration.value}:{toleration.effect}", + "propertyName": param_name, + "value": elyra_property.get_value_for_display(), }, ) - def _validate_kubernetes_pod_annotations( - self, node_id: str, node_label: str, annotations: List[KubernetesAnnotation], response: ValidationResponse - ) -> None: - """ - Checks the format of the user-provided annotations to ensure they're in the correct form - e.g. annotation_key=annotation_value - :param node_id: the unique ID of the node - :param node_label: the given node name or user customized name/label of the node - :param annotations: a KeyValueList of annotations to check - :param response: ValidationResponse containing the issue list to be updated - """ - for annotation in annotations: - # Ensure the annotation key is valid - if not is_valid_annotation_key(annotation.key): - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidKubernetesAnnotation", - message=f"'{annotation.key}' is not a valid Kubernetes annotation key.", - data={ - "nodeID": node_id, - "nodeName": node_label, - "propertyName": KUBERNETES_POD_ANNOTATIONS, - "value": KeyValueList.to_str(annotation.key, annotation.value), - }, - ) + param_value = node.get_component_parameter(param_name) + if param_value: + if isinstance(param_value, ElyraPropertyList): + for prop in param_value: + validate_elyra_owned_property(prop) + elif isinstance(param_value, ElyraProperty): + validate_elyra_owned_property(param_value) + elif required: + response.add_message( + severity=ValidationSeverity.Error, + message_type="invalidNodeProperty", + message="Required property value is missing.", + data={"nodeID": node.id, "nodeName": node_label, "propertyName": param_name}, + ) def _validate_filepath( self, @@ -892,26 +727,6 @@ def _validate_filepath( }, ) - def _validate_environmental_variables( - self, node_id: str, node_label: str, env_var: str, response: ValidationResponse - ) -> None: - """ - Checks the format of the env var to ensure its in the correct form - e.g. FOO = 'BAR' - :param node_id: the unique ID of the node - :param node_label: the given node name or user customized name/label of the node - :param env_var: the env_var key value pair to check - :param response: ValidationResponse containing the issue list to be updated - """ - result = [x.strip(" '\"") for x in env_var.split("=", 1)] - if len(result) != 2: - response.add_message( - severity=ValidationSeverity.Error, - message_type="invalidEnvPair", - message="Property has an improperly formatted env variable key value pair.", - data={"nodeID": node_id, "nodeName": node_label, "propertyName": ENV_VARIABLES, "value": env_var}, - ) - def _validate_label(self, node_id: str, node_label: str, response: ValidationResponse) -> None: """ KFP specific check for the label name when constructing the node operation using dsl @@ -969,7 +784,7 @@ def _validate_pipeline_graph(self, pipeline: dict, response: ValidationResponse) :param response: ValidationResponse containing the issue list to be updated :param pipeline: A dictionary describing the pipeline """ - pipeline_json = json.loads(json.dumps(pipeline, cls=DataClassJSONEncoder)) + pipeline_json = json.loads(json.dumps(pipeline, cls=ElyraPropertyJSONEncoder)) graph = nx.DiGraph() @@ -1020,7 +835,7 @@ def _get_pipeline_id(self, pipeline: dict, node_id: str) -> Optional[str]: :param node_id: the node ID of the node :return: the pipeline ID of where the node is located """ - pipeline_json = json.loads(json.dumps(pipeline, cls=DataClassJSONEncoder)) + pipeline_json = json.loads(json.dumps(pipeline, cls=ElyraPropertyJSONEncoder)) for single_pipeline in pipeline_json["pipelines"]: node_list = single_pipeline["nodes"] for node in node_list: @@ -1028,26 +843,25 @@ def _get_pipeline_id(self, pipeline: dict, node_id: str) -> Optional[str]: return single_pipeline["id"] return None - async def _get_component_properties(self, pipeline_runtime: str, components: dict, node_op: str) -> Dict: + async def _get_component_properties(self, node_op: str, pipeline_runtime: Optional[str] = None) -> Dict: """ Retrieve the full dict of properties associated with the node_op - :param components: list of components associated with the pipeline runtime being used e.g. kfp, airflow :param node_op: the node operation e.g. execute-notebook-node :return: a list of property names associated with the node op """ + if not pipeline_runtime: + pipeline_runtime = RuntimeProcessorType.LOCAL.name.lower() + + # list of components associated with the pipeline runtime being used + component_list = await PipelineProcessorManager.instance().get_components(pipeline_runtime) + components = ComponentCache.to_canvas_palette(component_list) - if node_op == "execute-notebook-node": - node_op = "notebooks" - elif node_op == "execute-r-node": - node_op = "r-script" - elif node_op == "execute-python-node": - node_op = "python-script" for category in components["categories"]: for node_type in category["node_types"]: if node_op == node_type["op"]: - component: Component = await PipelineProcessorManager.instance().get_component( - pipeline_runtime, node_op - ) + component = await PipelineProcessorManager.instance().get_component(pipeline_runtime, node_op) + if not component: # component is generic; retrieve using static method + component = ComponentCache.get_generic_component_from_op(node_op) component_properties = ComponentCache.to_canvas_properties(component) return component_properties @@ -1061,7 +875,7 @@ def _get_node_names(self, pipeline: dict, node_id_list: list) -> List: :return: a string representing the name of the node """ node_name_list = [] - pipeline_json = json.loads(json.dumps(pipeline, cls=DataClassJSONEncoder)) + pipeline_json = json.loads(json.dumps(pipeline, cls=ElyraPropertyJSONEncoder)) for node_id in node_id_list: found = False for single_pipeline in pipeline_json["pipelines"]: @@ -1087,7 +901,7 @@ def _get_node_labels(self, pipeline: dict, link_ids: List[str]) -> Optional[List if link_ids is None: return None - pipeline_json = json.loads(json.dumps(pipeline, cls=DataClassJSONEncoder)) + pipeline_json = json.loads(json.dumps(pipeline, cls=ElyraPropertyJSONEncoder)) node_labels = [] for link_id in link_ids: for single_pipeline in pipeline_json["pipelines"]: @@ -1131,8 +945,12 @@ def _is_required_property(self, property_dict: dict, node_property: str) -> bool :param node_property: the component property to check :return: """ - required_parameters = property_dict["properties"]["component_parameters"]["required"] - return node_property in required_parameters + required_parameters = property_dict["properties"]["component_parameters"].get("required") + if required_parameters: + return node_property in required_parameters + + param = property_dict["properties"]["component_parameters"]["properties"].get(node_property, {}) + return param.get("required", False) def _get_parent_id_list( self, pipeline_definition: PipelineDefinition, node_id_list: list, parent_list: list diff --git a/elyra/templates/airflow/airflow_template.jinja2 b/elyra/templates/airflow/airflow_template.jinja2 index e3b3d14aa..eb7994e6e 100644 --- a/elyra/templates/airflow/airflow_template.jinja2 +++ b/elyra/templates/airflow/airflow_template.jinja2 @@ -1,7 +1,6 @@ from airflow import DAG from airflow.utils.dates import days_ago - args = { 'project_id' : '{{ pipeline_name }}', } @@ -17,31 +16,26 @@ dag = DAG( is_paused_upon_creation={{ is_paused_upon_creation }}, ) -{{ render_secrets_for_cos(cos_secret) }} - {% for key, operation in operations_list.items() %} {% if not operation.is_generic_operator %} {% for import_statement in operation.imports %} {{import_statement}} {% endfor %} {% else %} -{{ render_secrets_for_generic_op(operation) }} -{{ render_volumes_for_generic_op(operation) }} - +from airflow.kubernetes.secret import Secret +from airflow.contrib.kubernetes.volume import Volume +from airflow.contrib.kubernetes.volume_mount import VolumeMount from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator {% endif %} -{% if operation.operator_source %}# Operator source: {{ operation.operator_source }} -{% endif %} +{% if operation.operator_source %}# Operator source: {{ operation.operator_source }}{% endif %} {% if not operation.is_generic_operator %} op_{{ operation.id|regex_replace }} = {{ operation.class_name }}( task_id='{{ operation.notebook|regex_replace }}', {% for param, value in operation.component_params.items() %} {{ param }}={{ value }}, {% endfor %} - {% if operation.volumes or operation.kubernetes_tolerations or operation.kubernetes_pod_annotations %} - executor_config={{ render_executor_config_for_custom_op(operation) }}, - {% endif %} + executor_config={{ processor.render_elyra_owned_properties(operation.elyra_params) }}, {% else %} op_{{ operation.id|regex_replace }} = KubernetesPodOperator(name='{{ operation.notebook|regex_replace }}', namespace='{{ user_namespace }}', @@ -64,22 +58,17 @@ op_{{ operation.id|regex_replace }} = KubernetesPodOperator(name='{{ operation.n {% if operation.gpu_limit %} 'limit_gpu': '{{ operation.gpu_limit }}', {% endif %} - }, + }, {% endif %} - {% if operation.secrets or cos_secret %} - secrets=[{% if operation.secrets %}{% for secret_var in operation.secret_vars %}{{ secret_var }},{% endfor %}{% endif %}{% if cos_secret %}env_var_secret_id, env_var_secret_key{% endif %}], - {% endif %} - {% if operation.volumes %} - volumes=[{% for volume_var in operation.volume_vars %}{{ volume_var }},{% endfor %}], - volume_mounts=[{% for mount_var in operation.volume_mount_vars %}{{ mount_var }},{% endfor %}], - {% endif %} - {% if operation.kubernetes_tolerations or operation.kubernetes_pod_annotations %} - executor_config={{ render_executor_config_for_generic_op(operation) }}, - {% endif %} - in_cluster={{ in_cluster }}, - config_file="{{ kube_config_path }}", - {% endif %} - dag=dag) + volumes=[{{ processor.render_volumes(operation.elyra_params) }}], + volume_mounts=[{{ processor.render_mounts(operation.elyra_params) }}], + secrets=[{{ processor.render_secrets(operation.elyra_params, cos_secret) }}], + annotations={{ processor.render_annotations(operation.elyra_params) }}, + tolerations=[{{ processor.render_tolerations(operation.elyra_params) }}], + in_cluster={{ in_cluster }}, + config_file="{{ kube_config_path }}", +{% endif %} + dag=dag) {% if operation.image_pull_policy %} op_{{ operation.id|regex_replace }}.image_pull_policy = '{{ operation.image_pull_policy }}' {% endif %} diff --git a/elyra/templates/components/canvas_properties_template.jinja2 b/elyra/templates/components/canvas_properties_template.jinja2 index 9c8cc5ca7..ab1417b07 100644 --- a/elyra/templates/components/canvas_properties_template.jinja2 +++ b/elyra/templates/components/canvas_properties_template.jinja2 @@ -35,7 +35,7 @@ "{{ property.ref }}": {{ render_parameter_details(property) }}, {% endfor %} {% endif %} - {% if additional_properties_apply %} + {% if elyra_owned_parameters %} "additional_properties_header": { "type": "null", "title": "Additional Properties", @@ -43,72 +43,13 @@ "uihints": { "ui:field": "header" } - } - {% endif %} - {% if "mounted_volumes" not in elyra_property_collisions_list %} - , - "mounted_volumes": { - "title": "Data Volumes", - "description": "Volumes to be mounted in this node. The specified Persistent Volume Claims must exist in the Kubernetes namespace where the node is executed or this node will not run.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "/mount/path=pvc-name" - }, - "keyValueEntries": true - } - } - {% endif %} - {% if "kubernetes_pod_annotations" not in elyra_property_collisions_list %} - , - "kubernetes_pod_annotations": { - "title": "Kubernetes Pod Annotations", - "description": "Metadata to be added to this node. The metadata is exposed as annotation in the Kubernetes pod that executes this node.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "annotation_key=annotation_value" - }, - "keyValueEntries": true - } - } - {% endif %} - {% if "kubernetes_tolerations" not in elyra_property_collisions_list %} - , - "kubernetes_tolerations": { - "title": "Kubernetes Tolerations", - "description": "Kubernetes tolerations to attach to the pod where the node is executed.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "TOL_ID=key:operator:value:effect" - }, - "keyValueEntries": true - } - } - {% endif %} - {% if "disallow_cached_output" not in elyra_property_collisions_list %} - , - "disallow_cached_output" : { - "type": "boolean", - "title": "Disallow cached output", - "description": "Disable caching to force node re-execution in the target runtime environment." - } + }, + {% for property in elyra_owned_parameters|sort(attribute="property_id") %} + "{{property.property_id}}": {{ property.get_schema()|tojson }} + {% if loop.index != loop|length %} + , + {% endif %} + {% endfor %} {% endif %} }, "required": [ diff --git a/elyra/templates/components/generic_properties_template.jinja2 b/elyra/templates/components/generic_properties_template.jinja2 index 3db9c152c..b2129842f 100644 --- a/elyra/templates/components/generic_properties_template.jinja2 +++ b/elyra/templates/components/generic_properties_template.jinja2 @@ -9,6 +9,14 @@ "component_parameters": { "type": "object", "properties": { + "inputs_header": { + "type": "null", + "title": "Inputs", + "description": "Input properties for this component.", + "uihints": { + "ui:field": "header" + } + }, "filename": { "type": "string", "title": "Filename", @@ -67,50 +75,12 @@ "description": "Recursively include subdirectories when submitting a pipeline (This may increase submission time).", "default": false }, - "env_vars": { - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "title": "Environment Variables", - "description": "Environment variables to be set on the execution environment.", - "uihints": { - "ui:placeholder": "env_var=VALUE", - "canRefresh": true, - "keyValueEntries": true - } - }, - "kubernetes_secrets": { - "title": "Kubernetes Secrets", - "description": "Kubernetes secrets to make available as environment variables to this node. The secret name and key given must be present in the Kubernetes namespace where the node is executed or this node will not run.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "env_var=secret-name:secret-key" - }, - "keyValueEntries": true - } - }, - "kubernetes_pod_annotations": { - "title": "Kubernetes Pod Annotations", - "description": "Metadata to be added to this node. The metadata is exposed as annotation in the Kubernetes pod that executes this node.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], + "outputs_header": { + "type": "null", + "title": "Outputs", + "description": "Outputs produced by this component.", "uihints": { - "items": { - "ui:placeholder": "annotation_key=annotation_value" - }, - "keyValueEntries": true + "ui:field": "header" } }, "outputs": { @@ -128,41 +98,22 @@ } } }, - "mounted_volumes": { - "title": "Data Volumes", - "description": "Volumes to be mounted in this node. The specified Persistent Volume Claims must exist in the Kubernetes namespace where the node is executed or this node will not run.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], + {% if elyra_owned_parameters %} + "additional_properties_header": { + "type": "null", + "title": "Additional Properties", + "description": "Additional properties used by Elyra that are not given in the component definition.", "uihints": { - "items": { - "ui:placeholder": "/mount/path=pvc-name" - }, - "keyValueEntries": true + "ui:field": "header" } }, - "kubernetes_tolerations": { - "title": "Kubernetes Tolerations", - "description": "Kubernetes tolerations to apply to the pod where the node is executed.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "TOL_ID=key:operator:value:effect" - }, - "keyValueEntries": true - } - } + {% for property in elyra_owned_parameters|sort(attribute="property_id") %} + "{{property.property_id}}": {{ property.get_schema()|tojson }}{% if loop.index != loop|length %},{% endif %} + {% endfor %} + {% endif %} }, "required": ["filename", "runtime_image"] } - }, + }, "required": ["component_parameters"] } diff --git a/elyra/templates/pipeline/pipeline_properties_template.jinja2 b/elyra/templates/pipeline/pipeline_properties_template.jinja2 index f645167a0..efd26b0bf 100644 --- a/elyra/templates/pipeline/pipeline_properties_template.jinja2 +++ b/elyra/templates/pipeline/pipeline_properties_template.jinja2 @@ -35,6 +35,7 @@ "ui:placeholder": "project/subproject" } }, + {% if elyra_owned_parameters %} "node_defaults_header": { "type": "null", "title": "Node Defaults", @@ -44,57 +45,10 @@ "ui:field": "header" } }, - "kubernetes_tolerations": { - "title": "Kubernetes Tolerations", - "description": - "Kubernetes tolerations to attach to the pod where the node is executed.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "TOL_ID=key:operator:value:effect" - }, - "keyValueEntries": true - } - }, - "mounted_volumes": { - "title": "Data Volumes", - "description": - "Volumes to be mounted in all nodes. The specified Persistent Volume Claims must exist in the Kubernetes namespace where the nodes are executed or the pipeline will not run.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "/mount/path=pvc-name" - }, - "keyValueEntries": true - } - }, - "kubernetes_pod_annotations": { - "title": "Kubernetes Pod Annotations", - "description": - "Metadata to be added to this node. The metadata is exposed as annotation in the Kubernetes pod that executes this node.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "annotation_key=annotation_value" - }, - "keyValueEntries": true - } - }, + {% endif %} + {% for property in elyra_owned_parameters %} + "{{property.property_id}}": {{ property.get_schema()|tojson }}, + {% endfor %} "generic_node_defaults_header": { "type": "null", "title": "Generic Node Defaults", @@ -105,44 +59,21 @@ } }, "runtime_image": { - "title": "Runtime image", + "title": "Runtime Image", "description": "Container image used as execution environment.", "type": "string" - }, - "env_vars": { - "title": "Environment Variables", - "description": - "Environment variables to be set on the execution environment.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "env_var=VALUE" - }, - "keyValueEntries": true - } - }, - "kubernetes_secrets": { - "title": "Kubernetes Secrets", - "description": - "Kubernetes secrets to make available as environment variables to this node. The secret name and key given must be present in the Kubernetes namespace where the nodes are executed or the pipeline will not run.", - "type": "array", - "items": { - "type": "string", - "default": "" - }, - "default": [], - "uihints": { - "items": { - "ui:placeholder": "env_var=secret-name:secret-key" - }, - "keyValueEntries": true - } - }, + } + {% if elyra_owned_generic_parameters %} + , + {% endif %} + {% for property in elyra_owned_generic_parameters %} + "{{property.property_id}}": {{ property.get_schema()|tojson }} + {% if loop.index != loop|length %} + , + {% endif %} + {% endfor %} + {% if elyra_owned_custom_parameters %} + , "custom_node_defaults_header": { "type": "null", "title": "Custom Node Defaults", @@ -152,11 +83,13 @@ "ui:field": "header" } }, - "disallow_cached_output": { - "type": "boolean", - "title": "Disallow cached output", - "description": "Disable caching to force node re-execution in the target runtime environment." - } + {% endif %} + {% for property in elyra_owned_custom_parameters %} + "{{property.property_id}}": {{ property.get_schema()|tojson }} + {% if loop.index != loop|length %} + , + {% endif %} + {% endfor %} } } } diff --git a/elyra/tests/cli/resources/pipelines/kfp_3_node_custom.pipeline b/elyra/tests/cli/resources/pipelines/kfp_3_node_custom.pipeline index c50fcd33b..345f60a0a 100644 --- a/elyra/tests/cli/resources/pipelines/kfp_3_node_custom.pipeline +++ b/elyra/tests/cli/resources/pipelines/kfp_3_node_custom.pipeline @@ -203,7 +203,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "runtime_type": "KUBEFLOW_PIPELINES", "properties": { "pipeline_defaults": {}, diff --git a/elyra/tests/cli/resources/pipelines/pipeline_with_notebooks.pipeline b/elyra/tests/cli/resources/pipelines/pipeline_with_notebooks.pipeline index aff8fe7d2..6e6d39056 100644 --- a/elyra/tests/cli/resources/pipelines/pipeline_with_notebooks.pipeline +++ b/elyra/tests/cli/resources/pipelines/pipeline_with_notebooks.pipeline @@ -18,8 +18,16 @@ "outputs": [], "env_vars": [], "kubernetes_secrets": [ - "USER_NAME=secret-1:myuserid", - "USER_PASSWORD=secret-1:mypassword" + { + "env_var": "USER_NAME", + "name": "secret-1", + "key": "myuserid" + }, + { + "env_var": "USER_PASSWORD", + "name": "secret-1", + "key": "mypassword" + } ], "dependencies": [], "include_subdirectories": false, @@ -132,7 +140,10 @@ "pipeline_defaults": { "runtime_image": "tensorflow/tensorflow:2.8.0", "mounted_volumes": [ - "/mnt/vol1=pvc-claim-1" + { + "path": "/mnt/vol1", + "pvc_name": "pvc-claim-1" + } ] }, "name": "pipeline_with_notebooks", diff --git a/elyra/tests/cli/resources/pipelines/pipeline_with_notebooks_and_scripts.pipeline b/elyra/tests/cli/resources/pipelines/pipeline_with_notebooks_and_scripts.pipeline index 68dd5b0ac..bd0358059 100644 --- a/elyra/tests/cli/resources/pipelines/pipeline_with_notebooks_and_scripts.pipeline +++ b/elyra/tests/cli/resources/pipelines/pipeline_with_notebooks_and_scripts.pipeline @@ -18,13 +18,24 @@ "outputs": [], "env_vars": [], "kubernetes_secrets": [ - "USER_NAME=secret-1:myuserid", - "USER_PASSWORD=secret-1:mypassword" + { + "env_var": "USER_NAME", + "name": "secret-1", + "key": "myuserid" + }, + { + "env_var": "USER_PASSWORD", + "name": "secret-1", + "key": "mypassword" + } ], "dependencies": [], "include_subdirectories": false, "mounted_volumes": [ - "/mnt/vol2=pvc-claim-2" + { + "path": "/mnt/vol2", + "pvc_name": "pvc-claim-2" + } ] }, "label": "", @@ -78,7 +89,10 @@ "dependencies": [], "include_subdirectories": false, "mounted_volumes": [ - "/mnt/vol2=pvc-claim-2" + { + "path": "/mnt/vol2", + "pvc_name": "pvc-claim-2" + } ] }, "label": "", @@ -136,7 +150,11 @@ "outputs": [], "env_vars": [], "kubernetes_secrets": [ - "API_KEY=secret-2:myapikey" + { + "env_var": "API_KEY", + "name": "secret-2", + "key": "myapikey" + } ], "dependencies": [], "include_subdirectories": false, @@ -198,7 +216,11 @@ "outputs": [], "env_vars": [], "kubernetes_secrets": [ - "API_KEY=secret-2:myapikey" + { + "env_var": "API_KEY", + "name": "secret-2", + "key": "myapikey" + } ], "dependencies": [], "include_subdirectories": false, @@ -253,12 +275,19 @@ "outputs": [], "env_vars": [], "kubernetes_secrets": [ - "TOKEN=secret-3:mytoken" + { + "env_var": "TOKEN", + "name": "secret-3", + "key": "mytoken" + } ], "dependencies": [], "include_subdirectories": false, "mounted_volumes": [ - "/mnt/vol3=pvc-claim-3" + { + "path": "/mnt/vol3", + "pvc_name": "pvc-claim-3" + } ], "runtime_image": "amancevice/pandas:1.4.1" }, @@ -323,7 +352,10 @@ "pipeline_defaults": { "runtime_image": "tensorflow/tensorflow:2.8.0-gpu", "mounted_volumes": [ - "/mnt/vol1=pvc-claim-1" + { + "path": "/mnt/vol1", + "pvc_name": "pvc-claim-1" + } ] }, "name": "pipeline_with_notebooks_and_scripts", diff --git a/elyra/tests/cli/resources/pipelines/pipeline_with_script.pipeline b/elyra/tests/cli/resources/pipelines/pipeline_with_script.pipeline index bd61a8420..a291c1f65 100644 --- a/elyra/tests/cli/resources/pipelines/pipeline_with_script.pipeline +++ b/elyra/tests/cli/resources/pipelines/pipeline_with_script.pipeline @@ -19,11 +19,17 @@ "filename": "scripts/dummy_script_1.py", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", "env_vars": [ - "OP_NAME=a" + { + "env_var": "OP_NAME", + "value": "a" + } ], "include_subdirectories": false, "mounted_volumes": [ - "/mount/test=rwx-test-claim" + { + "path": "/mount/test", + "pvc_name": "rwx-test-claim" + } ], "dependencies": [ "d.txt", @@ -82,10 +88,16 @@ "component_parameters": { "filename": "scripts/dummy_script_2.py", "mounted_volumes": [ - "/mount/test=rwx-test-claim-1" + { + "path": "/mount/test", + "pvc_name": "rwx-test-claim-1" + } ], "env_vars": [ - "OP_NAME=c" + { + "env_var": "OP_NAME", + "value": "c" + } ], "include_subdirectories": false }, diff --git a/elyra/tests/cli/resources/pipelines/pipeline_with_scripts.pipeline b/elyra/tests/cli/resources/pipelines/pipeline_with_scripts.pipeline index 5f398f5d5..8904dba97 100644 --- a/elyra/tests/cli/resources/pipelines/pipeline_with_scripts.pipeline +++ b/elyra/tests/cli/resources/pipelines/pipeline_with_scripts.pipeline @@ -21,7 +21,10 @@ "dependencies": [], "include_subdirectories": false, "mounted_volumes": [ - "/mnt/vol2=pvc-claim-2" + { + "path": "/mnt/vol2", + "pvc_name": "pvc-claim-2" + } ], "runtime_image": "tensorflow/tensorflow:2.8.0" }, @@ -129,7 +132,10 @@ "dependencies": [], "include_subdirectories": false, "mounted_volumes": [ - "mnt/vol3=pvc-claim-3" + { + "path": "mnt/vol3", + "pvc_name": "pvc-claim-3" + } ], "runtime_image": "tensorflow/tensorflow:2.8.0" }, @@ -192,7 +198,11 @@ "properties": { "pipeline_defaults": { "kubernetes_secrets": [ - "API_KEY=secret-2:myapikey" + { + "env_var": "API_KEY", + "name": "secret-2", + "key": "myapikey" + } ] }, "name": "pipeline_with_scripts", diff --git a/elyra/tests/pipeline/airflow/test_component_parser_airflow.py b/elyra/tests/pipeline/airflow/test_component_parser_airflow.py index d41918aed..9db52ac04 100644 --- a/elyra/tests/pipeline/airflow/test_component_parser_airflow.py +++ b/elyra/tests/pipeline/airflow/test_component_parser_airflow.py @@ -439,9 +439,8 @@ def test_parse_airflow_component_file_no_inputs(): properties_json = ComponentCache.to_canvas_properties(no_input_op) # Properties JSON should only include the four parameters common to every - # component: ('mounted_volumes', 'kubernetes_pod_annotations', - # 'kubernetes_tolerations', and 'disallow_cached_output') - num_common_params = 4 + # component: ('mounted_volumes', 'kubernetes_pod_annotations', and 'kubernetes_tolerations') + num_common_params = 3 properties_from_json = [ prop for prop in properties_json["properties"]["component_parameters"]["properties"].keys() diff --git a/elyra/tests/pipeline/airflow/test_processor_airflow.py b/elyra/tests/pipeline/airflow/test_processor_airflow.py index 898311d3c..e49cd0316 100644 --- a/elyra/tests/pipeline/airflow/test_processor_airflow.py +++ b/elyra/tests/pipeline/airflow/test_processor_airflow.py @@ -28,8 +28,10 @@ from elyra.metadata.metadata import Metadata from elyra.pipeline.airflow.processor_airflow import AirflowPipelineProcessor +from elyra.pipeline.component_parameter import ElyraProperty from elyra.pipeline.parser import PipelineParser from elyra.pipeline.pipeline import GenericOperation +from elyra.pipeline.pipeline_constants import MOUNTED_VOLUMES from elyra.pipeline.runtime_type import RuntimeProcessorType from elyra.tests.pipeline.test_pipeline_parser import _read_pipeline_resource from elyra.util.github import GithubClient @@ -203,14 +205,14 @@ def test_create_file(monkeypatch, processor, parsed_pipeline, parsed_ordered_dic sub_list_line_counter = 0 # Gets sub-list slice starting where the Notebook Op starts init_line = i + 1 - for line in file_as_lines[init_line:]: + for idx, line in enumerate(file_as_lines[init_line:], start=init_line): if "namespace=" in line: assert sample_metadata["metadata"]["user_namespace"] == read_key_pair(line)["value"] elif "cos_endpoint=" in line: assert sample_metadata["metadata"]["cos_endpoint"] == read_key_pair(line)["value"] elif "cos_bucket=" in line: assert sample_metadata["metadata"]["cos_bucket"] == read_key_pair(line)["value"] - elif "name=" in line: + elif "name=" in line and "Volume" not in file_as_lines[idx - 1]: assert node["app_data"]["ui_data"]["label"] == read_key_pair(line)["value"] elif "notebook=" in line: assert component_parameters["filename"] == read_key_pair(line)["value"] @@ -218,7 +220,7 @@ def test_create_file(monkeypatch, processor, parsed_pipeline, parsed_ordered_dic assert component_parameters["runtime_image"] == read_key_pair(line)["value"] elif "env_vars=" in line: for env in component_parameters["env_vars"]: - var, value = env.split("=") + var, value = env.get("env_var"), env.get("value") # Gets sub-list slice starting where the env vars starts start_env = i + sub_list_line_counter + 2 for env_line in file_as_lines[start_env:]: @@ -271,6 +273,8 @@ def test_create_file_custom_components( monkeypatch.setattr(processor, "_upload_dependencies_to_object_store", lambda w, x, y, prefix: True) monkeypatch.setattr(processor, "_cc_pipeline", lambda x, y, z: parsed_ordered_dict) + print(parsed_ordered_dict) + with tempfile.TemporaryDirectory() as temp_dir: export_pipeline_output_path = os.path.join(temp_dir, f"{export_pipeline_name}.py") @@ -324,7 +328,13 @@ def test_create_file_custom_components( # Find 'parameter=' clause in file_as_lines list r = re.compile(rf"\s*{parameter}=.*") parameter_clause = i + 1 - assert len(list(filter(r.match, file_as_lines[parameter_clause:]))) > 0 + parameter_in_args = len(list(filter(r.match, file_as_lines[parameter_clause:]))) > 0 + if parameter == MOUNTED_VOLUMES and "DeriveFromTestOperator" in node["op"]: + # Asserts that "DeriveFromTestOperator", which does not define its own `mounted_volumes` + # property, does not include the property in the Operator constructor args + assert not parameter_in_args + else: + assert parameter_in_args # Test that parameter value processing proceeded as expected for each data type op_id = "bb9606ca-29ec-4133-a36a-67bd2a1f6dc3" @@ -427,7 +437,7 @@ def test_pipeline_tree_creation(parsed_ordered_dict, sample_metadata, sample_ima assert ordered_dict[key]["image_pull_policy"] == image.metadata["pull_policy"] print(ordered_dict[key]) for env in component_parameters["env_vars"]: - var, value = env.split("=") + var, value = env.get("env_var"), env.get("value") assert ordered_dict[key]["pipeline_envs"][var] == value assert ( ordered_dict[key]["pipeline_envs"]["AWS_ACCESS_KEY_ID"] @@ -449,26 +459,24 @@ def test_collect_envs(processor): # add system-owned envs with bogus values to ensure they get set to system-derived values, # and include some user-provided edge cases operation_envs = [ - 'ELYRA_RUNTIME_ENV="bogus_runtime"', - 'ELYRA_ENABLE_PIPELINE_INFO="bogus_pipeline"', - "ELYRA_WRITABLE_CONTAINER_DIR=", # simulate operation reference in pipeline - 'AWS_ACCESS_KEY_ID="bogus_key"', - 'AWS_SECRET_ACCESS_KEY="bogus_secret"', - "USER_EMPTY_VALUE= ", - "USER_TWO_EQUALS=KEY=value", - "USER_NO_VALUE=", + {"env_var": "ELYRA_RUNTIME_ENV", "value": '"bogus_runtime"'}, + {"env_var": "ELYRA_ENABLE_PIPELINE_INFO", "value": '"bogus_pipeline"'}, + {"env_var": "ELYRA_WRITABLE_CONTAINER_DIR", "value": ""}, # simulate operation reference in pipeline + {"env_var": "AWS_ACCESS_KEY_ID", "value": '"bogus_key"'}, + {"env_var": "AWS_SECRET_ACCESS_KEY", "value": '"bogus_secret"'}, + {"env_var": "USER_EMPTY_VALUE", "value": " "}, + {"env_var": "USER_TWO_EQUALS", "value": "KEY=value"}, + {"env_var": "USER_NO_VALUE", "value": ""}, ] - component_parameters = { - "filename": pipelines_test_file, - "env_vars": operation_envs, - "runtime_image": "tensorflow/tensorflow:latest", - } + converted_envs = ElyraProperty.create_instance("env_vars", operation_envs) + test_operation = GenericOperation( id="this-is-a-test-id", type="execution-node", classifier="execute-notebook-node", name="test", - component_params=component_parameters, + component_params={"filename": pipelines_test_file, "runtime_image": "tensorflow/tensorflow:latest"}, + elyra_params={"env_vars": converted_envs}, ) envs = processor._collect_envs(test_operation, cos_secret=None, cos_username="Alice", cos_password="secret") diff --git a/elyra/tests/pipeline/kfp/test_component_parser_kfp.py b/elyra/tests/pipeline/kfp/test_component_parser_kfp.py index c5541255c..8784be89c 100644 --- a/elyra/tests/pipeline/kfp/test_component_parser_kfp.py +++ b/elyra/tests/pipeline/kfp/test_component_parser_kfp.py @@ -378,7 +378,7 @@ def test_parse_kfp_component_file_no_inputs(): # Properties JSON should only include the five parameters common to every # component ('mounted_volumes', 'kubernetes_pod_annotations', 'kubernetes_tolerations', - # and 'disallow_cached_output), and the output parameter for this component + # and 'disable_node_caching), and the output parameter for this component num_common_params = 5 properties_from_json = [ prop diff --git a/elyra/tests/pipeline/kfp/test_processor_kfp.py b/elyra/tests/pipeline/kfp/test_processor_kfp.py index 0e6f56db7..ab6112f4a 100644 --- a/elyra/tests/pipeline/kfp/test_processor_kfp.py +++ b/elyra/tests/pipeline/kfp/test_processor_kfp.py @@ -27,6 +27,7 @@ from elyra.pipeline.catalog_connector import UrlComponentCatalogConnector from elyra.pipeline.component import Component from elyra.pipeline.component import ComponentParameter +from elyra.pipeline.component_parameter import ElyraProperty from elyra.pipeline.kfp.processor_kfp import KfpPipelineProcessor from elyra.pipeline.parser import PipelineParser from elyra.pipeline.pipeline import GenericOperation @@ -159,27 +160,24 @@ def test_collect_envs(processor): # add system-owned envs with bogus values to ensure they get set to system-derived values, # and include some user-provided edge cases operation_envs = [ - 'ELYRA_RUNTIME_ENV="bogus_runtime"', - 'ELYRA_ENABLE_PIPELINE_INFO="bogus_pipeline"', - "ELYRA_WRITABLE_CONTAINER_DIR=", # simulate operation reference in pipeline - 'AWS_ACCESS_KEY_ID="bogus_key"', - 'AWS_SECRET_ACCESS_KEY="bogus_secret"', - "USER_EMPTY_VALUE= ", - "USER_TWO_EQUALS=KEY=value", - "USER_NO_VALUE=", + {"env_var": "ELYRA_RUNTIME_ENV", "value": '"bogus_runtime"'}, + {"env_var": "ELYRA_ENABLE_PIPELINE_INFO", "value": '"bogus_pipeline"'}, + {"env_var": "ELYRA_WRITABLE_CONTAINER_DIR", "value": ""}, # simulate operation reference in pipeline + {"env_var": "AWS_ACCESS_KEY_ID", "value": '"bogus_key"'}, + {"env_var": "AWS_SECRET_ACCESS_KEY", "value": '"bogus_secret"'}, + {"env_var": "USER_EMPTY_VALUE", "value": " "}, + {"env_var": "USER_TWO_EQUALS", "value": "KEY=value"}, + {"env_var": "USER_NO_VALUE", "value": ""}, ] + converted_envs = ElyraProperty.create_instance("env_vars", operation_envs) - component_parameters = { - "filename": pipelines_test_file, - "env_vars": operation_envs, - "runtime_image": "tensorflow/tensorflow:latest", - } test_operation = GenericOperation( id="this-is-a-test-id", type="execution-node", classifier="execute-notebook-node", name="test", - component_params=component_parameters, + component_params={"filename": pipelines_test_file, "runtime_image": "tensorflow/tensorflow:latest"}, + elyra_params={"env_vars": converted_envs}, ) envs = processor._collect_envs(test_operation, cos_secret=None, cos_username="Alice", cos_password="secret") diff --git a/elyra/tests/pipeline/resources/additional_generic_properties.json b/elyra/tests/pipeline/resources/additional_generic_properties.json new file mode 100644 index 000000000..27fd5f731 --- /dev/null +++ b/elyra/tests/pipeline/resources/additional_generic_properties.json @@ -0,0 +1,178 @@ +{ + "kubernetes_pod_annotations": { + "title": "Kubernetes Pod Annotations", + "description": " Metadata to be added to this node. The metadata is exposed as annotation in the Kubernetes pod that executes this node. ", + "type": "array", + "items": { + "type": "object", + "properties": { + "key": { + "type": "string", + "title": "Key", + "default": "" + }, + "value": { + "type": "string", + "title": "Value", + "default": "" + } + }, + "required": ["key", "value"] + }, + "uihints": { + "items": { + "key": { + "ui:placeholder": "annotation_key" + }, + "value": { + "ui:placeholder": "annotation_value" + } + } + }, + "default": [] + }, + "mounted_volumes": { + "title": "Data Volumes", + "description": " Volumes to be mounted in this node. The specified Persistent Volume Claims must exist in the Kubernetes namespace where the node is executed or this node will not run. ", + "type": "array", + "items": { + "type": "object", + "properties": { + "path": { + "type": "string", + "title": "Mount Path", + "default": "" + }, + "pvc_name": { + "type": "string", + "title": "Persistent Volume Claim Name", + "default": "" + } + }, + "required": ["path", "pvc_name"] + }, + "uihints": { + "items": { + "path": { + "ui:placeholder": "/mount/path" + }, + "pvc_name": { + "ui:placeholder": "pvc-name" + } + } + }, + "default": [] + }, + "kubernetes_secrets": { + "title": "Kubernetes Secrets", + "description": " Kubernetes secrets to make available as environment variables to this node. The secret name and key given must be present in the Kubernetes namespace where the node is executed or this node will not run. ", + "type": "array", + "items": { + "type": "object", + "properties": { + "env_var": { + "type": "string", + "title": "Environment Variable", + "default": "" + }, + "name": { + "type": "string", + "title": "Secret Name", + "default": "" + }, + "key": { + "type": "string", + "title": "Secret Key", + "default": "" + } + }, + "required": ["env_var", "name", "key"] + }, + "uihints": { + "items": { + "env_var": { + "ui:placeholder": "ENV_VAR" + }, + "name": { + "ui:placeholder": "secret-name" + }, + "key": { + "ui:placeholder": "secret-key" + } + } + }, + "default": [] + }, + "env_vars": { + "title": "Environment Variables", + "description": " Environment variables to be set on the execution environment. ", + "type": "array", + "items": { + "type": "object", + "properties": { + "env_var": { + "type": "string", + "title": "Environment Variable", + "default": "" + }, + "value": { + "type": "string", + "title": "Value", + "default": "" + } + }, + "required": ["env_var"] + }, + "uihints": { + "canRefresh": true + }, + "default": [] + }, + "kubernetes_tolerations": { + "title": "Kubernetes Tolerations", + "description": " Kubernetes tolerations to apply to the pod where the node is executed. ", + "type": "array", + "items": { + "type": "object", + "properties": { + "key": { + "type": "string", + "title": "Key", + "default": "" + }, + "operator": { + "type": "string", + "title": "Operator", + "default": "Equal", + "enum": ["Equal", "Exists"] + }, + "value": { + "type": "string", + "title": "Value", + "default": "" + }, + "effect": { + "type": "string", + "title": "Effect", + "default": "", + "enum": ["", "NoExecute", "NoSchedule", "PreferNoSchedule"] + } + }, + "required": ["operator"] + }, + "uihints": { + "items": { + "key": { + "ui:placeholder": "key" + }, + "value": { + "ui:placeholder": "value" + }, + "effect": { + "ui:placeholder": "NoSchedule" + } + } + }, + "default": [] + } +} diff --git a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_dependency_complex.json b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_dependency_complex.json index 1b9e7dea6..d20bc7380 100644 --- a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_dependency_complex.json +++ b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_dependency_complex.json @@ -17,9 +17,19 @@ "component_parameters": { "filename": "sandbox/a.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=a"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "a" + } + ], "include_subdirectories": false, - "mounted_volumes": ["/mount/test=rwx-test-claim"], + "mounted_volumes": [ + { + "path": "/mount/test", + "pvc_name": "rwx-test-claim" + } + ], "outputs": ["d.txt", "e.txt", "f.txt"] }, "ui_data": { @@ -68,7 +78,12 @@ "component_parameters": { "filename": "sandbox/b.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=b"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "b" + } + ], "include_subdirectories": false, "inputs": ["d.txt", "e.txt", "f.txt"] }, @@ -125,7 +140,12 @@ "component_parameters": { "filename": "sandbox/c.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=c"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "c" + } + ], "include_subdirectories": false }, "ui_data": { @@ -181,7 +201,12 @@ "component_parameters": { "filename": "sandbox/d.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=d"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "d" + } + ], "include_subdirectories": false }, "ui_data": { @@ -237,7 +262,12 @@ "component_parameters": { "filename": "sandbox/e.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=e"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "e" + } + ], "include_subdirectories": false }, "ui_data": { @@ -293,7 +323,12 @@ "component_parameters": { "filename": "sandbox/f.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=f"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "f" + } + ], "include_subdirectories": false }, "ui_data": { @@ -349,7 +384,12 @@ "component_parameters": { "filename": "sandbox/g.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=g"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "g" + } + ], "include_subdirectories": false }, "ui_data": { @@ -410,7 +450,12 @@ "component_parameters": { "filename": "sandbox/x.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=x"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "x" + } + ], "include_subdirectories": false }, "ui_data": { @@ -459,7 +504,12 @@ "component_parameters": { "filename": "sandbox/y.ipynb", "runtime_image": "elyra/examples:1.0.0-py3", - "env_vars": ["OP_NAME=y"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "y" + } + ], "include_subdirectories": false }, "ui_data": { @@ -515,7 +565,12 @@ "component_parameters": { "filename": "sandbox/h.ipynb", "runtime_image": "tensorflow/tensorflow:2.0.0-py3", - "env_vars": ["OP_NAME=h"], + "env_vars": [ + { + "env_var": "OP_NAME", + "value": "h" + } + ], "include_subdirectories": false }, "ui_data": { diff --git a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_valid.json b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_valid.json index aa2e20686..7597a56e4 100644 --- a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_valid.json +++ b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_valid.json @@ -17,7 +17,10 @@ "component_parameters": { "filename": "{{filename}}", "runtime_image": "{{runtime_image}}", - "env_vars": ["var1=var1", "var2=var2"], + "env_vars": [ + { "env_var": "var1", "value": "var1" }, + { "env_var": "var2", "value": "var2" } + ], "include_subdirectories": false, "dependencies": ["a.txt", "b.txt", "c.txt"], "outputs": ["d.txt", "e.txt", "f.txt"] diff --git a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_valid_with_pipeline_default.json b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_valid_with_pipeline_default.json index 1f307be3c..00161f033 100644 --- a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_valid_with_pipeline_default.json +++ b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_valid_with_pipeline_default.json @@ -17,11 +17,13 @@ "component_parameters": { "filename": "{{filename}}", "runtime_image": "{{runtime_image}}", - "env_vars": ["var1=var1", "var2=var2", "var3="], + "env_vars": [ + { "env_var": "var1", "value": "var1" }, + { "env_var": "var2", "value": "var2" } + ], "include_subdirectories": false, "dependencies": ["a.txt", "b.txt", "c.txt"], - "outputs": ["d.txt", "e.txt", "f.txt"], - "kv_test_property": ["var1 =var1", "var3= "] + "outputs": ["d.txt", "e.txt", "f.txt"] }, "ui_data": { "label": "{{label}}", @@ -52,7 +54,9 @@ "app_data": { "label": "{{label}}", "component_parameters": { - "mounted_volumes": ["/mnt/vol1=pvc-claim-1"] + "mounted_volumes": [ + { "path": "/mnt/vol1", "pvc_name": "pvc-claim-1" } + ] }, "component_source": "{{component_source}}", "ui_data": { @@ -89,13 +93,14 @@ "name": "{{name}}", "pipeline_defaults": { "runtime_image": "{{ default_image }}", - "env_vars": ["var1=var_one", "var2=var_two", "var3=var_three"], - "kv_test_property": [ - "var1=var_one ", - "var2=var2", - "var3 =var_three" + "env_vars": [ + { "env_var": "var1", "value": "var_one" }, + { "env_var": "var2", "value": "var_two" }, + { "env_var": "var3", "value": "var_three" } ], - "mounted_volumes": ["/mnt/vol2=pvc-claim-2"] + "mounted_volumes": [ + { "path": "/mnt/vol2", "pvc_name": "pvc-claim-2" } + ] }, "runtime": "{{runtime description}}" }, diff --git a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_with_airflow_components.json b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_with_airflow_components.json index e246daf93..3e7fd427d 100644 --- a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_with_airflow_components.json +++ b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_with_airflow_components.json @@ -128,7 +128,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "runtime": "airflow", "runtime_config": "airflow-yukked1", "runtime_type": "APACHE_AIRFLOW", diff --git a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_with_invalid_list_values.json b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_with_invalid_list_values.json index 816f5cd32..194805f16 100644 --- a/elyra/tests/pipeline/resources/sample_pipelines/pipeline_with_invalid_list_values.json +++ b/elyra/tests/pipeline/resources/sample_pipelines/pipeline_with_invalid_list_values.json @@ -17,7 +17,13 @@ "component_parameters": { "filename": "{{filename}}", "runtime_image": "{{runtime_image}}", - "env_vars": ["var1=var1", "var2=var2", "", null], + "env_vars": [ + { "env_var": "var1", "value": "var1" }, + { "env_var": "var2", "value": "var2" }, + { "env_var": "" }, + {}, + null + ], "dependencies": ["a.txt", "b.txt", "", null, "c.txt"], "outputs": ["d.txt", null, "", "e.txt", "f.txt"] }, diff --git a/elyra/tests/pipeline/resources/validation_pipelines/aa_operator_same_name.pipeline b/elyra/tests/pipeline/resources/validation_pipelines/aa_operator_same_name.pipeline index 2df83e197..b50e87729 100644 --- a/elyra/tests/pipeline/resources/validation_pipelines/aa_operator_same_name.pipeline +++ b/elyra/tests/pipeline/resources/validation_pipelines/aa_operator_same_name.pipeline @@ -211,7 +211,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "runtime": "airflow", "runtime_type": "APACHE_AIRFLOW", "runtime_config": "aa-yukked1", @@ -224,4 +224,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/elyra/tests/pipeline/resources/validation_pipelines/aa_parent_node_missing_xcom.pipeline b/elyra/tests/pipeline/resources/validation_pipelines/aa_parent_node_missing_xcom.pipeline index 962449142..fbd10dd06 100644 --- a/elyra/tests/pipeline/resources/validation_pipelines/aa_parent_node_missing_xcom.pipeline +++ b/elyra/tests/pipeline/resources/validation_pipelines/aa_parent_node_missing_xcom.pipeline @@ -157,7 +157,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "runtime_type": "APACHE_AIRFLOW", "properties": { "name": "aa_parent_node_missing_xcom", @@ -168,4 +168,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/elyra/tests/pipeline/resources/validation_pipelines/generic_basic_pipeline_only_notebook.pipeline b/elyra/tests/pipeline/resources/validation_pipelines/generic_basic_pipeline_only_notebook.pipeline index 49b24bf9d..2fc190a0d 100644 --- a/elyra/tests/pipeline/resources/validation_pipelines/generic_basic_pipeline_only_notebook.pipeline +++ b/elyra/tests/pipeline/resources/validation_pipelines/generic_basic_pipeline_only_notebook.pipeline @@ -359,7 +359,7 @@ "runtime": null, "comments": [] }, - "version": 7.5, + "version": 8, "properties": { "name": "generic_basic_pipeline_only_notebook", "runtime": "Generic" @@ -369,4 +369,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/elyra/tests/pipeline/resources/validation_pipelines/generic_basic_pipeline_with_scripts.pipeline b/elyra/tests/pipeline/resources/validation_pipelines/generic_basic_pipeline_with_scripts.pipeline index 94e15e459..316eaf199 100644 --- a/elyra/tests/pipeline/resources/validation_pipelines/generic_basic_pipeline_with_scripts.pipeline +++ b/elyra/tests/pipeline/resources/validation_pipelines/generic_basic_pipeline_with_scripts.pipeline @@ -248,7 +248,7 @@ "runtime": null, "comments": [] }, - "version": 7.5, + "version": 8, "properties": { "name": "generic_basic_pipeline_with_scripts", "runtime": "Generic" @@ -258,4 +258,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/elyra/tests/pipeline/resources/validation_pipelines/kf_inputpath_parameter.pipeline b/elyra/tests/pipeline/resources/validation_pipelines/kf_inputpath_parameter.pipeline index dd36d0805..c9b3982dc 100644 --- a/elyra/tests/pipeline/resources/validation_pipelines/kf_inputpath_parameter.pipeline +++ b/elyra/tests/pipeline/resources/validation_pipelines/kf_inputpath_parameter.pipeline @@ -205,7 +205,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "runtime_type": "KUBEFLOW_PIPELINES", "properties": { "name": "kf_inputpath_parameter", @@ -216,4 +216,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_inputpath_missing_connection.pipeline b/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_inputpath_missing_connection.pipeline index f1b633f7b..3082695ff 100644 --- a/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_inputpath_missing_connection.pipeline +++ b/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_inputpath_missing_connection.pipeline @@ -190,7 +190,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "runtime_type": "KUBEFLOW_PIPELINES", "properties": { "name": "kf_invalid_inputpath_missing_connection", @@ -201,4 +201,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_inputpath_parameter.pipeline b/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_inputpath_parameter.pipeline index 7fe2f9ac5..2a7f1bde2 100644 --- a/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_inputpath_parameter.pipeline +++ b/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_inputpath_parameter.pipeline @@ -204,7 +204,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "runtime_type": "KUBEFLOW_PIPELINES", "properties": { "name": "kf_invalid_inputpath_parameter", @@ -215,4 +215,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_node_property_in_component.pipeline b/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_node_property_in_component.pipeline index a68ef3693..87d99f8e7 100644 --- a/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_node_property_in_component.pipeline +++ b/elyra/tests/pipeline/resources/validation_pipelines/kf_invalid_node_property_in_component.pipeline @@ -163,7 +163,7 @@ }, "comments": [] }, - "version": 7.5, + "version": 8, "properties": { "name": "kf_invalid_node_property_in_component", "runtime": "Kubeflow Pipelines" @@ -174,4 +174,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/elyra/tests/pipeline/test_handlers.py b/elyra/tests/pipeline/test_handlers.py index a2872ec05..891b9b0e1 100644 --- a/elyra/tests/pipeline/test_handlers.py +++ b/elyra/tests/pipeline/test_handlers.py @@ -28,7 +28,7 @@ from elyra.pipeline.parser import PipelineParser from elyra.pipeline.pipeline_constants import ( COS_OBJECT_PREFIX, - DISALLOW_CACHED_OUTPUT, + DISABLE_NODE_CACHING, ENV_VARIABLES, KUBERNETES_POD_ANNOTATIONS, KUBERNETES_SECRETS, @@ -134,9 +134,22 @@ async def test_get_component_properties_config(jp_fetch): payload = json.loads(response.body.decode()) template = pkg_resources.read_text(resources, "generic_properties_template.jinja2") - properties = json.loads( - template.replace("{{ component.name }}", "Notebook").replace("{{ component.extensions|tojson }}", '[".ipynb"]') - ) + template = template.replace("{{ component.name }}", "Notebook") + template = template.replace("{{ component.extensions|tojson }}", '[".ipynb"]') + template = template.replace("{% if elyra_owned_parameters %}", "") + template = template.replace( + """, + {% for property in elyra_owned_parameters|sort(attribute="property_id") %} + "{{property.property_id}}": {{ property.get_schema()|tojson }}{% if loop.index != loop|length %},{% endif %} + {% endfor %} + {% endif %}""", + "", + ) # remove Elyra-owned property rendering loop + properties = json.loads(template) + + # Fetch Elyra-owned properties + elyra_properties = json.loads(pkg_resources.read_text(resources, "additional_generic_properties.json")) + properties["properties"]["component_parameters"]["properties"].update(elyra_properties) # update property dict assert payload == properties @@ -249,8 +262,10 @@ async def test_get_pipeline_properties_definition(jp_fetch): KUBERNETES_TOLERATIONS, MOUNTED_VOLUMES, KUBERNETES_POD_ANNOTATIONS, - DISALLOW_CACHED_OUTPUT, + DISABLE_NODE_CACHING, ] + if runtime == "airflow": + default_properties.remove(DISABLE_NODE_CACHING) assert all(prop in payload["properties"][PIPELINE_DEFAULTS]["properties"] for prop in default_properties) diff --git a/elyra/tests/pipeline/test_pipeline_constructor.py b/elyra/tests/pipeline/test_pipeline_constructor.py index d95acd987..d93da544a 100644 --- a/elyra/tests/pipeline/test_pipeline_constructor.py +++ b/elyra/tests/pipeline/test_pipeline_constructor.py @@ -17,6 +17,7 @@ import pytest +from elyra.pipeline.component_parameter import ElyraProperty from elyra.pipeline.pipeline import GenericOperation from elyra.pipeline.pipeline import Operation from elyra.pipeline.pipeline import Pipeline @@ -101,7 +102,6 @@ def test_create_operation_with_environmental_variables(): component_parameters = { "filename": "elyra/pipeline/tests/resources/archive/test.ipynb", - "env_vars": env_variables, "runtime_image": "tensorflow/tensorflow:latest", } test_operation = GenericOperation( @@ -110,6 +110,7 @@ def test_create_operation_with_environmental_variables(): classifier="execute-notebook-node", name="test", component_params=component_parameters, + elyra_params={"env_vars": env_variables}, ) assert test_operation.env_vars == env_variables @@ -334,34 +335,22 @@ def test_fail_pipelines_are_equal(good_pipeline): def test_env_list_to_dict_function(): + env_variables_dict = {"KEY": "val", "KEY2": "value2", "TWO_EQUALS": "KEY=value", "": "no_key"} env_variables = [ - "KEY=value", - None, - "", - " =empty_key", - "=no_key", - "EMPTY_VALUE= ", - "NO_VALUE=", - "KEY2=value2", - "TWO_EQUALS=KEY=value", - "==", + {"env_var": "KEY", "value": "val"}, # valid + {"env_var": "", "value": ""}, # empty key and value + {"env_var": " ", "value": "empty_key"}, # empty key + {"env_var": "", "value": "no_key"}, # empty key with value + {"env_var": "EMPTY_VALUE", "value": " "}, # empty value + {"env_var": "NO_VALUE"}, # no value + {"env_var": "KEY2", "value": "value2"}, # valid + {"env_var": "TWO_EQUALS", "value": "KEY=value"}, # valid + {}, # no values + None, # None value ] - env_variables_dict = {"KEY": "value", "KEY2": "value2", "TWO_EQUALS": "KEY=value"} - component_parameters = { - "filename": "elyra/pipeline/tests/resources/archive/test.ipynb", - "env_vars": env_variables, - "runtime_image": "tensorflow/tensorflow:latest", - } - test_operation = GenericOperation( - id="test-id", - type="execution-node", - classifier="execute-notebook-node", - name="test", - component_params=component_parameters, - ) - - assert test_operation.env_vars.to_dict() == env_variables_dict + converted_list = ElyraProperty.create_instance("env_vars", env_variables) + assert converted_list.to_dict() == env_variables_dict def test_validate_resource_values(): diff --git a/elyra/tests/pipeline/test_pipeline_definition.py b/elyra/tests/pipeline/test_pipeline_definition.py index e54d76293..30b071ef6 100644 --- a/elyra/tests/pipeline/test_pipeline_definition.py +++ b/elyra/tests/pipeline/test_pipeline_definition.py @@ -19,8 +19,9 @@ import pytest from elyra.pipeline import pipeline_constants -from elyra.pipeline.pipeline import KeyValueList -from elyra.pipeline.pipeline import VolumeMount +from elyra.pipeline.component_parameter import ElyraProperty +from elyra.pipeline.component_parameter import ElyraPropertyList +from elyra.pipeline.component_parameter import KubernetesSecret from elyra.pipeline.pipeline_constants import ENV_VARIABLES from elyra.pipeline.pipeline_constants import KUBERNETES_SECRETS from elyra.pipeline.pipeline_constants import MOUNTED_VOLUMES @@ -108,61 +109,33 @@ def test_updates_to_nodes_updates_pipeline_definition(): def test_envs_to_dict(): - test_list = ["TEST= one", "TEST_TWO=two ", "TEST_THREE =", " TEST_FOUR=1", "TEST_FIVE = fi=ve "] - test_dict_correct = {"TEST": "one", "TEST_TWO": "two", "TEST_FOUR": "1", "TEST_FIVE": "fi=ve"} - assert KeyValueList(test_list).to_dict() == test_dict_correct - - -def test_env_dict_to_list(): - test_dict = {"TEST": "one", "TEST_TWO": "two", "TEST_FOUR": "1"} - test_list_correct = ["TEST=one", "TEST_TWO=two", "TEST_FOUR=1"] - assert KeyValueList.from_dict(test_dict) == test_list_correct - - -def test_convert_kv_properties(monkeypatch): - kv_test_property_name = "kv_test_property" - pipeline_json = _read_pipeline_resource("resources/sample_pipelines/pipeline_valid_with_pipeline_default.json") - - # Mock get_kv_properties() to ensure the "kv_test_property" variable is included in the list - mock_kv_property_list = [pipeline_constants.ENV_VARIABLES, kv_test_property_name] - monkeypatch.setattr(PipelineDefinition, "get_kv_properties", mock.Mock(return_value=mock_kv_property_list)) - - # Mock set_elyra_properties_to_skip() so that a ComponentCache instance is not created unnecessarily - monkeypatch.setattr(Node, "set_elyra_properties_to_skip", mock.Mock(return_value=None)) - - pipeline_definition = PipelineDefinition(pipeline_definition=pipeline_json) - - node = None - for node in pipeline_definition.pipeline_nodes: - if node.op == "execute-notebook-node": # assign the generic node to the node variable - break - pipeline_defaults = pipeline_definition.primary_pipeline.get_property(pipeline_constants.PIPELINE_DEFAULTS) - - for kv_property in mock_kv_property_list: - assert isinstance(node.get_component_parameter(kv_property), KeyValueList) - assert isinstance(pipeline_defaults[kv_property], KeyValueList) + env_variables = [ + {"env_var": "TEST", "value": " one"}, + {"env_var": "TEST_TWO", "value": "two"}, + {"env_var": "TEST_THREE", "value": ""}, + {"env_var": "TEST_FOUR", "value": "1"}, + {"env_var": "TEST_TWO"}, + ] + converted_list = ElyraProperty.create_instance("env_vars", env_variables) + test_dict_correct = {"TEST": "one", "TEST_TWO": "two", "TEST_FOUR": "1"} + assert converted_list.to_dict() == test_dict_correct - # Ensure a non-list property is not converted to a KeyValueList - assert not isinstance( - pipeline_definition.primary_pipeline.get_property(pipeline_constants.RUNTIME_IMAGE), KeyValueList - ) - # Ensure plain list property is not converted to a KeyValueList - assert not isinstance(node.get_component_parameter("outputs"), KeyValueList) +def test_elyra_property_list_difference(): + env_variables = [ + {"env_var": "TEST", "value": "one"}, + {"env_var": "TEST_TWO", "value": "two"}, + {"env_var": "TEST_FOUR", "value": "1"}, + ] + converted_list = ElyraProperty.create_instance("env_vars", env_variables) + empty_list = ElyraPropertyList.difference(converted_list, converted_list) + assert empty_list == [] -def test_propagate_pipeline_default_properties(monkeypatch): - kv_list_correct = ["var1=var1", "var2=var2", "var3=var_three"] - kv_test_property_name = "kv_test_property" +def test_propagate_pipeline_default_properties(monkeypatch, component_cache): + kv_dict = {"var1": "var1", "var2": "var2", "var3": "var_three"} pipeline_json = _read_pipeline_resource("resources/sample_pipelines/pipeline_valid_with_pipeline_default.json") - # Mock get_kv_properties() to ensure the "kv_test_property" variable is included in the list - mock_kv_property_list = [pipeline_constants.ENV_VARIABLES, kv_test_property_name] - monkeypatch.setattr(PipelineDefinition, "get_kv_properties", mock.Mock(return_value=mock_kv_property_list)) - - # Mock set_elyra_properties_to_skip() so that a ComponentCache instance is not created unnecessarily - monkeypatch.setattr(Node, "set_elyra_properties_to_skip", mock.Mock(return_value=None)) - pipeline_definition = PipelineDefinition(pipeline_definition=pipeline_json) generic_node = None @@ -180,8 +153,8 @@ def test_propagate_pipeline_default_properties(monkeypatch): custom_node_test = node # Ensure that default properties have been propagated - assert generic_node.get_component_parameter(pipeline_constants.ENV_VARIABLES) == kv_list_correct - assert generic_node.get_component_parameter(kv_test_property_name) == kv_list_correct + generic_envs = generic_node.get_component_parameter(pipeline_constants.ENV_VARIABLES) + assert generic_envs.to_dict() == kv_dict # Ensure that runtime image and env vars are not propagated to custom components assert custom_node_test.get_component_parameter(RUNTIME_IMAGE) is None @@ -207,22 +180,22 @@ def test_property_id_collision_with_system_property(monkeypatch, catalog_instanc # DeriveFromTestOperator does not define its own 'mounted_volumes' # property and should not skip the Elyra 'mounted_volumes' property - assert MOUNTED_VOLUMES not in custom_node_derive1.elyra_properties_to_skip - assert MOUNTED_VOLUMES not in custom_node_derive2.elyra_properties_to_skip + assert MOUNTED_VOLUMES in custom_node_derive1.elyra_owned_properties + assert MOUNTED_VOLUMES in custom_node_derive2.elyra_owned_properties # Property value should be a combination of the lists given on the # pipeline node and in the pipeline default properties - assert custom_node_derive1.get_component_parameter(MOUNTED_VOLUMES) == [ - VolumeMount(path="/mnt/vol2", pvc_name="pvc-claim-2"), - VolumeMount(path="/mnt/vol1", pvc_name="pvc-claim-1"), - ] - assert custom_node_derive2.get_component_parameter(MOUNTED_VOLUMES) == [ - VolumeMount(path="/mnt/vol2", pvc_name="pvc-claim-2") - ] - - # TestOperator defines its own 'mounted_volumes' property + derive1_vols = custom_node_derive1.get_component_parameter(MOUNTED_VOLUMES) + assert derive1_vols.to_dict() == { + "/mnt/vol2": {"path": "/mnt/vol2", "pvc_name": "pvc-claim-2"}, + "/mnt/vol1": {"path": "/mnt/vol1", "pvc_name": "pvc-claim-1"}, + } + derive2_vols = custom_node_derive2.get_component_parameter(MOUNTED_VOLUMES) + assert derive2_vols.to_dict() == {"/mnt/vol2": {"path": "/mnt/vol2", "pvc_name": "pvc-claim-2"}} + + # TestOperator defines its own "mounted_volumes" property # and should skip the Elyra system property of the same name - assert MOUNTED_VOLUMES in custom_node_test.elyra_properties_to_skip + assert MOUNTED_VOLUMES not in custom_node_test.elyra_owned_properties # Property value should be as-assigned in pipeline file assert custom_node_test.get_component_parameter(MOUNTED_VOLUMES) == "a component-parsed property" @@ -232,7 +205,9 @@ def test_remove_env_vars_with_matching_secrets(monkeypatch): pipeline_json = _read_pipeline_resource("resources/sample_pipelines/pipeline_valid_with_pipeline_default.json") # Mock set_elyra_properties_to_skip() so that a ComponentCache instance is not created unnecessarily - monkeypatch.setattr(Node, "set_elyra_properties_to_skip", mock.Mock(return_value=None)) + monkeypatch.setattr(Node, "set_elyra_owned_properties", mock.Mock(return_value=None)) + monkeypatch.setattr(Node, "elyra_owned_properties", {KUBERNETES_SECRETS, ENV_VARIABLES}) + monkeypatch.setattr(Node, "unset_elyra_owned_properties", mock.Mock(return_value=None)) pipeline_definition = PipelineDefinition(pipeline_definition=pipeline_json) node = None @@ -241,9 +216,14 @@ def test_remove_env_vars_with_matching_secrets(monkeypatch): break # Set kubernetes_secret property to have all the same keys as those in the env_vars property - kubernetes_secrets = KeyValueList(["var1=name1:key1", "var2=name2:key2", "var3=name3:key3"]) + kubernetes_secrets = ElyraPropertyList( + [ + KubernetesSecret(env_var="var1", name="name1", key="key1"), + KubernetesSecret(env_var="var2", name="name2", key="key2"), + KubernetesSecret(env_var="var3", name="name3", key="key3"), + ] + ) node.set_component_parameter(KUBERNETES_SECRETS, kubernetes_secrets) - node.remove_env_vars_with_matching_secrets() assert node.get_component_parameter(ENV_VARIABLES) == [] diff --git a/elyra/tests/pipeline/test_pipeline_parser.py b/elyra/tests/pipeline/test_pipeline_parser.py index 4a2648bb8..ed7c603eb 100644 --- a/elyra/tests/pipeline/test_pipeline_parser.py +++ b/elyra/tests/pipeline/test_pipeline_parser.py @@ -16,18 +16,21 @@ from conftest import AIRFLOW_TEST_OPERATOR_CATALOG import pytest +from elyra.pipeline.component_parameter import ElyraPropertyList +from elyra.pipeline.component_parameter import EnvironmentVariable from elyra.pipeline.parser import PipelineParser from elyra.pipeline.pipeline import GenericOperation +from elyra.pipeline.pipeline_constants import ENV_VARIABLES from elyra.pipeline.pipeline_constants import MOUNTED_VOLUMES from elyra.tests.pipeline.util import _read_pipeline_resource @pytest.fixture def valid_operation(): + env_vars = [EnvironmentVariable(env_var="var1", value="var1"), EnvironmentVariable(env_var="var2", value="var2")] component_parameters = { "filename": "{{filename}}", "runtime_image": "{{runtime_image}}", - "env_vars": ["var1=var1", "var2=var2"], "dependencies": ["a.txt", "b.txt", "c.txt"], "outputs": ["d.txt", "e.txt", "f.txt"], } @@ -37,6 +40,7 @@ def valid_operation(): classifier="execute-notebook-node", name="{{label}}", component_params=component_parameters, + elyra_params={"env_vars": ElyraPropertyList(env_vars)}, ) @@ -49,6 +53,11 @@ def test_valid_pipeline(valid_operation): assert pipeline.runtime == "{{runtime}}" assert pipeline.runtime_config == "{{runtime-config}}" assert len(pipeline.operations) == 1 + + pipeline_op_envs = pipeline.operations["{{uuid}}"].elyra_params.pop(ENV_VARIABLES) + valid_op_envs = valid_operation.elyra_params.pop(ENV_VARIABLES) + assert pipeline_op_envs.to_dict() == valid_op_envs.to_dict() + assert pipeline.operations["{{uuid}}"] == valid_operation @@ -61,6 +70,11 @@ def test_pipeline_with_dirty_list_values(valid_operation): assert pipeline.runtime == "{{runtime}}" assert pipeline.runtime_config == "{{runtime-config}}" assert len(pipeline.operations) == 1 + + pipeline_op_envs = pipeline.operations["{{uuid}}"].elyra_params.pop(ENV_VARIABLES) + valid_op_envs = valid_operation.elyra_params.pop(ENV_VARIABLES) + assert pipeline_op_envs.to_dict() == valid_op_envs.to_dict() + assert pipeline.operations["{{uuid}}"] == valid_operation @@ -244,10 +258,11 @@ def test_custom_component_parsed_properties(monkeypatch, catalog_instance): custom_op = parsed_pipeline.operations[operation_id] # Ensure this operation's component params does not include the empty mounted volumes list - assert custom_op.component_params_as_dict.get(MOUNTED_VOLUMES) is None + assert custom_op.elyra_params.get(MOUNTED_VOLUMES) == [] operation_id = "bb9606ca-29ec-4133-a36a-67bd2a1f6dc3" custom_op = parsed_pipeline.operations[operation_id] # Ensure this operation's component params includes the value for the component-defined mounted volumes property assert custom_op.component_params_as_dict.get(MOUNTED_VOLUMES)["value"] == "a component-defined property" + assert custom_op.elyra_params.get(MOUNTED_VOLUMES) is None diff --git a/elyra/tests/pipeline/test_validation.py b/elyra/tests/pipeline/test_validation.py index 0af655dd4..a61e7fc52 100644 --- a/elyra/tests/pipeline/test_validation.py +++ b/elyra/tests/pipeline/test_validation.py @@ -20,15 +20,19 @@ from conftest import KFP_COMPONENT_CACHE_INSTANCE import pytest -from elyra.pipeline.pipeline import KubernetesAnnotation -from elyra.pipeline.pipeline import KubernetesSecret -from elyra.pipeline.pipeline import KubernetesToleration +from elyra.pipeline.component_parameter import ElyraPropertyList +from elyra.pipeline.component_parameter import EnvironmentVariable +from elyra.pipeline.component_parameter import KubernetesAnnotation +from elyra.pipeline.component_parameter import KubernetesSecret +from elyra.pipeline.component_parameter import KubernetesToleration +from elyra.pipeline.component_parameter import VolumeMount from elyra.pipeline.pipeline import PIPELINE_CURRENT_VERSION -from elyra.pipeline.pipeline import VolumeMount +from elyra.pipeline.pipeline_constants import ENV_VARIABLES from elyra.pipeline.pipeline_constants import KUBERNETES_POD_ANNOTATIONS from elyra.pipeline.pipeline_constants import KUBERNETES_SECRETS from elyra.pipeline.pipeline_constants import KUBERNETES_TOLERATIONS from elyra.pipeline.pipeline_constants import MOUNTED_VOLUMES +from elyra.pipeline.pipeline_definition import Node from elyra.pipeline.pipeline_definition import PipelineDefinition from elyra.pipeline.validation import PipelineValidationManager from elyra.pipeline.validation import ValidationResponse @@ -408,35 +412,90 @@ def test_invalid_node_property_resource_value(validation_manager, load_pipeline) def test_invalid_node_property_env_var(validation_manager): response = ValidationResponse() - node = {"id": "test-id", "app_data": {"label": "test"}} - invalid_env_var = 'TEST_ENV_ONE"test_one"' - validation_manager._validate_environmental_variables( - node_id=node["id"], node_label=node["app_data"]["label"], env_var=invalid_env_var, response=response + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} + + invalid_env_vars = ElyraPropertyList( + [ + EnvironmentVariable(env_var="TEST_ENV SPACE", value="value"), + EnvironmentVariable(env_var="", value="no key"), + ] + ) + node_dict["app_data"]["component_parameters"][ENV_VARIABLES] = invalid_env_vars + + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=ENV_VARIABLES, response=response ) issues = response.to_json().get("issues") assert issues[0]["severity"] == 1 - assert issues[0]["type"] == "invalidEnvPair" + assert issues[0]["type"] == "invalidEnvironmentVariable" assert issues[0]["data"]["propertyName"] == "env_vars" assert issues[0]["data"]["nodeID"] == "test-id" + assert issues[0]["message"] == "Environment variable 'TEST_ENV SPACE' includes invalid space character(s)." + + assert issues[0]["severity"] == 1 + assert issues[1]["type"] == "invalidEnvironmentVariable" + assert issues[1]["data"]["propertyName"] == "env_vars" + assert issues[1]["data"]["nodeID"] == "test-id" + assert issues[1]["message"] == "Required environment variable was not specified." + + +def test_valid_node_property_volumes(validation_manager): + response = ValidationResponse() + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} + + volumes = ElyraPropertyList( + [ + VolumeMount(path="/mount/test", pvc_name="rwx-test-claim"), # valid + VolumeMount(path="/mount/test_two", pvc_name="second-claim"), # valid + ] + ) + node_dict["app_data"]["component_parameters"][MOUNTED_VOLUMES] = volumes + + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=MOUNTED_VOLUMES, response=response + ) + issues = response.to_json().get("issues") + assert len(issues) == 0 def test_invalid_node_property_volumes(validation_manager): response = ValidationResponse() - node = {"id": "test-id", "app_data": {"label": "test"}} - volumes = [ - VolumeMount("/mount/test", "rwx-test-claim"), # valid - VolumeMount("/mount/test_two", "second-claim"), # valid - VolumeMount("/mount/test_four", "second#claim"), # invalid pvc name - ] - validation_manager._validate_mounted_volumes( - node_id=node["id"], node_label=node["app_data"]["label"], volumes=volumes, response=response + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} + + volumes = ElyraPropertyList( + [ + VolumeMount(path="", pvc_name=""), # missing mount path and pvc name + VolumeMount(path=None, pvc_name=None), # missing mount path and pvc name + VolumeMount(path="", pvc_name="pvc"), # missing mount path + VolumeMount(path=None, pvc_name="pvc"), # missing mount path + VolumeMount(path="/path", pvc_name=""), # missing pvc name + VolumeMount(path="/path/", pvc_name=None), # missing pvc name + VolumeMount(path="/mount/test_four", pvc_name="second#claim"), # invalid pvc name + ] + ) + node_dict["app_data"]["component_parameters"][MOUNTED_VOLUMES] = volumes + + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=MOUNTED_VOLUMES, response=response ) issues = response.to_json().get("issues") + assert len(issues) == 9, issues assert issues[0]["severity"] == 1 assert issues[0]["type"] == "invalidVolumeMount" assert issues[0]["data"]["propertyName"] == MOUNTED_VOLUMES assert issues[0]["data"]["nodeID"] == "test-id" - assert "not a valid Kubernetes resource name" in issues[0]["message"] + assert "Required mount path was not specified." in issues[0]["message"] + assert "Required persistent volume claim name was not specified." in issues[1]["message"] + assert "Required mount path was not specified." in issues[2]["message"] + assert "Required persistent volume claim name was not specified." in issues[3]["message"] + assert "Required mount path was not specified." in issues[4]["message"] + assert "Required mount path was not specified." in issues[5]["message"] + assert "Required persistent volume claim name was not specified." in issues[6]["message"] + assert "Required persistent volume claim name was not specified." in issues[7]["message"] + assert "PVC name 'second#claim' is not a valid Kubernetes resource name." in issues[8]["message"] def test_valid_node_property_kubernetes_toleration(validation_manager): @@ -446,18 +505,21 @@ def test_valid_node_property_kubernetes_toleration(validation_manager): https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#toleration-v1-core """ response = ValidationResponse() - node = {"id": "test-id", "app_data": {"label": "test"}} - # The following tolerations are valid - tolerations = [ - # parameters are key, operator, value, effect - KubernetesToleration("", "Exists", "", "NoExecute"), - KubernetesToleration("key0", "Exists", "", ""), - KubernetesToleration("key1", "Exists", "", "NoSchedule"), - KubernetesToleration("key2", "Equal", "value2", "NoExecute"), - KubernetesToleration("key3", "Equal", "value3", "PreferNoSchedule"), - ] - validation_manager._validate_kubernetes_tolerations( - node_id=node["id"], node_label=node["app_data"]["label"], tolerations=tolerations, response=response + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} + tolerations = ElyraPropertyList( + [ + KubernetesToleration(key="", operator="Exists", value="", effect="NoExecute"), + KubernetesToleration(key="key0", operator="Exists", value="", effect=""), + KubernetesToleration(key="key1", operator="Exists", value="", effect="NoSchedule"), + KubernetesToleration(key="key2", operator="Equal", value="value2", effect="NoExecute"), + KubernetesToleration(key="key3", operator="Equal", value="value3", effect="PreferNoSchedule"), + ] + ) + node_dict["app_data"]["component_parameters"][KUBERNETES_TOLERATIONS] = tolerations + + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=KUBERNETES_TOLERATIONS, response=response ) issues = response.to_json().get("issues") assert len(issues) == 0, response.to_json() @@ -470,25 +532,27 @@ def test_valid_node_property_kubernetes_pod_annotation(validation_manager): https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set """ response = ValidationResponse() - node = {"id": "test-id", "app_data": {"label": "test"}} - # The following annotations are valid - annotations = [ - # parameters are key and value - KubernetesAnnotation("k", ""), - KubernetesAnnotation("key", "value"), - KubernetesAnnotation("n-a-m-e", "value"), - KubernetesAnnotation("n.a.m.e", "value"), - KubernetesAnnotation("n_a_m_e", "value"), - KubernetesAnnotation("n-a.m_e", "value"), - KubernetesAnnotation("prefix/name", "value"), - KubernetesAnnotation("abc.def/name", "value"), - KubernetesAnnotation("abc.def.ghi/n-a-m-e", "value"), - KubernetesAnnotation("abc.def.ghi.jkl/n.a.m.e", "value"), - KubernetesAnnotation("abc.def.ghi.jkl.mno/n_a_m_e", "value"), - KubernetesAnnotation("abc.def.ghijklmno.pqr/n-a.m_e", "value"), - ] - validation_manager._validate_kubernetes_pod_annotations( - node_id=node["id"], node_label=node["app_data"]["label"], annotations=annotations, response=response + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} + annotations = ElyraPropertyList( + [ + KubernetesAnnotation(key="key", value="value"), + KubernetesAnnotation(key="n-a-m-e", value="value"), + KubernetesAnnotation(key="n.a.m.e", value="value"), + KubernetesAnnotation(key="n_a_m_e", value="value"), + KubernetesAnnotation(key="n-a.m_e", value="value"), + KubernetesAnnotation(key="prefix/name", value="value"), + KubernetesAnnotation(key="abc.def/name", value="value"), + KubernetesAnnotation(key="abc.def.ghi/n-a-m-e", value="value"), + KubernetesAnnotation(key="abc.def.ghi.jkl/n.a.m.e", value="value"), + KubernetesAnnotation(key="abc.def.ghi.jkl.mno/n_a_m_e", value="value"), + KubernetesAnnotation(key="abc.def.ghijklmno.pqr/n-a.m_e", value="value"), + ] + ) + node_dict["app_data"]["component_parameters"][KUBERNETES_POD_ANNOTATIONS] = annotations + + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=KUBERNETES_POD_ANNOTATIONS, response=response ) issues = response.to_json().get("issues") assert len(issues) == 0, response.to_json() @@ -501,39 +565,43 @@ def test_invalid_node_property_kubernetes_toleration(validation_manager): https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#toleration-v1-core """ response = ValidationResponse() - node = {"id": "test-id", "app_data": {"label": "test"}} - # The following tolerations are invalid - invalid_tolerations = [ - # parameters are key, operator, value, effect - KubernetesToleration("", "", "", ""), # cannot be all empty - # invalid values for 'operator' - KubernetesToleration("", "Equal", "value", ""), # empty key requires 'Exists' - KubernetesToleration("key0", "exists", "", ""), # wrong case - KubernetesToleration("key1", "Exist", "", ""), # wrong keyword - KubernetesToleration("key2", "", "", ""), # wrong keyword (technically valid but enforced) - # invalid values for 'value' - KubernetesToleration("key3", "Exists", "value3", ""), # 'Exists' -> no value - # invalid values for 'effect' - KubernetesToleration("key4", "Exists", "", "noschedule"), # wrong case - KubernetesToleration("key5", "Exists", "", "no-such-effect"), # wrong keyword - ] + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} + invalid_tolerations = ElyraPropertyList( + [ + KubernetesToleration(key="", operator="", value="", effect=""), # cannot be all empty + # invalid values for 'operator' + KubernetesToleration(key="", operator="Equal", value="value", effect=""), # empty key requires 'Exists' + KubernetesToleration(key="key0", operator="exists", value="", effect=""), # wrong case + KubernetesToleration(key="key1", operator="Exist", value="", effect=""), # wrong keyword + KubernetesToleration( + key="key2", operator="", value="", effect="" + ), # wrong keyword (technically valid but enforced) # noqa + # invalid values for 'value' + KubernetesToleration(key="key3", operator="Exists", value="value3", effect=""), # 'Exists' -> no value + # invalid values for 'effect' + KubernetesToleration(key="key4", operator="Exists", value="", effect="noschedule"), # wrong case + KubernetesToleration(key="key5", operator="Exists", value="", effect="no-such-effect"), # wrong keyword + ] + ) expected_error_messages = [ - "'' is not a valid operator. The value must be one of 'Exists' or 'Equal'.", - "'Equal' is not a valid operator. Operator must be 'Exists' if no key is specified.", - "'exists' is not a valid operator. The value must be one of 'Exists' or 'Equal'.", - "'Exist' is not a valid operator. The value must be one of 'Exists' or 'Equal'.", - "'' is not a valid operator. The value must be one of 'Exists' or 'Equal'.", - "'value3' is not a valid value. It should be empty if operator is 'Exists'.", - "'noschedule' is not a valid effect. Effect must be one of 'NoExecute', 'NoSchedule', or 'PreferNoSchedule'.", - "'no-such-effect' is not a valid effect. Effect must be one of 'NoExecute', " + "'' is not a valid operator: the value must be one of 'Exists' or 'Equal'.", + "'Equal' is not a valid operator: operator must be 'Exists' if no key is specified.", + "'exists' is not a valid operator: the value must be one of 'Exists' or 'Equal'.", + "'Exist' is not a valid operator: the value must be one of 'Exists' or 'Equal'.", + "'' is not a valid operator: the value must be one of 'Exists' or 'Equal'.", + "'value3' is not a valid value: value should be empty if operator is 'Exists'.", + "'noschedule' is not a valid effect: effect must be one of 'NoExecute', 'NoSchedule', or 'PreferNoSchedule'.", + "'no-such-effect' is not a valid effect: effect must be one of 'NoExecute', " "'NoSchedule', or 'PreferNoSchedule'.", ] # verify that the number of tolerations in this test matches the number of error messages assert len(invalid_tolerations) == len(expected_error_messages), "Test setup error. " + node_dict["app_data"]["component_parameters"][KUBERNETES_TOLERATIONS] = invalid_tolerations - validation_manager._validate_kubernetes_tolerations( - node_id=node["id"], node_label=node["app_data"]["label"], tolerations=invalid_tolerations, response=response + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=KUBERNETES_TOLERATIONS, response=response ) issues = response.to_json().get("issues") assert len(issues) == len(invalid_tolerations), response.to_json() @@ -548,44 +616,51 @@ def test_invalid_node_property_kubernetes_toleration(validation_manager): def test_invalid_node_property_kubernetes_pod_annotation(validation_manager): """ - Validate that valid kubernetes pod annotation definitions are not flagged as invalid. + Validate that invalid kubernetes pod annotation definitions are flagged as invalid. Constraints are documented in https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set """ response = ValidationResponse() - node = {"id": "test-id", "app_data": {"label": "test"}} + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} TOO_SHORT_LENGTH = 0 MAX_PREFIX_LENGTH = 253 MAX_NAME_LENGTH = 63 TOO_LONG_LENGTH = MAX_PREFIX_LENGTH + 1 + MAX_NAME_LENGTH + 1 # prefix + '/' + name # The following annotations are invalid - invalid_annotations = [ - # parameters are key and value - # test length violations (key name and prefix) - KubernetesAnnotation("a" * (TOO_SHORT_LENGTH), ""), # empty key (min 1) - KubernetesAnnotation("a" * (TOO_LONG_LENGTH), ""), # key too long - KubernetesAnnotation(f"{'a' * (MAX_PREFIX_LENGTH + 1)}/b", ""), # key prefix too long - KubernetesAnnotation(f"{'a' * (MAX_NAME_LENGTH + 1)}", ""), # key name too long - KubernetesAnnotation(f"prefix/{'a' * (MAX_NAME_LENGTH + 1)}", ""), # key name too long - KubernetesAnnotation(f"{'a' * (MAX_PREFIX_LENGTH + 1)}/name", ""), # key prefix too long - # test character violations (key name) - KubernetesAnnotation("-", ""), # name must start and end with alphanum - KubernetesAnnotation("-a", ""), # name must start with alphanum - KubernetesAnnotation("a-", ""), # name must start with alphanum - KubernetesAnnotation("prefix/-b", ""), # name start with alphanum - KubernetesAnnotation("prefix/b-", ""), # name must end with alphanum - # test character violations (key prefix) - KubernetesAnnotation("PREFIX/name", ""), # prefix must be lowercase - KubernetesAnnotation("pref!x/name", ""), # prefix must contain alnum, '-' or '.' - KubernetesAnnotation("pre.fx./name", ""), # prefix must contain alnum, '-' or '.' - KubernetesAnnotation("-pre.fx.com/name", ""), # prefix must contain alnum, '-' or '.' - KubernetesAnnotation("pre.fx-./name", ""), # prefix must contain alnum, '-' or '.' - KubernetesAnnotation("a/b/c", ""), # only one separator char - ] + invalid_annotations = ElyraPropertyList( + [ + # test length violations (key name and prefix) + KubernetesAnnotation(key="a", value=""), # empty value (min 1) + KubernetesAnnotation(key="a", value=None), # empty value (min 1) + KubernetesAnnotation(key="a" * TOO_SHORT_LENGTH, value="val"), # empty key (min 1) + KubernetesAnnotation(key=None, value="val"), # empty key (min 1) + KubernetesAnnotation(key="a" * TOO_LONG_LENGTH, value="val"), # key too long + KubernetesAnnotation(key=f"{'a' * (MAX_PREFIX_LENGTH + 1)}/b", value="val"), # key prefix too long + KubernetesAnnotation(key=f"{'a' * (MAX_NAME_LENGTH + 1)}", value="val"), # key name too long + KubernetesAnnotation(key=f"prefix/{'a' * (MAX_NAME_LENGTH + 1)}", value="val"), # key name too long + KubernetesAnnotation(key=f"{'a' * (MAX_PREFIX_LENGTH + 1)}/name", value="val"), # key prefix too long + # test character violations (key name) + KubernetesAnnotation(key="-", value="val"), # name must start and end with alphanum + KubernetesAnnotation(key="-a", value="val"), # name must start with alphanum + KubernetesAnnotation(key="a-", value="val"), # name must start with alphanum + KubernetesAnnotation(key="prefix/-b", value="val"), # name start with alphanum + KubernetesAnnotation(key="prefix/b-", value="val"), # name must end with alphanum + # test character violations (key prefix) + KubernetesAnnotation(key="PREFIX/name", value="val"), # prefix must be lowercase + KubernetesAnnotation(key="pref!x/name", value="val"), # prefix must contain alnum, '-' or '.' + KubernetesAnnotation(key="pre.fx./name", value="val"), # prefix must contain alnum, '-' or '.' + KubernetesAnnotation(key="-pre.fx.com/name", value="val"), # prefix must contain alnum, '-' or '.' + KubernetesAnnotation(key="pre.fx-./name", value="val"), # prefix must contain alnum, '-' or '.' + KubernetesAnnotation(key="a/b/c", value="val"), # only one separator char + ] + ) expected_error_messages = [ - "'' is not a valid Kubernetes annotation key.", - f"'{'a' * (TOO_LONG_LENGTH)}' is not a valid Kubernetes annotation key.", + "Required annotation value was not specified.", + "Required annotation value was not specified.", + "Required annotation key was not specified.", + "Required annotation key was not specified.", + f"'{'a' * TOO_LONG_LENGTH}' is not a valid Kubernetes annotation key.", f"'{'a' * (MAX_PREFIX_LENGTH + 1)}/b' is not a valid Kubernetes annotation key.", f"'{'a' * (MAX_NAME_LENGTH + 1)}' is not a valid Kubernetes annotation key.", f"'prefix/{'a' * (MAX_NAME_LENGTH + 1)}' is not a valid Kubernetes annotation key.", @@ -605,9 +680,11 @@ def test_invalid_node_property_kubernetes_pod_annotation(validation_manager): # verify that the number of annotations in this test matches the number of error messages assert len(invalid_annotations) == len(expected_error_messages), "Test implementation error. " + node_dict["app_data"]["component_parameters"][KUBERNETES_POD_ANNOTATIONS] = invalid_annotations - validation_manager._validate_kubernetes_pod_annotations( - node_id=node["id"], node_label=node["app_data"]["label"], annotations=invalid_annotations, response=response + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=KUBERNETES_POD_ANNOTATIONS, response=response ) issues = response.to_json().get("issues") assert len(issues) == len( @@ -622,27 +699,78 @@ def test_invalid_node_property_kubernetes_pod_annotation(validation_manager): index = index + 1 +def test_valid_node_property_secrets(validation_manager): + response = ValidationResponse() + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} + secrets = ElyraPropertyList( + [ + KubernetesSecret(env_var="ENV_VAR1", name="test-secret", key="test-key1"), # valid + KubernetesSecret(env_var="ENV_VAR2", name="test-secret", key="test-key2"), # valid + ] + ) + node_dict["app_data"]["component_parameters"][KUBERNETES_SECRETS] = secrets + + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=KUBERNETES_SECRETS, response=response + ) + issues = response.to_json().get("issues") + assert len(issues) == 0, issues + + def test_invalid_node_property_secrets(validation_manager): response = ValidationResponse() - node = {"id": "test-id", "app_data": {"label": "test"}} - secrets = [ - KubernetesSecret("ENV_VAR1", "test-secret", "test-key1"), # valid - KubernetesSecret("ENV_VAR2", "test-secret", "test-key2"), # valid - KubernetesSecret("ENV_VAR3", "test-secret", ""), # invalid: improper format of secret name/key - KubernetesSecret("ENV_VAR5", "test%secret", "test-key"), # invalid: not a valid Kubernetes resource name - KubernetesSecret("ENV_VAR6", "test-secret", "test$key2"), # invalid: not a valid Kubernetes secret key - ] - validation_manager._validate_kubernetes_secrets( - node_id=node["id"], node_label=node["app_data"]["label"], secrets=secrets, response=response + node_dict = {"id": "test-id", "app_data": {"label": "test", "ui_data": {}, "component_parameters": {}}} + secrets = ElyraPropertyList( + [ + KubernetesSecret(env_var="", name="test-secret", key="test-key1"), # missing env var name + KubernetesSecret(env_var=None, name="test-secret", key="test-key1"), # missing env var name + KubernetesSecret(env_var="ENV_VAR1", name="", key="key"), # missing secret name + KubernetesSecret(env_var="ENV_VAR2", name=None, key="key"), # missing secret name + KubernetesSecret(env_var="ENV_VAR3", name="test-secret", key=""), # missing secret key + KubernetesSecret(env_var="ENV_VAR4", name="test-secret", key=None), # missing secret key + KubernetesSecret(env_var="ENV_VAR5", name="test%secret", key="test-key"), # invalid k8s resource name + KubernetesSecret(env_var="ENV_VAR6", name="test-secret", key="test$key2"), # invalid k8s secret key + KubernetesSecret(env_var="", name="", key=""), # invalid - all required information is missing + KubernetesSecret(env_var=None, name=None, key=None), # invalid - all required information is missing + ] + ) + node_dict["app_data"]["component_parameters"][KUBERNETES_SECRETS] = secrets + + node = Node(node_dict) + validation_manager._validate_elyra_owned_property( + node_id=node.id, node_label=node.label, node=node, param_name=KUBERNETES_SECRETS, response=response ) issues = response.to_json().get("issues") + assert len(issues) == 14, issues assert issues[0]["severity"] == 1 assert issues[0]["type"] == "invalidKubernetesSecret" assert issues[0]["data"]["propertyName"] == KUBERNETES_SECRETS assert issues[0]["data"]["nodeID"] == "test-id" - assert "improperly formatted representation of secret name and key" in issues[0]["message"] - assert "not a valid Kubernetes resource name" in issues[1]["message"] - assert "not a valid Kubernetes secret key" in issues[2]["message"] + + # triggered by KubernetesSecret(env_var="", name="test-secret", key="test-key1") + assert "Required environment variable was not specified." in issues[0]["message"] + # triggered by KubernetesSecret(env_var=None, name="test-secret", key="test-key1") + assert "Required environment variable was not specified." in issues[1]["message"] + # triggered by KubernetesSecret(env_var="ENV_VAR1", name="", key="key") + assert "Required secret name was not specified." in issues[2]["message"] + # triggered by KubernetesSecret(env_var="ENV_VAR2", name=None, key="key") + assert "Required secret name was not specified." in issues[3]["message"] + # triggered by KubernetesSecret(env_var="ENV_VAR3", name="test-secret", key="") + assert "Required secret key was not specified." in issues[4]["message"] + # triggered by KubernetesSecret(env_var="ENV_VAR4", name="test-secret", key=None) + assert "Required secret key was not specified." in issues[5]["message"] + # triggered by KubernetesSecret(env_var="ENV_VAR5", name="test%secret", key="test-key") + assert "Secret name 'test%secret' is not a valid Kubernetes resource name." in issues[6]["message"] + # triggered by KubernetesSecret(env_var="ENV_VAR6", name="test-secret", key="test$key2") + assert "Key 'test$key2' is not a valid Kubernetes secret key." in issues[7]["message"] + # triggered by KubernetesSecret(env_var="", name="", key="") + assert "Required environment variable was not specified." in issues[8]["message"] + assert "Required secret name was not specified." in issues[9]["message"] + assert "Required secret key was not specified." in issues[10]["message"] + assert "Required environment variable was not specified." in issues[11]["message"] + assert "Required secret name was not specified." in issues[12]["message"] + assert "Required secret key was not specified." in issues[13]["message"] def test_valid_node_property_label(validation_manager): diff --git a/elyra/tests/pipeline/util.py b/elyra/tests/pipeline/util.py index 433f04ff1..c8b0c9a1a 100644 --- a/elyra/tests/pipeline/util.py +++ b/elyra/tests/pipeline/util.py @@ -21,6 +21,7 @@ from typing import Optional import uuid +from elyra.pipeline.component_parameter import ElyraProperty from elyra.pipeline.pipeline import GenericOperation from elyra.pipeline.pipeline import Pipeline @@ -77,30 +78,31 @@ def __init__( def get_operation(self) -> GenericOperation: - self.env_vars = [] + env_vars = [] if self.fail: # NODE_FILENAME is required, so skip if triggering failure if "NODE_FILENAME" in os.environ: # remove entry if present os.environ.pop("NODE_FILENAME") else: - self.env_vars.append(f"NODE_FILENAME={self.filename}") + env_vars.append({"env_var": "NODE_FILENAME", "value": self.filename}) if self.inputs: - self.env_vars.append(f"INPUT_FILENAMES={';'.join(self.inputs)}") + env_vars.append({"env_var": "INPUT_FILENAMES", "value": ";".join(self.inputs)}) if self.outputs: - self.env_vars.append(f"OUTPUT_FILENAMES={';'.join(self.outputs)}") + env_vars.append({"env_var": "OUTPUT_FILENAMES", "value": ";".join(self.outputs)}) # Convey the pipeline name assert self.pipeline_name is not None, "Pipeline name has not been set during construction!" - self.env_vars.append(f"PIPELINE_NAME={self.pipeline_name}") + env_vars.append({"env_var": "PIPELINE_NAME", "value": self.pipeline_name}) # Add system-owned here with bogus or no value... - self.env_vars.append("ELYRA_RUNTIME_ENV=bogus_runtime") + env_vars.append({"env_var": "ELYRA_RUNTIME_ENV", "value": "bogus_runtime"}) + + self.env_vars = ElyraProperty.create_instance("env_vars", env_vars) component_parameters = { "filename": self.filename, "runtime_image": self.image_name or "NA", "dependencies": self.dependencies, - "env_vars": self.env_vars, "inputs": self.inputs, "outputs": self.outputs, } @@ -111,6 +113,7 @@ def get_operation(self) -> GenericOperation: self.classifier, parent_operation_ids=self.parent_operations, component_params=component_parameters, + elyra_params={"env_vars": self.env_vars}, ) diff --git a/packages/pipeline-editor/package.json b/packages/pipeline-editor/package.json index 9560599f7..e39f9f80d 100644 --- a/packages/pipeline-editor/package.json +++ b/packages/pipeline-editor/package.json @@ -46,8 +46,8 @@ }, "dependencies": { "@elyra/metadata-common": "3.12.0-dev", - "@elyra/pipeline-editor": "1.10.0-rc.0", - "@elyra/pipeline-services": "1.10.0-rc.0", + "@elyra/pipeline-editor": "~1.10.0-rc.1", + "@elyra/pipeline-services": "~1.10.0-rc.1", "@elyra/services": "3.12.0-dev", "@elyra/ui-components": "3.12.0-dev", "@jupyterlab/application": "^3.4.6", diff --git a/packages/pipeline-editor/src/PipelineEditorWidget.tsx b/packages/pipeline-editor/src/PipelineEditorWidget.tsx index fc5876dcf..d5c72712a 100644 --- a/packages/pipeline-editor/src/PipelineEditorWidget.tsx +++ b/packages/pipeline-editor/src/PipelineEditorWidget.tsx @@ -304,14 +304,23 @@ const PipelineWrapper: React.FC = ({ }, [runtimeDisplayName]); const onChange = useCallback((pipelineJson: any): void => { - const removeNullValues = (data: any): void => { + const removeNullValues = (data: any, removeEmptyString?: boolean): void => { for (const key in data) { - if (data[key] === null) { + if ( + data[key] === null || + data[key] === undefined || + (removeEmptyString && data[key] === '') + ) { delete data[key]; } else if (Array.isArray(data[key])) { const newArray = []; for (const i in data[key]) { - if (data[key][i] !== null && data[key][i] !== '') { + if (typeof data[key][i] === 'object') { + removeNullValues(data[key][i], true); + if (Object.keys(data[key][i]).length > 0) { + newArray.push(data[key][i]); + } + } else if (data[key][i] !== null && data[key][i] !== '') { newArray.push(data[key][i]); } } @@ -487,20 +496,28 @@ const PipelineWrapper: React.FC = ({ }; const onPropertiesUpdateRequested = async (args: any): Promise => { + if (!contextRef.current.path) { + return args; + } const path = PipelineService.getWorkspaceRelativeNodePath( contextRef.current.path, args.component_parameters.filename ); - const new_env_vars = await ContentParser.getEnvVars( - path - ).then((response: any) => response.map((str: string) => str + '=')); + const new_env_vars = await ContentParser.getEnvVars(path).then( + (response: any) => + response.map((str: string) => { + return { env_var: str }; + }) + ); const env_vars = args.component_parameters?.env_vars ?? []; const merged_env_vars = [ ...env_vars, ...new_env_vars.filter( - (new_var: string) => - !env_vars.some((old_var: string) => old_var.startsWith(new_var)) + (new_var: any) => + !env_vars.some((old_var: any) => { + return old_var.env_var === new_var.env_var; + }) ) ]; diff --git a/packages/pipeline-editor/style/index.css b/packages/pipeline-editor/style/index.css index 4f19180e1..74844bd21 100644 --- a/packages/pipeline-editor/style/index.css +++ b/packages/pipeline-editor/style/index.css @@ -511,6 +511,26 @@ span.bx--list-box__label { margin-right: 3px; } +.field-array-of-object .array-item:hover { + background-color: var(--jp-border-color3); +} + +.field-array-of-object .array-item { + display: flex; + flex-direction: column; + border: 1px solid var(--jp-border-color2); + padding: 10px; + border-radius: 5px; +} + +.field-array-of-object button.jp-mod-styled.jp-mod-warn { + width: fit-content; +} + +.field-array-of-object label.control-label { + margin-top: 5px; +} + .elyra-actionItem:hover { background-color: var(--jp-border-color1); } diff --git a/packages/ui-components/style/formeditor.css b/packages/ui-components/style/formeditor.css index 6183d77fb..a8f00e04a 100644 --- a/packages/ui-components/style/formeditor.css +++ b/packages/ui-components/style/formeditor.css @@ -318,7 +318,8 @@ button.elyra-editor-tag.unapplied-tag { padding-right: 10px; } -.elyra-formEditor .array-item .form-group.field input { +.elyra-formEditor .array-item .form-group.field input, +.elyra-formEditor .array-item .form-group.field select { margin-top: 0; } diff --git a/tests/assets/helloworld.pipeline b/tests/assets/helloworld.pipeline index 80fca52d3..c7ed16269 100644 --- a/tests/assets/helloworld.pipeline +++ b/tests/assets/helloworld.pipeline @@ -18,8 +18,14 @@ "filename": "helloworld.ipynb", "runtime_image": "continuumio/anaconda3:2020.07", "env_vars": [ - "TEST_ENV_1=1", - "TEST_ENV_2=2" + { + "env_var": "TEST_ENV_1", + "value": "1" + }, + { + "env_var": "TEST_ENV_2", + "value": "2" + } ], "include_subdirectories": false, "outputs": [], @@ -122,7 +128,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "properties": { "name": "helloworld", "runtime": "Generic" @@ -132,4 +138,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/tests/assets/invalid.pipeline b/tests/assets/invalid.pipeline index aeee884c8..bbe5dea8b 100644 --- a/tests/assets/invalid.pipeline +++ b/tests/assets/invalid.pipeline @@ -137,7 +137,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "properties": { "name": "invalid", "runtime": "Generic" @@ -147,4 +147,4 @@ } ], "schemas": [] -} \ No newline at end of file +} diff --git a/tests/assets/pipelines/producer-consumer.pipeline b/tests/assets/pipelines/producer-consumer.pipeline index 64868d5b0..e492ad809 100644 --- a/tests/assets/pipelines/producer-consumer.pipeline +++ b/tests/assets/pipelines/producer-consumer.pipeline @@ -18,7 +18,10 @@ "filename": "producer.ipynb", "runtime_image": "elyra/elyra-example-anaconda-py3:latest", "env_vars": [ - "NB_USER=" + { + "env_var": "NB_USER", + "value": "" + } ], "include_subdirectories": false, "outputs": [ @@ -314,7 +317,7 @@ "ui_data": { "comments": [] }, - "version": 7.5, + "version": 8, "properties": { "name": "producer-consumer", "runtime": "Generic" diff --git a/tests/integration/pipeline.ts b/tests/integration/pipeline.ts index 0d5196a03..a21a10137 100644 --- a/tests/integration/pipeline.ts +++ b/tests/integration/pipeline.ts @@ -541,27 +541,35 @@ describe('Pipeline Editor tests', () => { cy.findByRole('menuitem', { name: /properties/i }).click(); - cy.get('input[value="TEST_ENV_1=1"]').should('exist'); + cy.get('input[value="TEST_ENV_1"]').should('exist'); + cy.get('input[value="1"]').should('exist'); cy.findByText('helloworld.py').click(); cy.get('#root_component_parameters_env_vars').within(() => { cy.findByRole('button', { name: /add/i }).click(); - cy.get('input[id="root_component_parameters_env_vars_0"]').type( - 'BAD=two' + cy.get('input[id="root_component_parameters_env_vars_0_env_var"]').type( + 'BAD' + ); + cy.get('input[id="root_component_parameters_env_vars_0_value"]').type( + 'two' ); }); - cy.get('input[value="BAD=two"]').should('exist'); + cy.get('input[value="BAD"]').should('exist'); + cy.get('input[value="two"]').should('exist'); cy.findByText('helloworld.ipynb').click(); - cy.get('input[value="TEST_ENV_1=1"]').should('exist'); - cy.get('input[value="BAD=two"]').should('not.exist'); + cy.get('input[value="TEST_ENV_1"]').should('exist'); + cy.get('input[value="1"]').should('exist'); + cy.get('input[value="BAD"]').should('not.exist'); + cy.get('input[value="two"]').should('not.exist'); cy.findByText('helloworld.py').click(); - cy.get('input[value="BAD=two"]').should('exist'); + cy.get('input[value="BAD"]').should('exist'); + cy.get('input[value="two"]').should('exist'); }); }); diff --git a/tests/snapshots/pipeline-editor-tests/matches-complex-pipeline-snapshot.1.snap b/tests/snapshots/pipeline-editor-tests/matches-complex-pipeline-snapshot.1.snap index acd649af3..4097e0f62 100644 --- a/tests/snapshots/pipeline-editor-tests/matches-complex-pipeline-snapshot.1.snap +++ b/tests/snapshots/pipeline-editor-tests/matches-complex-pipeline-snapshot.1.snap @@ -12,7 +12,7 @@ Object { "ui_data": Object { "comments": Array [], }, - "version": 7.5, + "version": 8, }, "id": "primary", "nodes": Array [ @@ -21,7 +21,9 @@ Object { "component_parameters": Object { "dependencies": Array [], "env_vars": Array [ - "NB_USER=", + Object { + "env_var": "NB_USER", + }, ], "filename": "producer.ipynb", "include_subdirectories": false, diff --git a/tests/snapshots/pipeline-editor-tests/matches-empty-pipeline-snapshot.1.snap b/tests/snapshots/pipeline-editor-tests/matches-empty-pipeline-snapshot.1.snap index c2bb77a12..b9d7f78e0 100644 --- a/tests/snapshots/pipeline-editor-tests/matches-empty-pipeline-snapshot.1.snap +++ b/tests/snapshots/pipeline-editor-tests/matches-empty-pipeline-snapshot.1.snap @@ -12,7 +12,7 @@ Object { "ui_data": Object { "comments": Array [], }, - "version": 7.5, + "version": 8, }, "id": "primary", "nodes": Array [], diff --git a/tests/snapshots/pipeline-editor-tests/matches-simple-pipeline-snapshot.1.snap b/tests/snapshots/pipeline-editor-tests/matches-simple-pipeline-snapshot.1.snap index c4d792c69..ecb7e7a2a 100644 --- a/tests/snapshots/pipeline-editor-tests/matches-simple-pipeline-snapshot.1.snap +++ b/tests/snapshots/pipeline-editor-tests/matches-simple-pipeline-snapshot.1.snap @@ -12,7 +12,7 @@ Object { "ui_data": Object { "comments": Array [], }, - "version": 7.5, + "version": 8, }, "id": "primary", "nodes": Array [ @@ -21,8 +21,12 @@ Object { "component_parameters": Object { "dependencies": Array [], "env_vars": Array [ - "TEST_ENV_1=", - "TEST_ENV_2=", + Object { + "env_var": "TEST_ENV_1", + }, + Object { + "env_var": "TEST_ENV_2", + }, ], "filename": "helloworld.ipynb", "include_subdirectories": false, diff --git a/yarn.lock b/yarn.lock index 45dd62cdc..b2f83dbd4 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1267,13 +1267,13 @@ seedrandom "^3.0.5" uuid "^8.3.0" -"@elyra/pipeline-editor@1.10.0-rc.0": - version "1.10.0-rc.0" - resolved "https://registry.yarnpkg.com/@elyra/pipeline-editor/-/pipeline-editor-1.10.0-rc.0.tgz#3fc84753d224cf277335983fcc629645c62962fd" - integrity sha512-+OHdWEpvjsjqY1LVktgcHuc77z1FXWoo879rzf6VWEifu/ROnTXqjEVjryKyaJC03TA5+/OmPfJLM++7r0/70w== +"@elyra/pipeline-editor@~1.10.0-rc.1": + version "1.10.0-rc.1" + resolved "https://registry.yarnpkg.com/@elyra/pipeline-editor/-/pipeline-editor-1.10.0-rc.1.tgz#b2427ffcccd9b33e5d2d9ff9b5fced4c45083391" + integrity sha512-vtgXTO6/+Qf5U7YJSRxHw9XRZMPB5jC3URgfV4aU84Fj9vmed/8+kBO5hbyF9vDAAdTSfVFYzvY9bIEjOJo9qg== dependencies: "@elyra/canvas" "^12.11.2" - "@elyra/pipeline-services" "^1.10.0-rc.0" + "@elyra/pipeline-services" "^1.10.0-rc.1" "@rjsf/core" "^4.2.3" "@rjsf/utils" "^5.0.0-beta.2" "@rjsf/validator-ajv6" "^5.0.0-beta.2" @@ -1294,10 +1294,10 @@ resolved "https://registry.yarnpkg.com/@elyra/pipeline-schemas/-/pipeline-schemas-3.0.60.tgz#60ed19a3175563dd6639a1983923926dc85581f2" integrity sha512-g+wxeie99eocdduoP++TEKZgD9JlBWxvO1ysRGts2ehko0mJryUCmkWADNKIjRaE0qOU/363X9/ScwNUGpJkpQ== -"@elyra/pipeline-services@1.10.0-rc.0", "@elyra/pipeline-services@^1.10.0-rc.0": - version "1.10.0-rc.0" - resolved "https://registry.yarnpkg.com/@elyra/pipeline-services/-/pipeline-services-1.10.0-rc.0.tgz#da1e5267a4dafe2a2d32ed9c507302a3c6a029ff" - integrity sha512-32RWC/B3lwnnLrycEbahZUugH3KZiy/wRS2WJlP0BMi7WrFgHxz0wZ7T40Z6SDEfAWVvmuaKZTBmSI4iINdYXA== +"@elyra/pipeline-services@^1.10.0-rc.1", "@elyra/pipeline-services@~1.10.0-rc.1": + version "1.10.0-rc.1" + resolved "https://registry.yarnpkg.com/@elyra/pipeline-services/-/pipeline-services-1.10.0-rc.1.tgz#3108796ef18abc66451d10197dcab6c58d1b2cfe" + integrity sha512-uegTPfrBufUma9QV5cOlh8xonGVRl1O1lGKLFyUwwXPpeErg3xmZFl2nUHH0WhDRzc1cvTTBDUD1XRXraV3iJw== dependencies: immer "^9.0.7" jsonc-parser "^3.0.0"