Filter available runtimes based on configuration setting

docs/source/index.rst

@@ -44,6 +44,7 @@ Elyra is a set of AI-centric extensions to JupyterLab Notebooks.
 
    user_guide/jupyterlab-interface.md
    user_guide/command-line-interface.md
+   user_guide/pipeline-editor-configuration.md
    user_guide/runtime-conf.md
    user_guide/runtime-image-conf.md
    user_guide/pipelines.md

docs/source/user_guide/jupyterlab-interface.md

@@ -24,7 +24,7 @@ Many of these tasks can also be accomplished using the [Elyra command line inter
 
 ### Launcher
 
-Elyra adds a new category to the JupyterLab launcher, providing access to the Visual Pipeline Editor, the [Python editor](enhanced-script-support.html#python-script-execution-support), the [R editor](enhanced-script-support.html#r-script-execution-support), and the [Elyra documentation](https://elyra.readthedocs.io/en/latest/).
+Elyra adds a new category to the JupyterLab launcher, providing access to the [Visual Pipeline Editor](#visual-pipeline-editor), the [Python editor](enhanced-script-support.html#python-script-execution-support), the [R editor](enhanced-script-support.html#r-script-execution-support), and the [Elyra documentation](https://elyra.readthedocs.io/en/latest/).
 
 ![Elyra category in JupyterLab launcher](../images/user_guide/jupyterlab-interface/launcher.png)
 
@@ -44,7 +44,7 @@ The canvas is the main work area, where you [assemble the pipeline by adding nod
 
 The properties panel is used to configure pipeline properties and node properties.
 
-You can customize the pipeline editor behavior by opening the settings link in the empty editor window or by navigating in the [JupyterLab menu bar](https://jupyterlab.readthedocs.io/en/stable/user/interface.html#menu-bar) to  `Settings > Advanced Settings Editor` and searching for `elyra`.
+Refer to the [_Configuring the pipeline editor_ topic](pipeline-editor-configuration.md) to learn about customizing the editor. 
 
 ### Metadata management sidebars
 

docs/source/user_guide/pipeline-editor-configuration.md

@@ -0,0 +1,90 @@
+<!--
+{% comment %}
+Copyright 2018-2023 Elyra Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Configuring the pipeline editor
+
+### Configuring supported runtimes
+
+The pipeline editor supports three runtimes: Kubeflow Pipelines, Apache Airflow, and local execution in JupyterLab. By default, support for all runtimes is enabled when you [install Elyra](../getting_started/installation.md). The JupyterLab launcher window under the _Elyra_ category includes a tile for each enabled runtime:
+
+![Pipeline editor tiles in the JupyterLab launcher](../images/user_guide/pipeline-editor-configuration/pipeline-editor-tiles-in-jupyterlab-launcher.png)
+
+If you are planning to use only a subset of the supported runtimes to execute pipelines, you can enable them selectively.
+
+#### Enabling specific runtimes
+
+When you explicitly enable one or more runtimes the other runtimes are disabled. You enable runtimes by overriding the Elyra default configuration. 
+
+##### Override default using command line parameters
+
+To enable one or more runtimes, launch JupyterLab with the Elyra-specific `--PipelineProcessorRegistry.runtimes` parameter:
+
+```
+$ jupyter lab --PipelineProcessorRegistry.runtimes=<runtime>
+```
+
+Supported parameter values for `<runtime>` are `kfp` (enable support for Kubeflow Pipelines), `airflow` (enable support for Apache Airflow), and `local` (enable support for local execution).
+
+For example, to enable only support for Kubeflow Pipelines, run
+
+```
+$ jupyter lab --PipelineProcessorRegistry.runtimes=kfp
+```
+
+![Kubeflow Pipelines editor tile in the JupyterLab launcher](../images/user_guide/pipeline-editor-configuration/pipeline-editor-kfp-only-in-jupyterlab-launcher.png)
+
+To enable support for more than one runtime, specify the parameter multiple times.
+
+```
+$ jupyter lab --PipelineProcessorRegistry.runtimes=kfp --PipelineProcessorRegistry.runtimes=local
+```
+
+![Kubeflow Pipelines and generic editor tiles in the JupyterLab launcher](../images/user_guide/pipeline-editor-configuration/pipeline-editor-subset-in-jupyterlab-launcher.png)
+
+##### Override default using customized configuration file
+
+To permanently apply your runtime selection create a customized configuration file.
+
+1. Stop JupyterLab.
+1. Generate the `jupyter_elyra_config.py` configuration file.
+
+   ```
+   $ jupyter elyra --generate-config
+   ```
+   > Note: You must specify `elyra` as the `jupyter` subcommand instead of `lab`. 
+1. Open the generated configuration file.
+1. Locate the `PipelineProcessorRegistry` configuration section.
+   ```
+   #------------------------------------------------------------------------------
+   # PipelineProcessorRegistry(SingletonConfigurable) configuration
+   #------------------------------------------------------------------------------
+   ```
+1. Locate the configuration entry for `PipelineProcessorRegistry.runtimes`
+   ```
+   # c.PipelineProcessorRegistry.runtimes = []
+   ```
+1. Remove the leading `#` and add one or more of `kfp`,`airflow`, or `local`.
+   ```
+   c.PipelineProcessorRegistry.runtimes = ['kfp', 'local']
+   ```
+1. Save the customized configuration file.
+1. Start JupyterLab. The pipeline editor tiles for the specified runtimes are displayed in the launcher window.
+
+### Customizing the pipeline editor
+
+You can customize the pipeline editor behavior by opening the settings link in the empty editor window or by navigating in the [JupyterLab menu bar](https://jupyterlab.readthedocs.io/en/stable/user/interface.html#menu-bar) to  `Settings > Advanced Settings Editor` and searching for `elyra`. Customization options vary by release.

elyra/elyra_app.py

@@ -44,7 +44,7 @@
 from elyra.pipeline.handlers import PipelineValidationHandler
 from elyra.pipeline.processor import PipelineProcessor
 from elyra.pipeline.processor import PipelineProcessorManager
-from elyra.pipeline.processor import PipelineProcessorRegistry
+from elyra.pipeline.registry import PipelineProcessorRegistry
 from elyra.pipeline.validation import PipelineValidationManager
 
 
@@ -60,7 +60,14 @@ class ElyraApp(ExtensionAppJinjaMixin, ExtensionApp):
     extension_url = "/lab"
     load_other_extensions = True
 
-    classes = [FileMetadataCache, MetadataManager, PipelineProcessor, ComponentCatalogConnector, ComponentCache]
+    classes = [
+        FileMetadataCache,
+        MetadataManager,
+        PipelineProcessorRegistry,
+        PipelineProcessor,
+        ComponentCatalogConnector,
+        ComponentCache,
+    ]
 
     # Local path to static files directory.
     static_paths = [

elyra/pipeline/component_catalog.py

@@ -44,6 +44,7 @@
 from elyra.pipeline.component import ComponentParser
 from elyra.pipeline.component_metadata import ComponentCatalogMetadata
 from elyra.pipeline.properties import ComponentProperty
+from elyra.pipeline.registry import PipelineProcessorRegistry
 from elyra.pipeline.runtime_type import RuntimeProcessorType
 
 BLOCKING_TIMEOUT = 0.5
@@ -451,7 +452,7 @@ def update(self, catalog: Metadata, action: str):
         """
         self._insert_request(self.update_queue, catalog, action)
 
-    def _insert_request(self, queue: Queue, catalog: Metadata, action: str):
+    def _insert_request(self, queue: Queue, catalog: ComponentCatalogMetadata, action: str):
         """
         If running as a server process, the request is submitted to the desired queue, otherwise
         it is posted to the manifest where the server process (if running) can detect the manifest
@@ -462,6 +463,10 @@ def _insert_request(self, queue: Queue, catalog: Metadata, action: str):
         instead, raise NotImplementedError in such cases, but we may want the ability to refresh
         the entire component cache from a CLI utility and the current implementation would allow that.
         """
+        # Ensure referenced runtime is available
+        if not PipelineProcessorRegistry.instance().is_valid_runtime_type(catalog.runtime_type.name):
+            return
+
         if self.is_server_process:
             queue.put((catalog, action))
         else:
@@ -575,6 +580,10 @@ def read_component_catalog(self, catalog: ComponentCatalogMetadata) -> Dict[str,
         """
         components: Dict[str, Component] = {}
 
+        # Ensure referenced runtime is available
+        if not PipelineProcessorRegistry.instance().is_valid_runtime_type(catalog.runtime_type.name):
+            return components
+
         # Assign component parser based on the runtime platform type
         parser = ComponentParser.create_instance(platform=catalog.runtime_type)
 

elyra/pipeline/handlers.py

@@ -35,7 +35,7 @@
 from elyra.pipeline.pipeline_constants import PIPELINE_PARAMETERS
 from elyra.pipeline.pipeline_definition import PipelineDefinition
 from elyra.pipeline.processor import PipelineProcessorManager
-from elyra.pipeline.processor import PipelineProcessorRegistry
+from elyra.pipeline.registry import PipelineProcessorRegistry
 from elyra.pipeline.runtime_type import RuntimeProcessorType
 from elyra.pipeline.runtime_type import RuntimeTypeResources
 from elyra.pipeline.validation import PipelineValidationManager

elyra/pipeline/processor.py

@@ -26,10 +26,8 @@
 from typing import Dict
 from typing import List
 from typing import Optional
-from typing import Set
 from typing import Union
 
-import entrypoints
 from minio.error import S3Error
 from traitlets.config import Bool
 from traitlets.config import LoggingConfigurable
@@ -51,6 +49,7 @@
 from elyra.pipeline.properties import KubernetesSecret
 from elyra.pipeline.properties import KubernetesToleration
 from elyra.pipeline.properties import VolumeMount
+from elyra.pipeline.registry import PipelineProcessorRegistry
 from elyra.pipeline.runtime_type import RuntimeProcessorType
 from elyra.pipeline.runtime_type import RuntimeTypeResources
 from elyra.util.archive import create_temp_archive
@@ -60,66 +59,6 @@
 elyra_log_pipeline_info = os.getenv("ELYRA_LOG_PIPELINE_INFO", True)
 
 
-class PipelineProcessorRegistry(SingletonConfigurable):
-    _processors: Dict[str, PipelineProcessor] = {}
-
-    def __init__(self, **kwargs):
-        root_dir: Optional[str] = kwargs.pop("root_dir", None)
-        super().__init__(**kwargs)
-        self.root_dir = get_expanded_path(root_dir)
-        # Register all known processors based on entrypoint configuration
-        for processor in entrypoints.get_group_all("elyra.pipeline.processors"):
-            try:
-                # instantiate an actual instance of the processor
-                processor_instance = processor.load()(root_dir=self.root_dir, parent=kwargs.get("parent"))
-                self.log.info(
-                    f"Registering {processor.name} processor '{processor.module_name}.{processor.object_name}'..."
-                )
-                self.add_processor(processor_instance)
-            except Exception as err:
-                # log and ignore initialization errors
-                self.log.error(
-                    f"Error registering {processor.name} processor "
-                    f'"{processor.module_name}.{processor.object_name}" - {err}'
-                )
-
-    def add_processor(self, processor):
-        self.log.debug(f"Registering {processor.type.value} runtime processor '{processor.name}'")
-        self._processors[processor.name] = processor
-
-    def get_processor(self, processor_name: str):
-        if self.is_valid_processor(processor_name):
-            return self._processors[processor_name]
-        else:
-            raise RuntimeError(f"Could not find pipeline processor '{processor_name}'")
-
-    def get_all_processors(self) -> List[PipelineProcessor]:
-        return list(self._processors.values())
-
-    def is_valid_processor(self, processor_name: str) -> bool:
-        return processor_name in self._processors.keys()
-
-    def is_valid_runtime_type(self, runtime_type_name: str) -> bool:
-        for processor in self._processors.values():
-            if processor.type.name == runtime_type_name.upper():
-                return True
-        return False
-
-    def get_runtime_types_resources(self) -> List[RuntimeTypeResources]:
-        """Returns the set of resource instances for each active runtime type"""
-
-        # Build set of active runtime types, then build list of resources instances
-        runtime_types: Set[RuntimeProcessorType] = set()
-        for name, processor in self._processors.items():
-            runtime_types.add(processor.type)
-
-        resources: List[RuntimeTypeResources] = list()
-        for runtime_type in runtime_types:
-            resources.append(RuntimeTypeResources.get_instance_by_type(runtime_type))
-
-        return resources
-
-
 class PipelineProcessorManager(SingletonConfigurable):
     _registry: PipelineProcessorRegistry