Releases: elyra-ai/elyra
v3.10.0
Quick links
New feature highlights
Pipeline editor: mount shared volumes in custom pipeline nodes
Pipeline nodes that are implemented using custom components or generic components can now be configured to utilize data volume mounts. Take advantage of volume mounts if two or more pipeline nodes need to efficiently share data. Data volume mounts can be configured as pipeline defaults (applying to all generic and custom nodes) or for individual nodes.
Volume mounts are only supported by the Kubeflow Pipelines and Apache Airflow runtimes.
URL and Airflow catalog connectors: support user credentials
Elyra uses catalog connectors to make custom components available to the Visual Pipeline Editor. The catalog connectors for the URL component catalog, the Apache Airflow package catalog, and the Apache Airflow provider package catalog were extended to allow for input of user credentials to support access to secured resources.
What's Changed
New Features
- Add authentication support to Airflow connectors by @ptitzler in #2791
- Add authentication support to URL catalog connector by @ptitzler in #2804
- Customize rendering of component catalog list by @ptitzler in #2813
- Support data exchange between custom & generic components via volumes by @kiersten-stokes in #2799
Bug Fixes
- Fix for Script Editor console does not properly highlight error messages by @VNA818-RPI in #2800
- Update docs to reflect proper generic op output file usage by @akchinSTC in #2798
- Fix invalid repository URLs in 'Elyra in an air gapped environment' topic by @ptitzler in #2810
- Fix Airflow operation processing for number data types by @kiersten-stokes in #2815
- Remove final instance of
BashOperatorfrom tests by @kiersten-stokes in #2817 - Optionally search for operators in airflow.contrib.operators package by @ptitzler in #2819
- Fix version issue in release script by @ptitzler in #2824
Other
- Add errors to tag editor by @marthacryan in #2769
- Add elapsed time information to identify long running pytests by @ptitzler in #2776
- CI: fetch packages info if server tests fail by @tal66 in #2783
- Install jupyterlab during normal builds by @ajbozarth in #2790
- Add GitHub release notes instructions to developer guide by @ptitzler in #2806
- Update developer guide with CI test analysis information by @ptitzler in #2807
- Update prerequisite details in documentation by @ptitzler in #2809
- [DEV-ONLY] Remove dependency on path location in pipeline tests by @kevin-bates in #2811
New Contributors
- @tal66 made their first contribution in #2783
- @VNA818-RPI made their first contribution in #2800
Full Changelog: v3.9.1...v3.10.0
Contributors
Assets 2
v3.9.1
Quick links
What's Changed
Bug Fixes
- Clear all singleton instances during shutdown by @kiersten-stokes in #2781
Other
- Update master branch references to main by @akchinSTC in #2771
- Do not persist the jupyter baseUrl in pipelines by @ajbozarth in #2775
Full Changelog: v3.9.0...v3.9.1
Contributors
Assets 2
v3.9.0
Quick links
New feature highlights
Access sensitive information in generic pipeline nodes
Jupyter notebooks, Python scripts or R scripts might require access to resources that are protected using sensitive information, such as an API key or a user id and password. If you are running pipelines on Kubeflow Pipelines or Apache Airflow you can take advantage of Kubernetes secrets that are defined in your cluster. Starting with version 3.9 you can configure pipelines to expose these secrets as environment variables, which notebooks or scripts can access.
Pipeline CLI: identify pipeline dependencies
The elyra-pipeline describe CLI command output now includes information about the following dependencies for nodes that utilize generic components: container images, data volumes, and Kubernetes secrets. The machine readable output (produced when the --json option is specified) is most commonly used to automate processes, such as impact analysis and dependency checking. In the example below the output of the command is piped to the jq command-line processor, which extracts information about the container images that the pipeline's notebooks or script are executed in:
$ elyra-pipeline describe --json my.pipeline | jq '.dependencies.container_images[]'
"tensorflow/tensorflow:2.8.0"
This information could be used to identify pipelines that use a specific container image version or to verify that the container images are available in a specific container registry.
Create code snippets from notebook cells
Create a code snippet by selecting one or more cells in a Jupyter notebook.
Documentation: Running Elyra in an air gapped environment
The new documentation topic covers considerations for running Elyra in an air gapped environment.
What's Changed
New Features
- Add backend support for Kubernetes secrets environment variables by @kiersten-stokes in #2715
- Elyra pipeline describe enhancements by @binayakdutta in #2722
- Add 'Running Elyra in an air gapped environment' topic to documentation by @ptitzler in #2744
- Update 'elyra-pipeline describe' command by @ptitzler in #2742
- Create code snippet from whole cells (#1199) by @xlegs in #2726
Bug Fixes
- Simplify R and Python icon svgs by @ajbozarth in #2705
- Prepend baseUrl to icon source by @ajbozarth in #2728
- Fix pipeline submit button reload bug by @karlaspuldaro in #2734
- Fix metadata dropdown style in dark mode by @karlaspuldaro in #2743
- Fix incorrect string replacement in release script by @akchinSTC in #2748
- Fix save as code snippet by @marthacryan in #2756
- Fix release script updates to schemas docs and package desc by @akchinSTC in #2752
- Fix tag bug by @marthacryan in #2761
- Fix metadata editor being able to open in multiple tabs by @marthacryan in #2762
- Fix missing placeholder text in metadata editor by @marthacryan in #2764
- Trim whitespace to prevent invalid tags in metadata editor by @marthacryan in #2768
Other
- Add link to new blog post to "published resources" documentation by @kiersten-stokes in #2707
- Update elyra individual extension package descriptions by @akchinSTC in #2708
- Remove dependency on ipython_genutils by @kevin-bates in #2710
- Update changelog header by @ptitzler in #2714
- Metadata tag whitespace proposed solution by @JoshuaAlter in #2538
- Improve github action workflow by @ptitzler in #2720
- Update node packages to fix security vulnerabilities by @akchinSTC in #2575
- Remove tensorflow 1.x sample runtime images by @akchinSTC in #2729
- Develop Elyra against the Jupyterlab source repo by @ajbozarth in #2709
- Validate runtime image environments in CI testing by @kevin-bates in #2721
- Bump NodeJS version requirement to v16 or higher by @akchinSTC in #2732
- Add elyra documentation link in elyra-metadata help by @sgonsal in #2731
- Update generic component operator to support python 3.7 by @akchinSTC in #2727
- Improve pipeline validation by @ajbozarth in #2698
- Fix processing of node properties for pipeline app commands by @kiersten-stokes in #2740
- Update installation topic in the 'getting started' documentation by @ptitzler in #2746
- Update metadata editor to use JupyterLab's new form editor by @marthacryan in #2464
- Fix scroll bug by @marthacryan in #2766
New Contributors
- @JoshuaAlter made their first contribution in #2538
- @sgonsal made their first contribution in #2731
- @binayakdutta made their first contribution in #2722
- @xlegs made their first contribution in #2726
Full Changelog: v3.8.1...v3.9.0
Contributors
Assets 2
v3.8.1
Quick links
What's Changed
Other
- Remove unreachable code by @ptitzler in #2695
- Update to JupyterLab 3.4 by @ajbozarth in #2632
- Fix incorrect regex search in release script by @akchinSTC in #2703
Full Changelog: v3.8.0...v3.8.1
Contributors
Assets 2
v3.8.0
Quick links
New feature highlights
Pipeline editor: define pipeline defaults for runtime images and environment variables
Pipeline nodes that are implemented using generic components (those being used to run Jupyter notebooks, Python scripts, or R scripts) are configured using properties. These properties define, for example, the container image to be used to run as the execution environment. In previous releases it was required to explicitly associate each pipeline node with a container image. Elyra 3.8+ allows for selection of pipeline defaults that are applied to all applicable nodes. These defaults can be optionally overridden for these nodes. Two such defaults are for the runtime image and environment variables, reducing the number of steps required to configure nodes.
Tip: Hover over a node in the canvas to view a summary of its properties.
Pipeline editor: mount shared volumes in generic pipeline nodes
Pipeline nodes that are implemented using generic components can now be configured to utilize data volume mounts. Take advantage of volume mounts if two or more pipeline nodes need to efficiently share data. Data volume mounts be configured as pipeline defaults (applying to all generic nodes) or for individual nodes.
Volume mounts are only supported by the Kubeflow Pipelines and Apache Airflow runtimes.
Pipeline editor: organize generic pipeline inputs and outputs on object storage
Pipeline nodes that are implemented using generic components (those being used to run Jupyter notebooks, Python scripts, or R scripts) utilize object storage buckets as storage for input artifacts, such as Jupyter notebooks or scripts, and output artifacts, like completed notebooks or data files. The new object storage path prefix pipeline property enables you to designate a custom location where those artifacts are stored.
Kubeflow Pipelines with Argo: support emissary workflow executor
Elyra now supports Kubeflow Pipelines installations that are configured to use the emissary executor as Argo workflow executor. There is no need to enable or configure anything in Elyra. Refer to the updated requirements for custom pipeline components for important information.
Kubeflow Pipelines runtime configurations: public API endpoint
Kubeflow Pipeline runtime configurations have been extended to allow for the optional configuration of a public API endpoint. You should configure this endpoint if your Kubeflow Pipelines authentication type is configured as KUBERNETES_SERVICE_ACCOUNT_TOKEN. If configured, Elyra uses this URL (instead of the API URL) to generate links that provide access to the Kubeflow Central Dashboard.
Metadata CLI: import metadata
The elyra-metadata CLI now supports import of runtime configurations, runtime images, code snippets, and component catalogs. Run elyra-metadata import --help to learn more or check out the documentation.
What's Changed
New Features
- Add support for dedicated public KFP endpoint by @ptitzler in #2631
- Support emissary as workflow executor for KFP with Argo by @ptitzler in #2650
- Add support for runtime image and env vars in pipeline default properties by @kiersten-stokes in #2656
- Support use of mounted volumes in Kubeflow Pipelines runtime by @ptitzler in #2675
- Add support for storing pipeline parameters and COS prefixes by @kiersten-stokes in #2669
- Add support for metadata import in the CLI by @sandhya-nayak in #2581
- Support use of mounted volumes in airflow runtime by @akchinSTC in #2680
- Add pipeline properties documentation to pipelines topic by @ptitzler in #2682
- Add 'Elyra JupyterLab interface' topic to user guide by @ptitzler in #2693
Bug Fixes
- [HOTFIX] Fix missing f string typo in operator by @akchinSTC in #2623
- Improve custom container image build instructions by @ptitzler in #2628
- fix gitlab port bug by @ptitzler in #2629
- Abort generic component processing when prereq download fails by @ptitzler in #2630
- Pipeline CLI: Add support for static text spinner by @ptitzler in #2633
- Tolerate editable packages in bootstrapper package install by @kiersten-stokes in #2643
- Improve argument handling in elyra-metadata by @kevin-bates in #2645
- Fix KFP component without inputs compilation bug by @ptitzler in #2646
- Update documentation URLs in schema definitions by @ptitzler in #2658
- Fix broken links to documentation by @karlaspuldaro in #2690
Other
- Update linting to use black formatter by @akchinSTC in #2618
- Uncap Jinja2 and update nbconvert dependencies by @akchinSTC in #2603
- Remove remaining references to airflow examples catalogs by @kiersten-stokes in #2636
- Fix failing tests on master by @kiersten-stokes in #2642
- Fix flaky integration test by @karlaspuldaro in #2639
- Improve documentation topics that reference the CLI by @ptitzler in #2651
- Clean up tests that rely on component cache instances by @kiersten-stokes in #2652
- Replace remaining ComponentCache instance with fixture in processor test by @kiersten-stokes in #2653
- Remove use of legacy resolver in builds and documentation by @romeokienzler in #2654
- Update repository README to include a link to releases page by @ptitzler in #2661
- Publish latest-tagged images when building releases by @kevin-bates in #2660
- Fix install-server target in Makefile by @akchinSTC in #2665
- Remove hacking dependency, capture version snapshot prior to CI testing by @kevin-bates in #2671
- Upgrade to canvas version 12 by @marthacryan in #2641
- Add documentation for pipeline default properties by @kiersten-stokes in #2668
- Fix file dependency bug by @karlaspuldaro in #2637
- Remove secret from airflow helm values by @kevin-bates in #2677
- Update development dockerfiles to use prebuilt extensions by @akchinSTC in #2673
- Suppor...
Contributors
Assets 2
v3.7.0
Quick links
New feature highlights
JupyterLab 3.3 (including settings editor)
Elyra v3.7 is the first release that takes advantage of JupyterLab 3.3. One of the highlights is the settings editor, which enables you to configure JupyterLab (and installed extensions). To customize the behavior of the Visual Pipeline Editor, click the settings link on the canvas or choose 'Settings' > 'Advanced Settings Editor' from the main menu.
Search for elyra to see the complete list of customization options. There's only one in version 3.7, defining what happens when you double click on a pipeline node.
Pipeline editor: view component definitions
The pipeline editor now allows for viewing of custom component definitions. Select a node that utilizes a custom component, open the context menu, and select 'Open Component Definition':
The component's source (YAML for Kubeflow Pipelines components, Python source code for Apache Airflow operators) is rendered using the new generic elyra-code-viewer-extension extension.
Note: editing of custom component definitions is not supported.
Pipeline editor: refresh the component palette on demand
The pipeline editor's palette displays components that are stored in local or remote component catalogs. Since the content of these catalogs can change at any time, Elyra now includes refresh buttons to allow for complete palette reload (1) or partial palette reload (2) in the 'Component Catalogs' panel.
Metadata CLI: export runtimes, runtime images, code snippets, and component catalogs
Elyra stores information about runtimes, runtime images, code snippets, and component catalogs in metadata files. You can export this metadata (e.g. to create a backup) to a local directory using the new export command of the elyra-metadata CLI. To learn more, run elyra-metadata export -h or take a look at this example.
Support for metadata import will be added in a future release. (#2500)
Metadata CLI: install command replaced by create and update
The elyra-metadata CLI supports two new commands, elyra-metadata create (to add a new runtime, code snippet, runtime image, etc) and elyra-metadata update (to update an existing runtime, code snippet, runtime image, etc). These commands replace the elyra-metadata install command.
If you are currently using the legacy command in your automation scripts or custom Dockerfiles, please migrate by replacing elyra-metadata install with elyra-metadata create and elyra-metadata install --replace with elyra-metadata update. The command parameters have not changed. To learn more about the two new commands, run elyra-metadata create --help or elyra-metadata update --help, respectively.
Breaking change in Elyra 4.0: The elyra-metadata install command is no longer supported. (#2580)
Pipeline CLI: export pipelines to runtime-specific format
Elyra uses a proprietary JSON format to store pipeline files, which cannot be used to natively run pipelines on Kubeflow Pipelines or Apache Airflow. Prior to version 3.7 only the pipeline editor supported exporting Elyra pipelines files to Kubeflow Pipelines and Apache Airflow native formats. The new elyra-pipeline export command extends this capability to the CLI. To learn more, run elyra-pipeline export --help or take a look at this example.
Pipeline CLI: monitor the status of Elyra pipelines running on Kubeflow Pipelines
The elyra-pipeline submit command was extended to allow for monitoring of the submitted pipeline on Kubeflow Pipelines. To learn more, run elyra-pipeline submit --help or take a look at this example.
Prebuilt extensions
The Elyra extensions are now published as prebuilt extensions. This eliminates the need to rebuild JupyterLab after running pip install, reducing the installation time. (Installation doc)
Refreshed system-owned runtime image configurations
Elyra utilizes runtime images to run pipeline nodes on container-based platforms, such as Kubeflow Pipelines or Apache Airflow. Every Elyra installation has a small set of runtime images pre-configured, which are system-owned and cannot be removed. Many of those images were outdated and have been replaced with more current versions:
Breaking change in Elyra 4.0: Elyra installations will no longer pre-configure system-owned runtime images. (#2476)
What's Changed
New Features
- Add a link to most recent blog to the documentation by @kiersten-stokes in #2486
- Document how to configure custom Airflow components for use in pipelines by @kiersten-stokes in #2485
- Add support for custom requirements in Elyra container image build by @ptitzler in #2478
- Add support for cli metadata export by @sandhya-nayak in #2448
- Support well-defined ComponentDefinition return values for catalog connectors by @kiersten-stokes in #2492
- Allow pipelines to be opened with the JSON widget by @ajbozarth in #2511
- Use well-defined EntryData return values in Airflow catalog connectors by @ptitzler in #2520
- Add a refresh button the header of Metadata Widget by @ajbozarth in #2531
- Add setting to specify node double click behavior by @marthacryan in #2517
- New extension - Code Viewer by @ajbozarth in #2553
- Add support for elyra-metadata 'create' and 'update' commands by @ptitzler in #2578
- Add troubleshooting information for 'component not found' errors by @ptitzler in #2591
- Add support for 'export' command to elyra-pipeline CLI by @ptitzler in #2582
- Add support for KFP pipeline run monitoring to 'elyra-pipeline submit' command by @ptitzler in #2584
- Enable display of custom component definitions by @kiersten-stokes in #2564
- Create Component Catalogs Widget by @ajbozarth in #2532
Bug Fixes
- Sort runtime types alphabetically, to enforce consistent order is displayed in launcher by @samali0996 in #2449
- Restore tag variable, update release script by @kevin-bates in #2465
- Fix default value rendering in properties template for KFP 'inputvalue' types by @kiersten-stokes in #2473
- Associate user doc to custom component by @yuhuishi-convect in #2474
- Fix templates for safe rendering of comments, pipeline & property descriptions by @kiersten-stokes in #2467
- Replace invalid characters in airflow parent node references by @akchinSTC in #2468
- Fix submit file dialog message by @karlaspuldaro in #2489
- Bump version requirement for nbformat, ipython, and traitlets by @ptitzler in #2493
- Fix invalid KFP component URL references by @ptitzler in #2503
- Sanitize KFP input/output params in kwargs by @thesuperzapper in #2533
- Fix error handling when replace used w/o schemaspace by @kevin-bates in #2540
- Fix broken link in 'comm...
Contributors
Assets 2
v3.6.0
New feature highlights
Duplicate runtime configurations, runtime images, code snippets, and more
Quickly duplicate existing runtime configurations, runtime images, code snippets, and component catalogs with a single click in the JupyterLab GUI:
Add Airflow operators from Airflow built distributions to the pipeline editor
Out of the box the visual pipeline editor for Apache Airflow pipelines only includes operators that allow for the execution of Jupyter notebooks, Python scripts, and R scripts. Starting with this release of Elyra you can add operators from your Apache Airflow built distribution to the palette:
- Create a catalog connector for Apache Airflow.
- Configure the connector to use the download link for the Apache Airflow built distribution that you have installed in your Airflow cluster.
- Open the pipeline editor and start using the imported operators.
Resources:
Add Airflow operators from Airflow provider packages to the pipeline editor
Starting with this release of Elyra you can add operators from provider packages to the palette:
- Create a provider package catalog connector for Apache Airflow.
- Configure the connector to use the download link for the Apache Airflow provider package that you have installed in your Airflow cluster.
- Open the pipeline editor and start using the imported operators.
Resources:
What's Changed
- Bump nanoid from 3.1.22 to 3.2.0 by @dependabot in #2411
- Bump ipython from 7.15.0 to 7.16.3 in /etc/generic by @dependabot in #2408
- Fix typo in RTC documentation topic by @ptitzler in #2412
- Fix git branch URL by @ptitzler in #2421
- Bump node-fetch from 2.6.1 to 2.6.7 by @dependabot in #2434
- Revert removal of dependency caps by @akchinSTC in #2445
- Restore dependency cap on click and pin black by @akchinSTC in #2446
- Validate numeric fields in submit file dialog by @karlaspuldaro in #2383
- Add GUI support for metadata instance duplication by @ptitzler in #2436
- Set 'EnumControl' property values to 'null' in properties JSON by @kiersten-stokes in #2432
- Add link to Apache Airflow custom components tutorial to 'Getting Started' guide by @ptitzler in #2450
- Improve Airflow parser functionality by @kiersten-stokes in #2418
- Add airflow package catalog connector by @ptitzler in #2437
- Add test for code snippet cloning by @ptitzler in #2451
- Move unit test file to proper test directory by @karlaspuldaro in #2452
- Update pipeline editor node pkg to 1.6.0 by @akchinSTC in #2454
- Add Airflow provider package catalog connector by @ptitzler in #2438
- Update built-in connector reference in 'Pipeline Components' documentation by @ptitzler in #2457
Full Changelog: v3.5.0...v3.6.0
Contributors
Assets 2
v3.5.3
New feature highlights
Support GitLab as DAG repository for Apache Airflow
Elyra now supports GitLab as DAG repository for Apache Airflow runtime configuration. This new feature is only enabled if the optional gitlab dependency is installed.
Resources:
Attach pipeline node comments to generated pipelines
Comment nodes provide you with the ability to add embedded documentation to pipelines. Starting with this release these comments are passed through to the target runtime environment.
Kubeflow Pipelines
For Kubeflow Pipelines node comments are attached to the Kubernetes pods as elyra/node-user-doc annotations:
Tip: If your pipeline includes a description, it is rendered in the Kubeflow Pipelines Dashboard when you open the pipeline:
Apache Airflow
For Apache Airflow node comments are attached to the task instance and can be accessed in the task details view:
Tip: If your pipeline includes a description, it is rendered in the Apache Airflow GUI when you open the DAG:
