Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite KFP code generation #2993

Merged
merged 30 commits into from
Nov 11, 2022
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
85ad80c
Rewrite KFP code generation
ptitzler Oct 28, 2022
2857304
Fix issues that surfaced in CLI test
ptitzler Oct 29, 2022
6601fd3
Fix bugs related to generic op inputs and outputs
ptitzler Oct 31, 2022
b0ae294
Add missing tests and fix uncovered bug
ptitzler Oct 31, 2022
8a81c0e
Add code-gen test for custom component pipeline
ptitzler Oct 31, 2022
9c9befc
Add debug to failing server test
ptitzler Nov 1, 2022
23aea4a
Update CLI and related tests
ptitzler Nov 1, 2022
4fff5c5
Fix linting
ptitzler Nov 1, 2022
bc10c23
Fix bugs and tests
ptitzler Nov 1, 2022
91db2e9
Try to fix failing cypress tests
ptitzler Nov 2, 2022
c94db93
Rename cypress test pipeline
ptitzler Nov 2, 2022
7ace98b
Move templates and fix generic op bug
ptitzler Nov 2, 2022
1aad138
Update documentation
ptitzler Nov 2, 2022
19ce9b3
Fix failing server test
ptitzler Nov 2, 2022
40e0f10
Merge branch 'main' of github.com:ptitzler/elyra into rewrite-kfp-cod…
ptitzler Nov 2, 2022
d34b4a8
Introduce new code gen test for generic pipeline components
ptitzler Nov 2, 2022
fee155e
Update code gen tests for generic components
ptitzler Nov 3, 2022
3363155
Finalize first code gen test
ptitzler Nov 3, 2022
baf51e5
Update code gen tests and related assets
ptitzler Nov 4, 2022
7471a22
Fix failing tests, add stubs for WIP tests, and use enum
ptitzler Nov 5, 2022
893ef96
Implement pipeline conf test and fix related issues
ptitzler Nov 7, 2022
b4ded74
Add test for CRIO env and fix related issues
ptitzler Nov 7, 2022
3861821
Tweak implementation
ptitzler Nov 8, 2022
6e0a306
Update docs/source/user_guide/pipelines.md
ptitzler Nov 9, 2022
bd5c6e8
Release module after compilation
ptitzler Nov 9, 2022
1a38194
Add comment to task instance in generated Python DSL
ptitzler Nov 10, 2022
9fa23b4
Update elyra/templates/kubeflow/v1/python_dsl_template.jinja2
ptitzler Nov 10, 2022
b418d6b
Update elyra/pipeline/kfp/processor_kfp.py
ptitzler Nov 10, 2022
c897c4c
Fix linting errors that were caused by accepting a proposed change in…
ptitzler Nov 10, 2022
337e3fc
Fix codeql issue
ptitzler Nov 10, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/source/user_guide/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,12 +372,16 @@ To export a pipeline from the Visual Pipeline Editor:

#### Exporting a pipeline from the command line interface

Use the [`elyra-pipeline`](command-line-interface.html#working-with-pipelines) `export` command to export a pipeline to a runtime-specific format, such as YAML for Kubeflow Pipelines or Python DAG for Apache Airflow.
Use the [`elyra-pipeline`](command-line-interface.html#working-with-pipelines) `export` command to export a pipeline to a runtime-specific format:
- Kubeflow Pipelines: [Python DSL](https://v1-5-branch.kubeflow.org/docs/components/pipelines/sdk/build-pipeline/) or YAML
- Apache Airflow: Python DAG

```bash
$ elyra-pipeline export a-notebook.pipeline --runtime-config kfp_dev_env --output /path/to/exported.yaml --overwrite
```

By default export produces YAML formatted output for Kubeflow Pipelines and Python DAG for Apache Airflow. To choose a different format, specify the `--format` option. Supported values are `py` and `yaml` for Kubeflow Pipelines and `py` for Apache Airflow.
ptitzler marked this conversation as resolved.
Show resolved Hide resolved

To learn more about supported parameters, run
```bash
$ elyra-pipeline export --help
Expand Down
22 changes: 17 additions & 5 deletions elyra/cli/pipeline_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -670,10 +670,16 @@ def describe(json_option, pipeline_path):
"--output",
required=False,
type=Path,
help="Exported file name (including optional path). Defaults to " " the current directory and the pipeline name.",
help="Exported file name (including optional path). Defaults to the current directory and the pipeline name.",
)
@click.option(
"--format",
required=False,
type=str,
help="File export format.",
)
@click.option("--overwrite", is_flag=True, help="Overwrite output file if it already exists.")
def export(pipeline_path, runtime_config, output, overwrite):
def export(pipeline_path, runtime_config, output, format, overwrite):
"""
Export a pipeline to a runtime-specific format
"""
Expand All @@ -699,14 +705,20 @@ def export(pipeline_path, runtime_config, output, overwrite):
param_hint="--runtime-config",
)

# Determine which export format(s) the runtime processor supports
resources = RuntimeTypeResources.get_instance_by_type(RuntimeProcessorType.get_instance_by_name(runtime_type))
supported_export_formats = resources.get_export_extensions()
if len(supported_export_formats) == 0:
raise click.ClickException(f"Runtime type '{runtime_type}' does not support export.")

# If, in the future, a runtime supports multiple export output formats,
# the user can choose one. For now, choose the only option.
selected_export_format = supported_export_formats[0]
# Verify that the user selected a valid format. If none was specified,
# the first from the supported list is selected as default.
selected_export_format = (format or supported_export_formats[0]).lower()
if selected_export_format not in supported_export_formats:
raise click.BadParameter(
f"Valid export formats are {supported_export_formats}.",
param_hint="--format",
)
selected_export_format_suffix = f".{selected_export_format}"

# generate output file name from the user-provided input
Expand Down
374 changes: 0 additions & 374 deletions elyra/kfp/operator.py

This file was deleted.

Loading