Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for 'export' command to elyra-pipeline CLI #2582

Merged
merged 7 commits into from
Mar 25, 2022

Conversation

ptitzler
Copy link
Member

@ptitzler ptitzler commented Mar 22, 2022

Adds support for the elyra-pipeline export command, which allows for export of a pipeline to a runtime-native format (KFP and AA). The command mimics the functionality that the VPE already provides).

$ elyra-pipeline --help
Usage: elyra-pipeline [OPTIONS] COMMAND [ARGS]...

  Run Elyra pipelines in your local environment or submit them to an
  external service, such as Kubeflow Pipelines or Apache Airflow.

  Find more information at: https://elyra.readthedocs.io/en/latest/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  describe  Display pipeline summary
  export    Export a pipeline to a runtime-specific format
  run       Run a pipeline in your local environment
  submit    Submit a pipeline to be executed on the server
  validate  Validate pipeline
$ elyra-pipeline export --help
Usage: elyra-pipeline export [OPTIONS] PIPELINE_PATH

  Export a pipeline to a runtime-specific format

Options:
  --runtime-config TEXT  Runtime configuration name.  [required]
  --output PATH          Exported file name (including optional path).
                         Defaults to  the current directory and the pipeline
                         name.

  --overwrite            Overwrite output file if it already exists.
  --help                 Show this message and exit.

output:

  • if not specified, the output file is named after the pipeline file, with the appropriate extension added. and stored in the current directory. Example: elyra-pipeline export analysis.pipeline ... --> ./analysis.yaml (KFP) ./analysis.py (AA)
  • if the specified value ends in yaml or .yml (KFP) .py (AA) it is used as output name, as is:
    • directories are created, if necessary
    • elyra-pipeline export analysis.pipeline --output /path/to/file.yaml ... --> output: /path/to/file.yaml
    • elyra-pipeline export analysis.pipeline --output /path/to/file.py ... --> output: /path/to/file.py
  • if the specified value does not end in yaml or .yml (KFP) .py (AA), it is interpreted as a directory name and the exported file named after the pipeline file, with the appropriate extension added
    • elyra-pipeline export analysis.pipeline --output /path/to/ ... --> output: /path/to/analysis.yaml

overwrite:

  • if specified, the output file is overwritten if it already exists

Documentation updates:

  • new export subsection
  • (unrelated) moved the describe content to its own subsection

image

Closes #2571

What changes were proposed in this pull request?

How was this pull request tested?

  • Manual testing using elyra-pipeline CLI
  • Added unit test cases to elyra/tests/cli/test_pipeline_app.py
  • Reviewed the output of make docs

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

@ptitzler ptitzler added kind:enhancement New feature or request status:Work in Progress Development in progress. A PR tagged with this label is not review ready unless stated otherwise. component:cli-tools Command line tools labels Mar 22, 2022
@ptitzler ptitzler added this to the 3.7.0 milestone Mar 22, 2022
@elyra-bot
Copy link

elyra-bot bot commented Mar 22, 2022

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link: Binder

@ptitzler
Copy link
Member Author

ptitzler commented Mar 22, 2022

Remaining todo's:

  • Documentation updates
  • Improve elyra/cli/pipeline_app.py

@ptitzler ptitzler removed the status:Work in Progress Development in progress. A PR tagged with this label is not review ready unless stated otherwise. label Mar 22, 2022
@akchinSTC akchinSTC self-requested a review March 22, 2022 21:39
@kevin-bates kevin-bates self-requested a review March 22, 2022 21:42
Copy link
Member

@kevin-bates kevin-bates left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes look good @ptitzler and there's some nice clean-up. I just had a couple of comments and have not taken it for a spin yet either.

elyra/cli/pipeline_app.py Show resolved Hide resolved
elyra/cli/pipeline_app.py Outdated Show resolved Hide resolved
elyra/cli/pipeline_app.py Outdated Show resolved Hide resolved
docs/source/user_guide/pipelines.md Show resolved Hide resolved
elyra/cli/pipeline_app.py Outdated Show resolved Hide resolved
elyra/cli/pipeline_app.py Show resolved Hide resolved
elyra/cli/pipeline_app.py Outdated Show resolved Hide resolved
@akchinSTC akchinSTC merged commit 0e623de into elyra-ai:master Mar 25, 2022
@ptitzler ptitzler deleted the cli-pipeline-export branch March 28, 2022 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:cli-tools Command line tools kind:enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Export pipeline (YAML) via CLI command
3 participants