Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move CLI component cache build to occur before pre-processing #2913

Merged
merged 2 commits into from
Aug 31, 2022

Conversation

kiersten-stokes
Copy link
Member

@kiersten-stokes kiersten-stokes commented Aug 30, 2022

Fixes #2911

What changes were proposed in this pull request?

This PR moves the build of the component cache for relevant CLI tasks (validate, export and submit) to occur before pipeline pre-processing. Pipeline CLI tests have also been updated to catch this case in the future. Long-winded explanation below.

Recall that ComponentCache instances are either considered server processes (usually a running Elyra instance) or not (mostly CLI processes), and only in the former case is a cache built. For the pipeline CLI tasks of validate, export and submit, we pass emulate_server_app=True to the creation/retrieval of the cache instance so that it builds and can reference the cache during pipeline validation.

The issue referenced above was caused by the fact that, as of #2799, pipeline pre-processing (e.g. the creation of a PipelineDefinition object) also creates/retrieves a ComponentCache instance (and does not specify anything in regards to emulate_server_app). Because pre-processing previously occurred before _build_component_cache in the CLI app, the ComponentCache instance was created during pre-process and did not appropriately set emulate_server_app to True. This PR moves the build of the cache to occur before pre-processing, such that creation of the ComponentCache instance is done "as a server process".

Another issue is that this problem was not caught by the tests because the component_cache fixture explicitly sets emulate_server_app to True during ComponentCache instance creation. To more accurately reflect how things flow during use of the CLI, I've created a new fixture that does not result in the creation of a ComponentCache instance.

How was this pull request tested?

Tested manually. Also ran the updated backend tests against current main to ensure that the expected tests fail on main, but pass on this branch.

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

@elyra-bot
Copy link

elyra-bot bot commented Aug 30, 2022

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link: Binder

@akchinSTC akchinSTC self-requested a review August 30, 2022 21:20
@ptitzler ptitzler self-requested a review August 31, 2022 05:16
@ptitzler ptitzler added kind:bug Something isn't working component:cli-tools Command line tools labels Aug 31, 2022
Copy link
Member

@ptitzler ptitzler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you!

@akchinSTC akchinSTC merged commit 50cc415 into elyra-ai:main Aug 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:cli-tools Command line tools kind:bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CLI: Pipeline validation fails for custom components
3 participants