Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix KFP component without inputs compilation bug #2646

Merged
merged 3 commits into from
Apr 8, 2022

Conversation

ptitzler
Copy link
Member

@ptitzler ptitzler commented Apr 6, 2022

Closes #2644

What changes were proposed in this pull request?

  • Fix bug
  • Add test case to elyra/tests/pipeline/kfp/test_processor_kfp.py (including resource)

How was this pull request tested?

  • Manual testing (add this component to a KFP pipeline and run it)
  • pytest -v elyra/tests/pipeline/kfp/test_processor_kfp.py

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

@ptitzler ptitzler added the kind:bug Something isn't working label Apr 6, 2022
@ptitzler ptitzler added this to the 3.8.0 milestone Apr 6, 2022
@elyra-bot
Copy link

elyra-bot bot commented Apr 6, 2022

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link: Binder

@ptitzler ptitzler added the component:pipeline-runtime issues related to pipeline runtimes e.g. kubeflow pipelines label Apr 6, 2022
@kiersten-stokes kiersten-stokes self-requested a review April 6, 2022 22:46
@ptitzler
Copy link
Member Author

ptitzler commented Apr 6, 2022

The failing test test_directory_based_component_catalog doesn't take into account that unrelated tests might add other resources (like it is done in this PR) and needs to be reworked to assure its original assumptions remain correct over time.

Copy link
Member

@kiersten-stokes kiersten-stokes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The failing test test_directory_based_component_catalog doesn't take into account that unrelated tests might add other resources (like it is done in this PR) and needs to be reworked to assure its original assumptions remain correct over time.

Each time a test uses the component_cache fixture (which the directory-based catalog test does), the component cache instance should be cleared both before and after the test completes (and hence not side-effect other tests). Obviously, that isn't happening here, so I'll keep looking into a lightweight solution.

elyra/tests/pipeline/resources/components/no_inputs.yaml Outdated Show resolved Hide resolved
@kiersten-stokes
Copy link
Member

Thoughts on going back to a 'greater than' comparison for the test_directory_based_component_catalog test? We had it this way as of a few commits ago, and it's probably the simplest solution. Since we go on to check for the presence of certain components, I don't think there would be much of a downside to this.

@ptitzler
Copy link
Member Author

ptitzler commented Apr 7, 2022

Updates on the CI test failures:

When the entire test is run, the two cases fail because the component cache is not empty at the start

$ pytest -v elyra/tests/pipeline/kfp/test_component_parser_kfp.py 

When only the two test cases are run, they succeed because its setup is not tainted:

$ pytest -v elyra/tests/pipeline/kfp/test_component_parser_kfp.py -k "test_directory_based_component_catalog"

@ptitzler
Copy link
Member Author

ptitzler commented Apr 7, 2022

Thoughts on going back to a 'greater than' comparison for the test_directory_based_component_catalog test? We had it this way as of a few commits ago, and it's probably the simplest solution. Since we go on to check for the presence of certain components, I don't think there would be much of a downside to this.

It'd be possible. The test would have to take an inventory of the existing components in the cache and eliminate duplicates from its catalog directory list for this to work. However, there's also a good chance that the cache already includes all of the pre-selected components, which would render the test result almost useless because it would only validate that adding components from an empty directory doesn't add any components.

@kiersten-stokes
Copy link
Member

It'd be possible. The test would have to take an inventory of the existing components in the cache and eliminate duplicates from its catalog directory list for this to work. However, there's also a good chance that the cache already includes all of the pre-selected components, which would render the test result almost useless because it would only validate that adding components from an empty directory doesn't add any components.

Right. And as of the last commit and my latest testing regarding cache cleanup in certain tests, it seems that we have a larger issue on our hands here.

Copy link
Member

@kevin-bates kevin-bates left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ptitzler ptitzler added the platform: pipeline-Kubeflow Related to usage of Kubeflow Pipelines as pipeline runtime label Apr 8, 2022
Copy link
Member

@kiersten-stokes kiersten-stokes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@akchinSTC akchinSTC merged commit d88a66f into elyra-ai:master Apr 8, 2022
@ptitzler ptitzler deleted the no-inputs branch April 8, 2022 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:pipeline-runtime issues related to pipeline runtimes e.g. kubeflow pipelines kind:bug Something isn't working platform: pipeline-Kubeflow Related to usage of Kubeflow Pipelines as pipeline runtime
Projects
None yet
Development

Successfully merging this pull request may close these issues.

KFP Pipeline processing fails if custom component does not define inputs
4 participants