Initial changes to support runtime sub package installs

[akchinSTC]

What changes were proposed in this pull request?

pip install elyra['airflow', 'airflow-examples', 'kfp', 'kfp-examples', 'kfp-tekton', 'test','all']

Enables the ability to install selective packages/groups in order to either include or exclude certain runtimes and their dependencies.

How was this pull request tested?

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[lresende]

@akchinSTC Could you please add to the description, the supported syntaxes you have in mind.

[akchinSTC]

@lresende
After further testing and research, this approach will probably not work. There doesn't seem to be any support for "conditional entrypoints" in any packaging tools.
We do have several workarounds to mimic the behavior that we want.

1. Create new python packages specifically for the adding entrypoints and dependencies for [kfp, kfp-tekton, airflow, possibly local(depending on current discussions) etc]. We could then pull these in as dependencies and respective entrypoints in elyra. I would argue that kfp/kfp-tekton should be included in default installation. Creation of these packages could be automated and templated via the release script we currently use for the extension packages. Downside is that this would increase the number of packages we maintain will increase and release pipeline will get longer. e.g. pypi -> conda
2. Leave the entrypoints in place and add conditional imports to check for installed python libraries and display information on how to remedy when the module and associated package is not found.

[lresende]

@lresende After further testing and research, this approach will probably not work. There doesn't seem to be any support for "conditional entrypoints" in any packaging tools. We do have several workarounds to mimic the behavior that we want.

1. Create new python packages specifically for the adding entrypoints and dependencies for [kfp, kfp-tekton, airflow, possibly local(depending on current discussions) etc]. We could then pull these in as dependencies and respective entrypoints in elyra. I would argue that kfp/kfp-tekton should be included in default installation. Creation of these packages could be automated and templated via the release script we currently use for the extension packages. Downside is that this would increase the number of packages we maintain will increase and release pipeline will get longer. e.g. pypi -> conda
2. Leave the entrypoints in place and add conditional imports to check for installed python libraries and display information on how to remedy when the module and associated package is not found.

How would the # 2 work? Basically don't continue processing the entrypoint load if a given dependency is not there? I think that would be a simple way to implement this.

[kevin-bates]

If we take workaround option 1 and split processors into their own packages, I would argue that the only OOTB runtime type should be the local runtime type. This would also satisfy those cases where folks want to simply try things out. I believe including any KFP or Airflow in the base installation is contrary to what #2136 is trying to address since Airflow users don't want/need KFP and vice versa.

If we take workaround option 2 (which seems more attenable and less "expensive" maintenance-wise), then the responsibility is on us to operate with no platform-specific packages and, more importantly, defer the loading of the processor entrypoints until necessary.

One approach that we could use here is to add code to the RuntimesSchemas SchemasProvider that confirms the applicable packages are installed (kinda similar to what already occurs with KFP engines) and, if not present, simply does not include those schemas out of its get_schemas() method. This information could then be used by the processor manager (or whatever loads the entrypoints) and have it only load the processor entrypoint if its corresponding schema is present in the Runtimes schemaspace. This is possible by virtue of the fact that Runtime entrypoint names must match the runtime schema name (.e.g, 'kfp' and 'airflow').

[kevin-bates]

@lresende and my responses crossed, but it looks like we're suggesting similar approaches.

[kevin-bates]

I don't think this will be sufficient. Won't we still need the ability for the user to physically indicate their intentions? Like pip install elyra[kfp] or pip install elyra[airflow].

Do we need a dummy airflow package that we could use (and install via the previous example) to make the user's intention known to server code? This package, while on PyPi and Conda-forge, would likely never have to change, so I'm not sure there would be much of a burden other than its initial creation.

Then elyra[kfp] would include the KFP sdk, elyra[kfp-tekon] would include elyra[kfp] + kfp-tekton and elyra[airflow] would include this dummy airflow placeholder as its dependency. It could even have a dependency on GitHub, if we needed that (and elyra[gitlab] could include elyra[airflow] + gitlab if we wanted that package tied to airflow.

[ptitzler]

I agree. There may be other reasons a package (like github) might be installed. For example, once the community provides a component catalog connector that relies on the package.

[ptitzler]

This PR should address the question raised in #2089

[ptitzler]

Do we need a dummy airflow package that we could use (and install via the previous example) to make the user's intention known to server code? This package, while on PyPi and Conda-forge, would likely never have to change, so I'm not sure there would be much of a burden other than its initial creation.

Instead of a dummy package, would it be possible to create a package for each runtime that serves a purpose, such as identifies the runtime capabilities? This way we would be more consistent across runtimes.

[kevin-bates]

Instead of a dummy package, would it be possible to create a package for each runtime that serves a purpose, such as identifies the runtime capabilities? This way we would be more consistent across runtimes.

That works too. We only need something for airflow, but could maybe have something relative to kfp as well. If so, I think these packages would need to be fairly static so as to prevent the need from having regular releases, etc.

Do you have an idea of how "runtime capabilities" would be expressed across the various runtimes? What are examples of such? And these capabilities would need to be included in "true" BYO runtime implementations as well.

[ptitzler]

Instead of a dummy package, would it be possible to create a package for each runtime that serves a purpose, such as identifies the runtime capabilities? This way we would be more consistent across runtimes.

That works too. We only need something for airflow, but could make have something relative to kfp as well. If so, I think these packages would need to be fairly static so as to prevent the need from having regular releases, etc.

For what I was considering that would be the case. I'm also thinking about a potential future 'local' runtime here, which doesn't necessarily have any third party dependencies.

Do you have an idea of how "runtime capabilities" would be expressed across the various runtimes? What are examples of such? And these capabilities would need to be included in "true" BYO runtime implementations as well.

Focusing on the 'what' might be useful to encode, I was thinking this would include whether a runtime supports certain runtime features such as:

• export pipeline to a runtime native format (e.g. not applicable to 'local')
• generic components
• custom components
• ...

The UI could use those hints to disable or hide things, like this PR proposes to do for VPE launcher tiles.