Create component support for odo devfile scenario

[elsony]

[kind/Feature]

Which functionality do you think we should add?

Create component support for odo devfile scenario:

• Provide ability to filter devfiles that can be supported by odo devfile support in a given Che devfile repository
• When the experimental flag is enabled (see EXPERIMENTAL feature flag  #2536), "odo catalog list components" command will also include the list of components (on top of the traditional odo components) that are provided by the Che devfile repository from above
• User can call the "odo component create" to create the devfile style of the component, it will result in a copy of the devfile (from the Che devfile repo) for that particular component
• With the existence of the devfile on the project, all further odo operations, e.g. odo push, will be using the devfile support

Why is this needed?

The IDP definition of the devfile style applications are defined in a Che devfile repository. The Che devfile repository will contain a mix of devfile for regular Che usage as well as the ones that can provide IDP support for odo. We'll need a way to filter the devfile repo to build the list of components that odo supports. Combined with the odo create component support on the devfile style components, the user will be able to build and run their application via the devfile style component support.

[elsony]

/kind feature

[GeekArthur]

/assign @GeekArthur

[GeekArthur]

@elsony I have two questions need to confirm:

1. Is this the Che devfile repository/registry location: https://github.com/eclipse/che-devfile-registry/tree/master/devfiles?
2. Should this issue also include pushing odo supported devfile (I think the odo supported devfile should have more customized fields and they are not fully decided yet) to Che devfile repository/registry?

[kadel]

2. Should this issue also include pushing odo supported devfile (I think the odo supported devfile should have more customized fields and they are not fully decided yet) to Che devfile repository/registry?

There should not be any special odo supported devfile. The goal is to have any Che supported devfile to be also odo supported devfile.

1. Is this the Che devfile repository/registry location: https://github.com/eclipse/che-devfile-registry/tree/master/devfiles?

That is the source for devfile registry. This devfile registry is supposed to run inside the cluster.
For example, for public che.redhat.com the registry is at https://che-devfile-registry.openshift.io//devfiles/index.json

[elsony]

There should not be any special odo supported devfile. The goal is to have any Che supported devfile to be also odo supported devfile.

The devfile that can be used by odo will need to have build and runtime information associate with it. Although the devfile that odo supports shares the same schema and format as the regular Che dev file, in order for a given Che devfile to be useful for odo, it needs to have some minimal requirement, e.g. certain commands defined for build. For example, if some devfiles on the repo only defines the Theia editor and some Che plugins, it will not have enough information to tell odo on how to build the application and do iterative dev development. That's why we still need some filtering mechanism to find out all the devfile that fulfill those extra odo extra requirement in order for us to show the list of devfile style of odo components properly.

[GeekArthur]

@elsony Given that, we still need to push some odo supported devfiles (with extra requirement) to che registry, right? And pushing odo supported devfiles to che registry also belongs to this issue?

[gorkem]

I would rather avoid devfiles for odo and devfiles for Che separation. Ultimately "stacks" should support both. I think eventually we can replace/update the existing devfiles on the registry.

[kadel]

There should not be any special odo supported devfile. The goal is to have any Che supported devfile to be also odo supported devfile.

The devfile that can be used by odo will need to have build and runtime information associate with it. Although the devfile that odo supports shares the same schema and format as the regular Che dev file, in order for a given Che devfile to be useful for odo, it needs to have some minimal requirement, e.g. certain commands defined for build. For example, if some devfiles on the repo only defines the Theia editor and some Che plugins, it will not have enough information to tell odo on how to build the application and do iterative dev development. That's why we still need some filtering mechanism to find out all the devfile that fulfill those extra odo extra requirement in order for us to show the list of devfile style of odo components properly.

Maybe the wrong wording on my side. Yes in your example of a Devfile that only defined Theia editor and plugins it won't make sense in odo context. But I would consider that an edge case.
Component type of cheEditor chePlugin in devfile won't make sense for odo as they are Che specific. But I would like to avoid adding another type that is odo specific. This would lead to a situation where you have Che and odo specific devfile that can't be reused in the other tools.

Let me put it in another way. Any devfile that works with odo should work with Che.

@GeekArthur

@elsony Given that, we still need to push some odo supported devfiles (with extra requirement) to che registry, right? And pushing odo supported devfiles to che registry also belongs to this issue?

What do you mean by extra requirements? If Devfile has at least one dockerimage component defined it should be supported by odo.
I think that the current version of che-devfile-registry is just read-only (you push to it by updating its source https://github.com/eclipse/che-devfile-registry)

Every devfile that is there right now https://github.com/eclipse/che-devfile-registry/tree/master/devfiles should at the end work with odo.

[GeekArthur]

@kadel I agree with you, there is no special for the devfile that support by odo. I didn't mean add new odo specific type, I mean the project devfile that includes specific commands for Codewind projects since Codewind will consume odo in the future.

The current status from today's contribution call:

• Pick some standard commands to decide if the component will show up on the odo catalog list
• Pick build and run command as the standard command. Need to check how many existing will match to decide the default list that will show on odo catalog
• Add a parameter to allow user to specific what command will be executed on a devfile

Finally we may need to push/update devfiles in Che devfile registry for specific commands support but not cover by this issue.

[GeekArthur]

If the devfile satisfies the following three conditions then it's supported devfile

• Devfile has dockerimage component type
• Devfile has build command
• Devfile has run command

[girishramnani]

Consideration of having a flag which allows users to specify commands for devfiles which do-not have build or run command. so we can support any devfile.

[GeekArthur]

@girishramnani We discussed the topic for supporting user specify commands by flag before, but from the previous discussion, IIRC we want to apply the flag to odo push instead of odo create. From my understanding, the reason is that the functionality of odo create is to download devfile to .odo folder which helps user create the devfile as configuration file, user doesn't know which command to apply before having the devfile locally.

[elsony]

If the devfile satisfies the following three conditions then it's supported devfile

* Devfile has **dockerimage** component type

* Devfile has **build** command

* Devfile has **run** command

If we use those 3 criteria, out of the existing 26 Che devfile in the existing devfile repo, 12 of them meet those criteria:
Odo Supported Devfile Components:
NAME PROJECT TAGS
apache-camel-springboot N/A N/A
cpp N/A N/A
dotnet N/A N/A
asp.net N/A N/A
java-gradle N/A N/A
java-maven N/A N/A
java-mongo N/A N/A
java-mysql N/A N/A
java-web-spring N/A N/A
java-web-vertx N/A N/A
nodejs-web-app N/A N/A
rust N/A N/A

To limit the testing that we need to do for MVP for tech preview, we are going to pick a unique key instead of the more common build and run keys for the filtering.

The revised criteria that we'll be using are:

If the devfile satisfies the following three conditions then it's supported devfile
* Devfile has dockerimage component type
* Devfile has devbuild command
* Devfile has devrun command

[girishramnani]

@GeekArthur would that also store these values in local config so that on the next push user doesn’t have to map the commands again?

[GeekArthur]

@girishramnani Currently it won't but I think store the latest user specify commands in the local config is a good suggestion.

[GeekArthur]

Current status of odo catalog list components command with devfile support:

• The command output includes both supported & unsupported OpenShift components and supported (Create component support for odo devfile scenario #2557 (comment)) & unsupported devfile components
• The devfile registry we use is Che devfile registry (https://github.com/eclipse/che-devfile-registry/tree/master/devfiles), we discuss this to use personal/temp repo to store the devfiles for tech preview for limiting test

Jingfus-MacBook-Pro:odo jingfuwang$ ./odo catalog list components
Odo Supported OpenShift Components:
NAME                       PROJECT       TAGS
nodejs                     openshift     10,8,8-RHOAR,latest
openjdk18                  openshift     latest
codewind-odo-openjdk18     che           latest

Odo Unsupported OpenShift Components:
NAME        PROJECT       TAGS
dotnet      openshift     2.0,latest
httpd       openshift     2.4,latest
nginx       openshift     1.10,1.12,1.8,latest
nodejs      openshift     6
perl        openshift     5.24,5.26,latest
php         openshift     7.0,7.1,latest
python      openshift     2.7,3.5,3.6,latest
ruby        openshift     2.3,2.4,2.5,latest
wildfly     openshift     10.0,10.1,11.0,12.0,13.0,8.1,9.0,latest

Odo Supported Devfile Components:
NAME                        PROJECT     TAGS
apache-camel-springboot     N/A         N/A
cpp                         N/A         N/A
dotnet                      N/A         N/A
asp.net                     N/A         N/A
java-gradle                 N/A         N/A
java-maven                  N/A         N/A
java-mongo                  N/A         N/A
java-mysql                  N/A         N/A
java-web-spring             N/A         N/A
java-web-vertx              N/A         N/A
nodejs-web-app              N/A         N/A
rust                        N/A         N/A

Odo Unsupported Devfile Components:
NAME               PROJECT     TAGS
angular            N/A         N/A
apache-camel-k     N/A         N/A
che4z              N/A         N/A
golang             N/A         N/A
nodejs             N/A         N/A
nodejs-mongo       N/A         N/A
react              N/A         N/A
php-laravel        N/A         N/A
php-mysql          N/A         N/A
php-symfony        N/A         N/A
php-web-simple     N/A         N/A
python             N/A         N/A
python-django      N/A         N/A
quarkus            N/A         N/A

New design of unique key for parsing is here: #2557 (comment)

[GeekArthur]

odo create has a corner case where OpenShift component and devfile component share the same component name. For example, Che devfile registry has nodejs component, OpenShift component has nodejs component as well, then the conflict issue will be raised if user runs odo create nodejs.

To resolve the conflict issue, I can think of two options:

1. OpenShift component == devfile component (priorities of them are identical)
   Provide error message for user to let user add flag (eg. --devfile or --S2I) to specify which component user wants to create
2. devfile component > OpenShift component (devfile component has higher priority)
   Create devfile component by default, if user wants to create OpenShift component, user has to add --S2I as flag. This option is consistent with the design of odo push here: Remove devfile push cmd #2598 (comment) where if both devfile.yaml and config.yaml present, odo push uses devfile.yaml

I am not sure which odo folk designs odo create, so just @ all odo folks in this issue. Please let us know which option odo team prefers? @kadel @girishramnani @gorkem

[kadel]

Current status of odo catalog list components command with devfile support:

Can you please open PR with the changes that you implemented? We would like to move quickly with small PRs rather than creating some big PRs with a lot of changes. We can hide partially implemented features with ODO_EXPERIMENTAL flag

• The command output includes both supported & unsupported OpenShift components and supported (#2557 (comment)) & unsupported devfile components

This is great I like that both are displayed with a clear distinction of what will be working without problems and what might need additional changes or flags.

• The devfile registry we use is Che devfile registry (https://github.com/eclipse/che-devfile-registry/tree/master/devfiles), we discuss this to use personal/temp repo to store the devfiles for tech preview for limiting test

You are reading it from the repository? Isn't that just the source code for the registry service? Shouldn't we use https://che-devfile-registry.openshift.io/ instead?

odo create has a corner case where OpenShift component and devfile component share the same component name. For example, Che devfile registry has nodejs component, OpenShift component has nodejs component as well, then the conflict issue will be raised if user runs odo create nodejs.

To resolve the conflict issue, I can think of two options:

1. OpenShift component == devfile component (priorities of them are identical)
   Provide error message for user to let user add flag (eg. --devfile or --S2I) to specify which component user wants to create
2. devfile component > OpenShift component (devfile component has higher priority)
   Create devfile component by default, if user wants to create OpenShift component, user has to add --S2I as flag. This option is consistent with the design of odo push here: #2598 (comment) where if both devfile.yaml and config.yaml present, odo push uses devfile.yaml

Ultimately, I would like to get to 2. and make Devfile the default.
But, we can't do it just yet, because we would break existing IDE plugins for VSCode and IntelliJ that are using odo.
But we could use ODO_EXPERIMENTAL flag to modify the default behavior.

$ odo preference set Experimental true
$ odo create nodejs
// create devfile component (./devfile.yaml)
$ odo create nodejs --provisioner s2i
// create  the old style component definition (.odo/config.yaml)

$ odo preference set Experimental false
$ odo create nodejs
// create  the old style component definition (.odo/config.yaml)

Once we have feature parity with the old-style components we would just remove the Experimental flag and make the behavior that it was enabling default one.

[GeekArthur]

@kadel Thanks for your reply!

Can you please open PR with the changes that you implemented? We would like to move quickly with small PRs rather than creating some big PRs with a lot of changes. We can hide partially implemented features with ODO_EXPERIMENTAL flag

Sure, I will do some code cleanup and put up the PR, actually my current changes already included hiding the implementation with ODO_EXPERIMENTAL flag, so the PR should be ready soon

You are reading it from the repository? Isn't that just the source code for the registry service? Shouldn't we use https://che-devfile-registry.openshift.io/ instead?

Yes, I am using https://che-devfile-registry.openshift.io/, sorry for the confusion

Ultimately, I would like to get to 2. and make Devfile the default.
But, we can't do it just yet, because we would break existing IDE plugins for VSCode and IntelliJ that are using odo.
But we could use ODO_EXPERIMENTAL flag to modify the default behavior.

That makes sense, thanks for your suggestions

[GeekArthur]

Given odo create is just download devfile instead of create devile + we want to drop config.yaml file, we should have a way to handle <component name> argument in odo create properly if user wants to pass in.

Discussed this today, we decide to store the <componet name> information in the .odo/.env/<env file> to support specify <component name> via odo create

[GeekArthur]

Another topic we should talk about is regarding our customized devfile, I believe the original issue is from @elsony 's comment here: #2557 (comment). @elsony 's point is that If we use Che devfile registry with conditions #2557 (comment) that we check, we will have 12 supported devfile components, that's too many for us to do the tech preview testing (I think the original design is focusing on open liberty, spring and nodejs for tech preview) given we want to fully support all supported devfile components. So we may have two ways to resolve the issue:

1. Push our own devfiles to Che devfile registry and add special keyword as @elsony mentions Create component support for odo devfile scenario #2557 (comment) to reduce the number of supported devfile components
2. Create our own devfile registry and add special keyword as @elsony mentions Create component support for odo devfile scenario #2557 (comment) to reduce the number of supported devfile components

We can discuss this on our next odo contribution meeting.