Support dynamic devfile registry

[GeekArthur]

User Story

As a user I want to use devfile from different registries to build and deploy devfile component so that odo should have a mechanism to let user specify the registry for registry flexibility.

Acceptance Criteria

• By default we automatically detect the registry on user's cluster to check if user has the private registry hosed on the cluster
• If user's cluster doesn't host private registry we can let user specify the registry. The registry that user specify can be:
• Public registry hosted by Che or somewhere else
• Private registry hosted by user (on user's cluster or GitHub)

Design documentation: https://docs.google.com/document/d/10a7xm5ynJa7asrY63xxjICmCwCRCUXTdO31bGb57-1E/edit?usp=sharing

Links

• Related issue:
  odo catalog list components command output improvement #2608
  Create component support for odo devfile scenario #2557

/kind user-story
/area devfile

[girishramnani]

we need to consider how we would handle conflicting devfiles ( having same named devfiles ) in different devfile registries.

[GeekArthur]

we need to consider how we would handle conflicting devfiles ( having same named devfiles ) in different devfile registries.

Yes, I think the rough design would be in our metadata we can append devfile registry information to devfile to address this, so should be something like:

<registry>/<devfile>

[girishramnani]

That would resolve the conflict while storing them, but how would be decide which one the user wants when they give a conflicting devfile as input? Would the user provide the whole <registry>/<devfile> Combination or would we select the registry which was added first?

[GeekArthur]

That conflict problem can be resolved by our original draft design here: #2608 (comment), which the current workspace/environment only sticks to one registry via preference/environment configuration (TBD).

[kadel]

Example of how this could work as discussed in odo contributor call

# add new registry to odo
odo preference add registry my-registry https://fffsadfasdfasfs/

# remove registry previously added
odo preference remove registry my-other-registry

odo catalog list components 
# new column in the output - registry name

odo create java
# choose right java devfile based on registry priorities (order?)

odo create java --registry my-registry 
# force use of the devfile registry

[GeekArthur]

NEW DESIGN summary based on today's discussion.

Requirements:

• User can
  • view devfile registry in the devfile registry list
  • add devfile registry to the devfile registry list
  • remove devfile registry from the devfile registry list
• Dynamically support multiple devfile registries together
• Handle name conflict (eg. same devfile component name in different devfile registries) properly

Designs:
Commands need to change:

# Add one more column to display the devfile registry that the devfile component belongs to
odo catalog list components

# View devfile registry in the devfile registry list
odo preference view

# Add devfile registry to the devfile registry list
odo preference add registry <registry name> <URL>

# Remove devfile registry from the devfile registry list
odo preference remove registry <registry name>

# Create devfile component with devfile registry specify
odo create [component type] [component name] --registry [registry name]

Handle name conflict:
Currently we have two default devfile registries (Che devfile registry, private devfile registry) and we can display them to users by default. If we hit name conflict, the user's private devfile registry always has higher priority than public devfile registry by default, but if user really wants to specify the devfile registry user can use --registry flag, it has the highest priority

[dharmit]

Idea of listing devfiles from multiple registries seems a bit counter-intuitive to me as a user. IMO, when user does odo catalog list components, they could be shown the devfiles from the currently enabled registry. But showing them devfiles from multiple registries could confuse the users.

Also, thinking of this from the IDE plugin point of view, I think it would make things simple for the end-user to have a registry configured in odo preferences. A CLI flag could override the registry to be used, but the odo catalog list components should list devfiles from only one registry at a time.

[GeekArthur]

We should consider support the secured devfile registry, we may do some research to check if there are some go packages that support https with self-signed certificate.

[GeekArthur]

@dharmit I bring the multiple registries support topic up in our this week's contributors meeting, from @kadel 's point of view, we should support that since Che support that as well, which means user can enable multiple registries and view/create component from multiple registries. I agree it's a little bit complicated from use case perspective, but it does provide more flexibility for user.

[dharmit]

IMO, the flexibility is not worth it if it makes things complex for the end-user. We can still support a dynamic devfile registry by letting user specify registry of their choice on CLI: odo catalog list components --registry <my-other-registry>. But when a user does odo catalog list components without the --registry flag, they should be provided with list of devfiles from the registry configured in odo preferences.

In any case (with or without --registry flag), IMO, odo catalog list components should list devfiles from only one registry at a time.

[GeekArthur]

@dharmit I rethink this design, actually I don't think display components with their registries via odo catalog list components is complex for user, it's really helpful for user because they know all the devfiles they can use from one output page instead of always check which registry they have then use --registry to display or check which registry they have and enable the registry then use without --registry to display. Also display all devfiles and corresponding registries within one output page can make user clearly know which component belongs to which registry and it's useful for the following operations such as odo create and odo push.

That being said, you can still bring more concerns up in the next contributors meeting, we have multiple architects in that meeting, the current design is agreed by all architects now, but feel free to arise another discussion there

[kadel]

We should consider support the secured devfile registry, we may do some research to check if there are some go packages that support https with self-signed certificate.

How can be self-signed considered secured?

[kadel]

IMO, the flexibility is not worth it if it makes things complex for the end-user. We can still support a dynamic devfile registry by letting user specify registry of their choice on CLI: odo catalog list components --registry <my-other-registry>. But when a user does odo catalog list components without the --registry flag, they should be provided with list of devfiles from the registry configured in odo preferences.

I see your point but I think that it will limit the usefulness of the devfile registries.
Usually, I'm not a big fan of analogies myself but bear with me: I look at the devfile registry as a software repository. Would you be ok if in your Fedora system you could use just one yum repository at a time?

[GeekArthur]

We should consider support the secured devfile registry, we may do some research to check if there are some go packages that support https with self-signed certificate.

How can be self-signed considered secured?

Yeah, sorry for the confusion, that comment is just to let us keep in mind we should support secured registry, self-signed is just a start and we can use that for testing purpose for the timing being, finally we should support and test CA certificate for sure.

[dharmit]

I see your point but I think that it will limit the usefulness of the devfile registries.
Usually, I'm not a big fan of analogies myself but bear with me: I look at the devfile registry as a software repository. Would you be ok if in your Fedora system you could use just one yum repository at a time?

Nope, I'd not like it if my Fedora system had only one registry. But I think the scenario we have resembles more with listing/searching container images from a container registry than listing rpms from a repo.

Going with multiple devfile registries just feels counter intuitive to me. And it's very likely that it's just me. It also asks for the code to consider various scenarios. And not to forget the plugins that would be built on top of odo. To choose nodejs component available from multiple available registries becomes little less straight-forward.

IMO, let's go with what's been discussed already among the stakeholders/teams. We can always change the way we do things based on the user feedback we might receive. 😉

Thanks for patiently answering @GeekArthur & @kadel.

[GeekArthur]

We have discussed about a new command design today for dynamic registry support, so currently we have two designs

Design 1:

# Add devfile registry to the devfile registry list
odo preference add registry <registry name> <URL>

# Update devfile registry in the devfile registry list
odo preference update registry <registry name> <URL>

# Remove devfile registry from the devfile registry list
odo preference remove registry <registry name>

# View devfile registry in the devfile registry list
odo preference view

Design 2:

# Add devfile registry to the devfile registry list, if <registry name> is present then update devfile registry in the devfile registry list
odo preference set registry <registry name> <URL>

# Remove devfile registry from the devfile registry list
odo preference unset registry <registry name>

# View devfile registry in the devfile registry list
odo preference view

IMO, Design 1 is more readable and clear to user but it introduces many new arguments and makes the command more complicated. Design 2 is more centralized and consistent with the existing odo preference but it can cause uncertainty to user so that results in some user experience issues. @kadel How do you think? Which one you prefer?

[GeekArthur]

We discussed the commands design in today's contributor call, here is the latest commands design:

Latest Commands Design: (We prefer this design)
# Add devfile registry to the devfile registry list
odo registry add <registry name> <registry URL>

# Update devfile registry in the devfile registry list
odo registry update <registry name> <registry URL>

# Remove devfile registry from the devfile registry list
odo registry remove <registry name>

# View devfile registry in the devfile registry list
odo registry list

[GeekArthur]

@cdrage FYI, I add the design documentation for dynamic registry support https://docs.google.com/document/d/10a7xm5ynJa7asrY63xxjICmCwCRCUXTdO31bGb57-1E/edit?usp=sharing, you can use that as reference when you update the odo documentation https://odo.dev/ if it's necessary.

[cdrage]

Thank you.

I've merged in: #3045

Once your PR for the code has been merged in, can you please push some documentation to: https://github.com/openshift/odo/blob/master/docs/public/deploying-a-devfile-using-odo.adoc ?

[GeekArthur]

Thank you.

I've merged in: #3045

Once your PR for the code has been merged in, can you please push some documentation to: https://github.com/openshift/odo/blob/master/docs/public/deploying-a-devfile-using-odo.adoc ?

Yup, will do that.