Support Devfile format to generate a workspace

[slemeur]

Goal

Today we have two kind of recipes to bootstrap a cloud developer workspace and make it portable: Chefile and Factories.

Over the last few month, we've been also iterating on the definition of a devfile format which defines the different tools needed to develop a container based application. That work has been initiated along with the requirements of Workspace.Next. You can find the defined format here: https://github.com/l0rd/devfile/

The goal of this epic is to support the devfile format in Eclipse Che on all the different ways we have to create/generate a workspace.

User Stories

With the following requirements:

• a repository like github.com/che-samples/spring-petclinic
• a devfile hosted under the repository

User Stories:

1. A developer can generate a workspace by just calling a url: che.host/f?repo_url
2. A developer can create a workspace by providing the url to the repository
3. A developer can create a workspace by importing the devfile
4. A developer can generate a workspace by sending a devfile to a rest API

Not in scope for now:

1. A developer can generate the devfile based on an existing workspace
2. A developer can import a repository without a devfile and be guided to select the tools he needs to create the properly tooled workspace
3. A developer can change the configuration of the workspace and changes are reflected + versioned in the devfile
4. A developer should be exposed to the devfile format when he is configuring a workspace
5. A developer can create a workspace based on multi-repo with multi devfiles

Sub-Tasks

A developer can generate a workspace by just calling a url: che.host/f?repo_url
A developer should be able to get a workspace from a repository where there is a devfile. Today a user is able to get a workspace by giving the url of the repository to the factory engine. It should be able to understand if the repository is having a devfile and if it has, use it to create the properly tooled workspace.

A developer can create a workspace by providing the url to the repository
We have different ways to create a new workspace from the dashboard. The user should be able to create a workspace by providing the URL to a repository. Che will be able to create the workspace from a project's devfile hosted in the repository.

• In the workspace creation page, when user provides the URL to a repository with a devfile, the devfile should be used to configure the workspaces.

A developer can create a workspace by importing the devfile
A developer should be able to provide the content of a devfile or import a devfile to create a workspace.

• In the workspace list, we could add an "import" button which will allow to import a devfile either by uploading or copy/pasting the content of the file.

A developer can generate a workspace by sending a devfile to a rest API
A developer should be able to send to a restAPI:

• a devfile
• a url of a repository with a devfile
  An get as a result a workspace created and the url to access it.

Phase 1

• Setup Devfile implementation plan fro Phase 1 Setup Devfile implementation plan fro Phase 1 #11719
• Allow running Che 7 Workspace with development tooling but without recipe Allow running Che 7 Workspace with development tooling but without recipe #11690
• Commands API for Workspace.Next Commands API for Workspace.Next #9546
• Implementation of local part of devfile feature Implementation of local part of devfile feature #11832
• Create a JSON-schema for the Devfile Create a JSON-schema for the Devfile  #11904

Phase 2

• command matches against a tool, but the tool can have multiple containers
• Devfile support in factory Devfile support in factory #12186
• Plan and POC implementation of remote part of devfile feature Plan and POC implementation of remote part of devfile feature #11833
• An ability to specify workspace environment in Devfile An ability to specify Kubernetes/OpenShift List as tool in Devfile #11906
• IDE should respect commands from Runtime IDE should respect commands from Runtime #12053
• Devfile Spec documentation generator Devfile Spec documentation generator #12272
• Add ability to specify workspace environment in Devfile An ability to specify Kubernetes/OpenShift List as tool in Devfile #11906 (doc, PR1)
  • An ability to specify Kubernetes/OpenShift List as tool in Devfile An ability to specify Kubernetes/OpenShift List as tool in Devfile #11906
  • Factory flow should support k8s/OS tool in Devfile Factory flow should support k8s/OS tool in Devfile #12266
  • K8s/OS infrastructure should use PVCs as Che Volumes K8s/OS infrastructure should use PVCs as Che Volumes #12269
  • K8s/OS infrastructure should use Ingresses/Service/Route as Che Server K8s/OS infrastructure should use Ingresses/Service/Route as Che Server #12268
• [Devfile] Support commands for Os/K8s tool [Devfile] Support commands for Os/K8s tool #12479
• Implement a way for the devfile user to specify the root path of the projects Implement a way for the devfile author to specify the root path of the projects. #12531
• Publish Devfile documentation on http://redhat-developer.github.io/devfile Publish Devfile documentation on http://redhat-developer.github.io/devfile #12524
• Support mountSources for non-plugin tools in devfile Support mountSources for non-plugin tools in devfile #12554
• Support custom environment variables for tools in devfile Support custom environment variables for components in devfile #12555
• Support checkoutPath in devfile Support clonePath in devfile #12557
• Support autodetection of checkoutPath of projects Support autodetection of clonePath of projects #12558
• REST API to create workspaces from a devfile REST API to create workspaces from a devfile #12572
• Study/Devfile: proper object hierarchy for tools in the schema and java model Study/Devfile: proper object hierarchy for tools in the schema and java model #12587
• Add support of dockerimage tool in Devfile Add support of dockerimage tool in Devfile #12389
• Add an ability to specify entrypoint for dockerimage tool in Devfile Add an ability to specify entrypoint for dockerimage tool in Devfile #12668
• Implement an ability to specify container for commands for K8s/OS tool with multiple containers Implement an ability to specify container for commands for K8s/OS tool with multiple containers #12606
• Make it possible to specify path for command execution in its configuration Make it possible to specify path for command execution in its configuration #11866
• Support workspace creating from repository if it contains .devfile.yaml/devfile.yaml file Support workspace creating from repository if it contains .devfile.yaml/devfile.yaml file #12675
• Support devfile with more than one Kubernetes tool Support devfile with more than one Kubernetes tool #12719
• Devfile on-boarding and learning experience Devfile on-boarding and learning experience #12732
• Add an ability to specify entrypoint for kube and openshift tool in Devfile Add an ability to specify entrypoint for kube and openshift tool in Devfile #12693
• [Devfile] OpenShift/K8s tool - reference to content [Devfile] OpenShift/K8s tool - reference to content #12733
• Inform users about what devifle was used to create a workspace with the factory | Inform users about what devifle was used to create a workspace with the factory #12741
• [devfile] Che commands (Theia task) targeting a container tool defined by url is not working [devfile] Che commands (Theia task) targeting a container tool defined by url is not working #12765
• Add more flexibility into devfile project source - checkout branch, commit ID etc
  Add more flexibility into devfile project source - checkout branch, commit ID etc #12775
• Support resolving referenced files when building a factory from a link to raw content of the devfile
  Support resolving referenced files when building a factory from a link to raw content of the devfile #12798
• Default devfile name Default devfile name  #12837
• Devfile: tools should be renamed components Devfile: tools should be renamed components  #12836
• Make it possible to add exposed ports for container in Devfile Add an ability to expose K8s/OS tool container port in Workspace Runtime #12808
• Support multiple dockerimage tools in Devfile Support multiple dockerimage tools in Devfile #12875

[slemeur]

Related to #4362

[l0rd]

Some thoughts:

• I think that stories 1. and 4. are related (it's about a new API), and 2. and 3. are related as well (about the client that call these API, the user dashboard). I think that 1. and 4. are prerequisites and should be handled with higher priority. And probably 2. and 3. will be implemented by a different team. Maybe we should split them in 2 different epics?

• What it's expected if a developer modify the devfile from Che IDE? Or if he creates a new devfile? Should the workspace config be changed and the user asked if he wants to reload the workspace? If not what will happen when the workspace is stopped and restarted? What if the URL of the github repo in the devfile (the project source) is changed?

• It looks like there isn't nothing to change with respect to chefile and factories. Devfile will be an alternative to these two features. Is it correct? Maybe we could make it explicit

[skabashnyuk]

Before we start use-case grouping I would like to have an agreement on some technical aspects.

I would say all stories can be divided into such categories.

• we have Devfile content
• we have an HTTP link to Devfile
• Devfile is under vcs. (This use case has the biggest effect on technical solution)

Next, we have two ways to have a working workspace from Devfile.

1. Get content of Devfile, transform it to workspace config 1 to 1 before workspace start.
2. Get content of Devfile during workspace start

From what I see, I think the second solution is better, especially for the most interesting case when Devfile is under vcs.

My proposal is to add new optional entry to workspace configuration.

1. For the case when we have Devfile content

        "devfile": {
          "type": "devfile",
          "content": "Content of devfile"
          "parameters": {}
        }

2. For the case when we have a http link for Devfile

        "devfile": {
          "type": "url",
          "content": "https://github.com/che-samples/console-java-simple/Devfile"
          "parameters": {}
        }

3. For the case when Devfile is under vcs

        "devfile": {
          "type": "git",
          "content": "https://github.com/che-samples/console-java-simple.git"
          "parameters": {}
        }

        "devfile": {
          "type": "hg",
          "content": "http://selenic.com/hg"
          "parameters": {}
        }

        "devfile": {
          "type": "svn",
          "content": "svn://xxx.xxx.xxx.xxx/Repository/"
          "parameters": {}
        }

Then on workspace start, we will have some DevFile Broker(dependent on type) that will play alongside with our ChePlugin broker to get effective workspace configuration.
The questions here:

• who, when and how many times are going to clone sources.
• how to generate Devfile from workspace with Devfile under vcs.

[l0rd]

Get content of Devfile during workspace start

devfile=description of your workspace. How are you going to start the workspace if you haven't read the devfile yet?

DevFile Broker(dependent on type) that will play alongside with our ChePlugin broker

Ok but keep in mind that plugins list is defined in the devfile too

who, when and how many times are going to clone sources.

When a git repo is specified we should clone first (who: the devfile broker, where: in the /projects volume, how many times: just one I hope :-) ) and start the workspace (editor + plugins) as a second step.

how to generate Devfile from workspace with Devfile under vcs

I am not sure I get your question. The scenario for generating the devfile is out of scope right?

[sleshchenko]

@skabashnyuk @slemeur @l0rd I believe it's a good time to revise this epic and close it, maybe create another one for some improvements since every issue linked here is fixed or closed by che-bot because of inactivity for 180 days.