Automatically generate meta.yaml given a vsx repo+branch+dockerfile

[l0rd]

Is your enhancement related to a problem

Adding a new vsx extension in the che-plugin-registry involves the manual and error prone creation of a meta.yaml. Most of the meta.yaml fields can be retrieved from a vscode extension package.json.

Describe the solution you'd like

Adding a new vs code extension in the plugin registry should be as simple as adding a new entry in a vscode-extensions.json file at the root of the GitHub repository:

{
   "repository": "https://bitbucket.org/atlassianlabs/atlascode",
   "checkout": "2.6.1"
}

This format is similar to open-vsx/publish-extensions extensions.json.

We should progressively replace the meta.yaml with corresponding entries in a vscode-extensions.jsonin the GitHub repository. The vscode-extensions.json to meta.yaml conversion should be part of the build of the che-plugin-registry.

This is only for vs-code extensions. Other plugins as theia or machine-exec will still be specified as meta.yaml files.

Included in epic #15819

Subtasks

1. Update the build script of the plugin registry to automatically generate the folders and meta.yaml for plugins given an entry in a vscode-extensions.json like:

   {
      "repository": "https://bitbucket.org/atlassianlabs/atlascode",
      "checkout": "2.6.1"
   }

   The meta.yaml informations should be automatically generated (at build time) reading the package.json of the extension. vsx should be automatically built and published on https://download.jboss.org/jbosstools/.
2. Add an "extensions JSON" PR check that verifies vscode-extensions.json format (alphabetic order of entries, mandatory "repository" and "checkout" fields, git repo and tag exist)
3. Update the plugin-checker to include vscode-extensions.json as a source to check for updates: if a newer tag exist create a PR to update the extension "checkout".
4. Update the vsx PR check Add plugin-registry PR check to validate new/updated vsx #17027 to include verification of new/updated entries in vscode-extensions.json.
5. Update the build script and PR check to support for vscode extensions running in a sidecar:

   {
      "repository": "https://github.com/redhat-developer/vscode-java",
      "checkout": "v0.57.0",
      "dockerfile": "./dockerfiles/redhat-developer.vscode-java/Dockerfile",
      "resources": {
         "memoryRequest": "10Mi",
         "memoryLimit": "128Mi",
         "cpuRequest": "50m",
         "cpuLimit": "250m"
      }
   }

   Endpoint and container sections of the meta.yaml should be automatically generated from the Dockerfile (we should extract EXPOSE <port> and VOLUME <path> instructions). In that case the meta.yaml container env, command and args don't need to be generated because those can be specified in the Dockerfile.
6. Gradually replace existing meta.yaml in che-plugin-registry gh repo with corresponding entries in vscode-extensions.json

[ericwill]

A few comments/questions:

1. How are we going to handle volumes in the meta.yaml file exactly? In the case of the Quarkus and Java plugins they are specifying /home/theia/.m2 as a mount path, but this is not reflected in the che-sidecar-java Dockerfile.

2. How do we handle dependent plugins? For example, Quarkus requires the jdt-ls, java-debug, and vscode-java-debug, but these are neither in the registry nor are they all in the same repositories. Maybe we could include a deps field in the .json file?

3. Can we get somewhere upstream (either Eclipse.org or somewhere else) to host the built vsix files + source tarballs? Currently we are using download.jboss.org which requires Red Hat VPN access, it would be great if we could host these vsix files somewhere upstream, as then we can do the entire build + publish step outside of the VPN.

[l0rd]

A few comments/questions:

1. How are we going to handle volumes in the meta.yaml file exactly? In the case of the Quarkus and Java plugins they are specifying /home/theia/.m2 as a mount path, but this is not reflected in the che-sidecar-java Dockerfile.

What about adding VOLUME /home/theia/.m2 instruction in the Dockerfile?

2. How do we handle dependent plugins? For example, Quarkus requires the jdt-ls, java-debug, and vscode-java-debug, but these are neither in the registry nor are they all in the same repositories. Maybe we could include a deps field in the .json file?

Dependencies are in package.json. I would not duplicate that field. But to be able to identify the dependencies in our registry we need to add the id field AND the vsx PR check must verify that dependencies exist in the registry. What do you think?

3. Can we get somewhere upstream (either Eclipse.org or somewhere else) to host the built vsix files + source tarballs? Currently we are using download.jboss.org which requires Red Hat VPN access, it would be great if we could host these vsix files somewhere upstream, as then we can do the entire build + publish step outside of the VPN.

Good point. I do not have an answer yet. I am investigating if we could use an OCI registry to store and pulls vsx so maybe quay.io? @gorkem any idea?

[gorkem]

I guess we need a CDN backed download are for these ?

[ericwill]

A few comments/questions:

1. How are we going to handle volumes in the meta.yaml file exactly? In the case of the Quarkus and Java plugins they are specifying /home/theia/.m2 as a mount path, but this is not reflected in the che-sidecar-java Dockerfile.

What about adding VOLUME /home/theia/.m2 instruction in the Dockerfile?

Yes that could work.

2. How do we handle dependent plugins? For example, Quarkus requires the jdt-ls, java-debug, and vscode-java-debug, but these are neither in the registry nor are they all in the same repositories. Maybe we could include a deps field in the .json file?

Dependencies are in package.json. I would not duplicate that field. But to be able to identify the dependencies in our registry we need to add the id field AND the vsx PR check must verify that dependencies exist in the registry. What do you think?

I looked into it a bit more and it works in the general case, however there are a few corner cases. The vscode-openshift-connector pacakge.json doesn't specify any extension dependencies, yet in the registry we have: vscode-kubernetes-tooling, vscode-yaml, and a custom vsix che-openshift-authentication-plugin.

So here we need to handle two cases:

• Dependencies not listed in the package.json
• Custom plugins like the che-openshift-authentication-plugin

Maybe a possible solution is to put these in the sidecar as well? I.e. leave meta.yaml only to handle the main extension case, and put all deps in the sidecar?

Speaking of sidecars, we will need a field for the sidecar image location.

[ericwill]

I guess we need a CDN backed download are for these ?

It depends what we are aiming for. If we are going to build and publish (vsix + sources) every upstream vsix, then yes eventually we'll need something like that. Right now we are using download.jboss.org

Alternatively we could allow users to contribute by pointing to a vsix file anywhere (provided it's not the VS Code marketplace), and only host downstream vsix's on download.jboss.org for use in CRW.

[l0rd]

@ericwill "Dependencies not listed in the package.json" looks like a bug. Why an extension has dependencies when running in Che but doesn't when running in vscode? And for che-openshift-authentication-plugin I think that more than a dependency of the openshift connector it should be added on every workspace if Che is deployed on OpenShift with OAuth is active.

[JPinkney]

I've just looked around the openshift connector repo and it looks like they add vscode-kubernetes-tools to their extension dependencies when they're packaging the vsx: https://github.com/redhat-developer/vscode-openshift-tools/blob/40b1716e3c78c08b0a4dddd811af53ffba19a931/Jenkinsfile#L33. I guess that would explain why vscode-kubernetes-tools and vscode-yaml are there as well

[benoitf]

yes, dependencies are added by the CI, they do exist in the final vsix

[l0rd]

@JPinkney @ericwill @benoitf that breaks our mechanism in 2 ways:

• building the vsx is not just npm install
• the dependencies are not in package.json checked-in the repo

For situation like this one (that I hope are rare exceptions) I think we have 3 options:

1. Keep using the meta.yaml to describe those "exceptional" plugins
2. Privilege the already built vsix in the github releases rather then building the extension ourselves AND extracting the package.json from the vsix to get the dependencies.
3. Extend the vscode-extensions.json format to include optional fields customBuildScript and dependencies

And in the short term I would go with 1) and, once we have more experience, decide how to move forward.

[gorkem]

Are there any other examples of dependencies that are not listed on the package.json? I think you are carrying a workaround for missing dependency installation at test environment. Perhaps you can ping @dgolovin for alternates.

@dgolovin Is it possible to solve this by installing the dependency extension to at the install-vcode.ts? We do this on vscode-tekton for instance.

[ericwill]

Are there any other examples of dependencies that are not listed on the package.json? I think you are carrying a workaround for missing dependency installation at test environment.

I agree -- I haven't found any other examples yet, if I do I'll raise them here.

[ericwill]

I don't think this is the same thing, but we also include the vscode-java-debug plugin in the Java meta.yaml, but vscode-java has no extensionDependencies field in its package.json.

A few other remarks:

1. Some plugins have no vsix files at all -- like the buildah, buildkit, and dirigible plugins. In fact in the latter case, there aren't even any releases upstream nor is there a package.json file. I haven't investigated deeply but I suspect they've put the vsix file in their sidecar container (if there even is one at all). We can probably handle these cases manually for now.
2. For sidecars, instead of listing the Dockerfile location wouldn't it make more sense to list the image location so it can be pulled? If not then we'll probably need both, it's not always possible to infer where an image is hosted just from the Dockerfile.

[l0rd]

I don't think this is the same thing, but we also include the vscode-java-debug plugin in the Java meta.yaml, but vscode-java has no extensionDependencies field in its package.json.

That maybe a workaround to make sure that they both run in the same sidecar container. That is a temporary solution while we are waiting for the proper solution to be implemented (the work is currently on hold because the controller team is completing the OpenShift console web terminal work).

1. Some plugins have no vsix files at all -- like the buildah, buildkit, and dirigible plugins. In fact in the latter case, there aren't even any releases upstream nor is there a package.json file. I haven't investigated deeply but I suspect they've put the vsix file in their sidecar container (if there even is one at all). We can probably handle these cases manually for now.

Right. The plugins you mention are not vscode extensions and the vscode-extensions.json mechanism described in this issue is for vscode extension only. Other plugins will be continue to be defined using a meta.yaml file.

2. For sidecars, instead of listing the Dockerfile location wouldn't it make more sense to list the image location so it can be pulled? If not then we'll probably need both, it's not always possible to infer where an image is hosted just from the Dockerfile.

Both approaches have their pros and cons. Inferring where an image is hosted is possible if we use a hosting / naming convention based on the vsx id and version (i.e. quay.io/vsx-sidecars/<vsx-id>:<vsx-version>). With an option to override the name using a label in the Dockerfile.

As a contributor I would probably like more the idea to add a Dockerfile (or reference an existing one) rather then build, push and maintain an image in my personal dockerhub or quay.io account.

[l0rd]

As mentioned in this other issue one vsx may be packaged in different sidecar images (think about jdt.ls in different openjdk images). To support this scenario we may use one of the following syntaxes:

{
   "repository": "https://github.com/redhat-developer/vscode-java",
   "checkout": "v0.57.0",
   "dockerfiles": [
       "./dockerfiles/redhat-developer.vscode-java/Dockerfile.java8",
       "./dockerfiles/redhat-developer.vscode-java/Dockerfile.java11"
    ]
}

or

{
   "repository": "https://github.com/redhat-developer/vscode-java",
   "checkout": "v0.57.0",
   "dockerfile": "./dockerfiles/redhat-developer.vscode-java/Dockerfile.java8",
   "alternativeDockerfiles": [
       "./dockerfiles/redhat-developer.vscode-java/Dockerfile.java11"
    ]
}

[ericwill]

The main issue blocking this work now is where to host the vsix files. If we are to build publish all extensions ourselves, we'll need to find a place to put them.

In the meantime (as a workaround) I can just point to the upstream GitHub hosted release asset, until a decision is made. I am also working on the report generation which should be finished by next week sometime.

[ericwill]

If we are to generate the meta.yaml files automatically we'll also need to push it to a different branch, since GitHub blocks all contributions to the master branch unless there is a second approving vote on the PR, and IIRC direct commits are prohibited.

My suggestion is to host the meta.yaml file on its own branch -- it shouldn't be too much of an issue, it will just mean tweaking some of the build scripts. A side benefit of this would mean the repository appears cleaner/less cluttered.

[l0rd]

I think we should NOT commit the meta.yaml files in the sources. If we are able to generate them from the file vscode-extensions.json there is no need to version them. We should generate them at build time ("The vscode-extensions.json to meta.yaml conversion should be part of the build of the che-plugin-registry." as mentioned in the description of this issue).

[ericwill]

I don't think we should NOT commit the meta.yaml files in the sources. If we are able to generate them from the file vscode-extensions.json there is no need to version them. We should generate them at build time ("The vscode-extensions.json to meta.yaml conversion should be part of the build of the che-plugin-registry." as mentioned in the description of this issue).

I think it might be beneficial to have them versioned, there are often cases where we need to patch a meta.yaml file manually to fix some bug, or corner cases where a plugin requires a vsix file for some unusual reason. For example we had this with the node-debug2 plugin, where we actually had to contribute a legacy extension in order to provide debugging support. If all meta.yaml files are generated at build time these cases will be difficult to maintain.

What do you think about this flow:

• The update script runs nightly, generates any meta.yaml files not already present in the registry based on the vscode-extensions.json file
• Generated meta.yaml files are pushed to their own branch.
• Build script works similarly to now, except meta.yaml files will come from the branch in the previous step.

This allows us to:

1. Manage the cases where we need to patch some meta.yaml files manually to fix bugs
2. Handle the plugins that don't use VS Code extensions (editors, for example).
3. Not manually maintain the "easy" plugins that can be automated, while allowing flexibility to manually maintain the plugins from 1.

WDYT?

[benoitf]

@ericwill done ?

[ericwill]

@ericwill done ?

Yes, apart from #17027