Improving Addon Installation

[carsakiller]

I have for a long time wanted to get this done, so let's get the ball rolling. Please feel free to provide feedback!

Current Implementation

Currently, if you use VS Code, you can use the addon manager, a webview UI that uses Vue.js to display the available addons from LLS-Addons.

More technically, the UI communicates with the VS Code extension through message ports, as the UI is in an <iframe> context. The UI does not handle the installing/enabling of addons, it simply sends a message to the extension to install/enable a specified addon.

The way that the addons are installed by the VS Code extension, is by using Git to clone LLS-Addons and then the specific addon submodule is initialized. This has a few positives and negatives:

Positives

• Addons can be locked to a specific version
• New addons and updates are all reviewed before being merged into LLS-Addons
• Git handles version control and downloading
• GitHub acts as a simple addon registry
• Using Git submodules allows users to use GitHub, GitLab, or others for hosting their addons

Negatives

• Every new addon as well as updates have to be manually reviewed
• Git submodules can have trouble scaling to many addons
• No support for other editors
• Requires Git

New Implementation Idea

Let me step through how I think we can improve this.

Creating an addon

This step is very similar. You can have definition files in a library/ folder, and you can have a plugin. Every addon can optionally contain a settings.json file, that defines settings that the language server should apply (this is the same info as the settings field in the current config.json. Every addon is required to have an addon.json file that contains info about that addon, like the below:

{
  "version": "1.0.0",
  "minServerVersion": "4.0.0",
  "name": "example",
  "author": "carsakiller",
  "description": "An example for demonstrating a basic use case for a lua language server addon.",
  "repository": "https://github.com/carsakiller/lls-addon-example"
}

Publishing an addon

LLS-Addons should be replaced with a new "addon registry" repo that contains an easy to parse JSON file of "officially" published addons. The JSON could look like this:

{
    "example": {
        "name": "Example",
        "author": "carsakiller",
        "description": "An example for demonstrating a basic use case for a lua language server addon.",
        "repository": "https://github.com/carsakiller/lls-addon-example"
    }
}

Firstly, you use GitHub actions or GitLab CI to create a release that contains your addon as a zip file. Then, once you have a public release available for download, you open a pull request that adds the above info about your addon to the JSON file. Any addon-installing client can then fetch the raw JSON file and have all the officially published addons.

Installing an addon

This is where the bulk of the work is.

While using Git is very convenient, we are not using it as intended and it can behave unexpectedly. Therefor, I think we should switch to simply using the GitHub and GitLab APIs. In the future, we could also add any other platforms that have an API.

The logic for installing addons should move to the language server, that way, wherever it goes, whichever editor someone is using, they are able to install and enable addons. It could instead become its own CLI tool, which should be pretty easy, but then users have to install a completely separate tool, and we will be duplicating the logic that enables addon settings 🙁. To integrate it into the language server, I will need help @sumneko with getting comfortable with the runtime and tooling. I have many questions, here are just a few:

• How do we handle CLI arguments, especially a lot of them (e.g. $ lua-language-server addon enable busted --plugin)
• How do we make HTTP requests
• How can we execute programs like zip

Install Flow

So let's say you want to enable the busted addon, which has been published to the official registry:

1. Download the raw JSON addon registry file
2. Look for an addon called busted. Assuming there is a match, we can look at the repository field to know where it lives.
3. Use either the GitHub or GitLab API, depending on where it lives, to get the latest release (or a specific release) and download the addon.
4. Unzip the addon
5. Set the workspace.library setting and any additional settings (according to its settings.json)
6. If the addon depends on another addon, it can be specified in its addon.json file, and after it is installed, any dependencies can be resolved and enabled.

But what if you have an addon for a small project that isn't published to the official registry? Well, we can also support simply providing the URL to it to install it:

$ lua-language-server addon enable https://github.com/example/addon

We can then use the GitHub API to get the latest release and then follow the same steps as above.

Addon Manager UI

Of course, these changes will completely break the addon manager UI that is being used by the VS Code extension. The UI itself may be easy to fix, as it is just passing messages, but the code that handles those messages and runs git commands will have to all be torn out and refactored. We could maybe add a command like $ lua-language-server addon list --json to get the addon registry JSON file. The UI could then continue displaying the official addon list and interact with the language server binary to enable addons.

---

Please let me know what you think of these ideas, and feel free to offer your own suggestions!

[loganmcbroom]

Without any expertise in this area, the feature that seems most critical to me is the ability to install an addon from a url. The json registry features you mention would be a nice convenience on top of that, but they're not something I'm personally attached to. The registry could be as simple as a web page with a list of links to addon repos for popular frameworks and I wouldn't complain. It also seems like if the json registry were to be set up, it would be built on the "download from url" tool, so I would think any immediate efforts should start there.

I don't know the details of how lls hooks into editors, so I don't know how enabling/disabling certain addons for certain contexts could work in a way that does not involve settings files specific to the current editor, but I'd imagine that's a feature people would need? Given that addons have unique names it could be something that happens within the script, e.g. at the top of a script you could have something along the lines of "---@require busted". With a registry setup that could be pushed as far as automatically scanning the registry for the requested addon and installing it, although that's probably overkill.

Alright that's all the thoughts I have, cool project =)

[carsakiller]

You're correct, the first step is definitely just getting installing from a URL working. The registry is meant to serve two purposes:

1. Make installing popular addons very easy:
   $ lua-language-server addon install busted vs $ lua-language-server addon install https://github.com/LuaCATS/busted.
2. Offer a centralized "official" location for people to publish their addons and have them be discovered.

The perceived value added by purpose number 1 may differ greatly from person to person. It is possible to even have that convenience without the registry by just looking in the LuaCATS organization for a matching addon name. However, purpose number 2 is much more valuable in my opinion. It allows people to publish their addon, give it a nice display name, and a description. Anyone can then use this JSON file for listing all the published addons, for example, the addon manager UI could use this.

The language server's settings are the same across editors. The only "special case" here is that VS Code can have additional VS Code specific settings applied using the settings.json file.

Looking for addons in require statements could be possible, but the performance cost would likely be relatively large for a feature that I personally don't see the need for. Furthermore, for addons that document global APIs (that are not required), we could not use actual require statements but would instead need to use an annotation, like you mention.

Thanks for the ideas! I had never considered not doing the registry at all and instead using the LuaCATS organization as the "registry". While I will probably still do the JSON registry, it is an interesting idea.

[loganmcbroom]

The registry would definitely be an upgrade if you can get it to work!

[CppCXY]

What you are trying to achieve is a command-line-based plugin management tool. In that case, there's no need to use LuaLS itself. You can use any programming language you prefer to implement such a tool and then configure LuaLS to recognize the location of the plugins based on a configuration file like ".luarc".

[CppCXY]

If we consider git as the command-line tool, what we need to do is set up a central index repository to index different versions of plugins. This central index repository would periodically release a release, and LuaLS can pull a plugin list from that release with each release it publishes. This way, when requiring a plugin, it can prompt for third-party plugins and provide an option to automatically download them (using git).

[carsakiller]

You have just described the current method almost exactly 😄. LLS-Addons is the "central index" and the VS Code extension clones it and enables specific addons.

Creating a completely separate CLI tool is an option, but like I mention above, it would be best if it could be implemented inside LuaLS:

The logic for installing addons should move to the language server, that way, wherever it goes, whichever editor someone is using, they are able to install and enable addons. It could instead become its own CLI tool, which should be pretty easy, but then users have to install a completely separate tool, and we will be duplicating the logic that enables addon settings

[carsakiller]

Actually, I just thought of an alternative to the "official" JSON registry idea. I can instead use the GitHub/GitLab API to search for repositories that have a certain topic. This could allow us to simply ask GitHub/GitLab for a list of all projects that have the LuaLS-Addon topic.

Using this instead means there is no quality control and reviewing of addons, but it does mean there is no need to maintain an "official" list. We could of course just have this as a fallback should the addon name provided not exist in the official list.

[goldenstein64]

What about using LuaRocks for installing/uninstalling addons? Devs can create a rockspec file that uploads their type definitions as part of their module or as a separate module.

Creating an Addon

Definitions can live in a types/ folder, which contains a library/ folder of type definitions, a plugin.lua, and config files. Developers that want to include their types in a module can specify it in their rockspec file, like this.

-- example-types-1.2.1-1.rockspec
name = "example-types"
version = "1.2.1-1"
description = {
  summary = "types for Example 1.2.1",
  detailed = [[
    This is an example of how types would be specified and published via LuaRocks.
  ]],
  homepage = "https://github.com/goldenstein64/example-types",
  license = "MIT",
  labels = { "types" },
  maintainer = "goldenstein64 <email>"
}
source = {
  url = "https://github.com/goldenstein64/example-types"
}
dependencies = {
  "lua >= 5.1",

  -- other type definitions this module depends on
  "other-types ~> 0.8.6",
}
build = {
  type = "none",
  -- this creates a folder at lua_modules/lib/luarocks/rocks-5.1/example-types/1.2.1/types
  copy_directories = { "types" }
}

The language server can also create extra tooling for help with writing this file, maybe by executing luarocks write_rockspec in the back-end.

Publishing an Addon

Developers can publish their addon as a module using the LuaRocks workflow, either through the website or via CLI, i.e. luarocks upload example-types-1.2.1-1.rockspec. They can add it to an "official" list of addons by adding their rock to the hypothetical lls-addons manifest, a public manifest created on the LuaRocks website ahead of time.

Manifests are described on the LuaRocks site as just a collection of modules. They are simple to create, and manifest maintainers have the choice of either manually reviewing included modules or marking the manifest as open, letting anyone contribute.

Installing an Addon

Users can install an addon using the LuaRocks workflow using their CLI, i.e. luarocks install example-types. Users can select a specific version via a simple CLI argument. Once installed, LuaLS has to detect that the rock was added, probably through a file watcher, and suggest adding the module to their library configuration. The language server can use a pattern like lua_modules/lib/luarocks/rocks-{LUA_VERSION}/{MODULE}/{MODULE_VERSION}/types to detect modules with type definitions.

Users can also browse addons on the LuaRocks website through the public manifest, i.e. https://luarocks.org/m/lls-addons.

For lesser-used addons, they don't have to be part of the lls-addons manifest, and it takes little effort to upload a module for widespread use.

[goldenstein64]

Also, as far as I understand, there isn't a way to define certain dependencies as only needed for development/testing and not as a runtime dependency.

You can define a test_dependencies field in your rockspec, which get installed when you run luarocks test.

test_dependencies = {
  "busted ~> 2.2",
  "moonscript ~> 0.5",
  -- other mock libaries and stuff here
}

I don't think there's a way to define modules only for development though.

[carsakiller]

I think if this is going to be made, it will have to be made separate from the language server. I would like it to be built-in, as it is very closely related to it, but there are just too many moving parts and problems:

1. No HTTPS support in bee.lua
2. Need to be able to unzip .zip files
3. Language server would need better support for CLI argument handling
4. The language server is going to be refactored in the near future

Developing this completely separately should make things a lot easier... but may require some code duplication.

I have pushed the Lua I wrote for adding the basic functionality of the addon manager. It requires an HTTP library in order to implement GitHub.lua and GitLab.lua... so they are pretty bare-bones.

[carsakiller]

I am looking more into the viability of using LuaRocks as the registry. Not having to use the GitHub/GitLab APIs and potentially sign in is a huge plus. However, I have encountered a few issues:

1. I haven't found a way to get a list of addons that belong to a specific manifest. For example, there is the luajit manifest, but I haven't found a manifest.json for it. I can only get the root manifest and then I don't know what belongs to which manifests.

2. rockspec files are written in Lua, making them hard to parse from any language that is not Lua. While I could possibly embed Lua in the addon manager CLI, I would much rather not have to embed a Lua VM just so I can read a config file. Hacking through it with regex's is also not ideal.

[goldenstein64]

1. I haven't found a way to get a list of addons that belong to a specific manifest. For example, there is the luajit manifest, but I haven't found a manifest.json for it. I can only get the root manifest and then I don't know what belongs to which manifests.

This has been an issue with the LuaRocks site for a while with not much discussion, e.g. luarocks/luarocks-site#59 and luarocks/luarocks-site#141.

2. rockspec files are written in Lua, making them hard to parse from any language that is not Lua. While I could possibly embed Lua in the addon manager CLI, I would much rather not have to embed a Lua VM just so I can read a config file. Hacking through it with regex's is also not ideal.

Not much to add here, but maybe you could use a parser over a full-blown VM? That probably wouldn't work well because even rockspecs expect Lua semantics to "build" the globals in a rockspec.

[carsakiller]

Seeing as that issue 59 has been around since 2015... I think getting access to a custom manifest's JSON file is off the table. Add on top that the rockspec files are written in Lua, and now using Luarocks is perhaps not possible.

[Frityet]

Also, as far as I understand, there isn't a way to define certain dependencies as only needed for development/testing and not as a runtime dependency.

You can define a test_dependencies field in your rockspec, which get installed when you run luarocks test.

test_dependencies = {
  "busted ~> 2.2",
  "moonscript ~> 0.5",
  -- other mock libaries and stuff here
}

I don't think there's a way to define modules only for development though.

there is build_dependencies which works

[CppCXY]

I created a simple GitHub-based lua package management tool, which might initially address this need. https://github.com/CppCXY/luap

[carsakiller]

This appears to be fairly similar to the current addon manager in the VS Code client, as it uses Git to install packages — without the use of LLS-Addons.

Using git has some positives, as explained in the original post, however, it also has some negatives like copying the entire directory structure. For many addons, it makes sense to have a “build” step where the addon is generated, and an output is prepared with the minimum files necessary. That is why I want to also support checking GitHub releases and GitLab releases. This would allow maintainers to have their action runner publish their addon with whatever files they choose.

This is sort of, roughly, similar to how Go's package management works where they first check their proxy for the package before defaulting to grabbing the package straight from GitHub/GitLab/etc. via Git. Ideally, we could also just refer to packages by their URL, like github.com/LuaCATS/busted. It is simple and has worked for Go for a long time. It also means we wouldn't need LLS-Addons, which can realistically only scale so far.

This is also a very slippery slope where you start by creating a simple package manager for addons for the language server, and very quickly start re-inventing LuaRocks with building C, a dedicated package registry, etc. 😆

[CppCXY]

This appears to be fairly similar to the current addon manager in the VS Code client, as it uses Git to install packages — without the use of LLS-Addons.

Using git has some positives, as explained in the original post, however, it also has some negatives like copying the entire directory structure. For many addons, it makes sense to have a “build” step where the addon is generated, and an output is prepared with the minimum files necessary. That is why I want to also support checking GitHub releases and GitLab releases. This would allow maintainers to have their action runner publish their addon with whatever files they choose.

This is sort of, roughly, similar to how Go's package management works where they first check their proxy for the package before defaulting to grabbing the package straight from GitHub/GitLab/etc. via Git. Ideally, we could also just refer to packages by their URL, like github.com/LuaCATS/busted. It is simple and has worked for Go for a long time. It also means we wouldn't need LLS-Addons, which can realistically only scale so far.

This is also a very slippery slope where you start by creating a simple package manager for addons for the language server, and very quickly start re-inventing LuaRocks with building C, a dedicated package registry, etc. 😆

Actually, I didn't use Git directly; I used libgit2. It's also easy to download from releases use rust.

Indeed, implementing a complete Lua package manager is like reinventing LuaRocks. It's not very difficult, but I don't have the energy to focus on these things, so I only implemented the package that can simply download docs for annotation. If anyone wants to further extend this project, they are very welcome to do so.