Plugin system

[kirawi]

This is a discussion for the future plugin system, created in order to prevent further clutter of #122.

Latest update: #3806 (comment)

---

• Latest status as of April 2024:

The plugin system is being worked on at #8675. The language will remain scheme, but the implementation will make it possible to easily swap languages if for example we choose to implement our own scheme. If we decide to stick with Steel, it can also load cdylibs (see https://github.com/mattwparas/helix-config/blob/master/term.scm as an example).

• Status as of December 2022:

  There's two prototypes we're exploring that could potentially exist side by side: a typed list/ML-like implementation for scripting and a Rust based interface for things that require performance. Could potentially run both in wasm but I'm personally a bit unhappy with how big wasm implementations are, easily several orders of magnitude compared to the editor

  Originally posted by @archseer in Plugin system #3806 (reply in thread)

[filipdutescu]

what is the current status? it is hard to follow the issue. is wasm still on the table? i think it was a great idea. if not what other alternatives are taken into account?

[theherk]

@d12bb I added it, but this list is not all that useful, as it includes things that are outdated from the conversation. Nevertheless, I'm happy to keep it up-to-date.

[connorlay]

Is there any reason not to go down the Lua path?

I'm not personally a fan of Lua after using it with Neovim for a while, although it is much better than VimScript 😉 That being said, picking Lua would ease the switch from Neovim to Helix for some users.

I like the idea of using a small embedded lisp.

[bb010g]

vonuvoli Scheme is another plugin language option.

[alexaandru]

Another +1 for Lua (and I'm also coming from Neovim :-) ). I don't really like Lua as a programming language however I absolutely do love it as a "runtime". And there are a ton of transpilers for it, I for one really love Fennel (all my neovim config is in Fennel and I wrote several plugins in Fennel as well).

With that combo you get the best of both worlds: a lisp-like language with the batteries of Lua. I just kept looking for excuses to write more and more Fennel code :-) That was a lot of fun...

With Lua you'd not just make it easier for some Neovim users to come over, you'd also make it easier for plugin authors to port their work over.

[talvdav]

@d12bb afaik guile is not available on windows, so how would that work?

[stappersg]

Proposing a candidate for a plugin system: rhai.

[archseer]

Rhai is too new and has a bus factor of one. If the maintainer stops maintaining it or it turns out we need certain features or bugfixes then we're stuck.

[SoraTenshi]

Would Javascript be an option? Active development and its probably rather easy to expose an API.

[happysalada]

I think he is just younger and isnt necessarily aware of the fact that there are probably hundreds of people subscribed to this thread receiving an email on every comment.
Give it time and someone passionate can make a great project that the community will depend on.

@LeoniePhiline the energy you have is great, but the maintainers here have to work on this on their free time and they are streched quite thin. Theyve invested countless hours into this and they will have to suffer the consequences of the decision they make. Be careful when you post a comment as this could be badly intererpreted. Someone here could become a co contributor on a project you make later or it could become something you wish you didnt say and cant take back. You likely have a bright future ahead, careful on the way there!

@archseer @pascalkuthe thanks a lot for the work and patience you put on this project. For every little thing that is annoying, there are probably hundreds of people who are very grateful. There is so much work left to do, but so much has been done already ! Keep up the good work, im really looking forward to what helix brings up next !

[GATETECAT4]

Honestly I don't really like the idea of Lisp, and I have no control over what the creators think, because they are the creators for a reason, but I would give a suggestion that I doubt they would accept or even see: Ruby.

Again, ruby is not widely used and I doubt they will implement it. It was just a suggestion that doesn't really add much.

[jhoughjr]

I agree on choosing lisp over wasm might be why there doesnt seem to be plugins as of end of July 2023. It seems wasm would be a perfect solution to allow a larger plugin ecosystem and more development, Far more people on the planet could use a wasm interface over a lisp interface.

[archseer]

might be why there doesnt seem to be plugins as of end of July 2023

Or perhaps it's because we had three separate attempts at integrating WASM that didn't lead anywhere? :)

Integrating WASM isn't just a hundred lines of code either, and the lack of a standard ABI interface means we'd still be targetting something on WASM rather than WASM. e.g. if we integrate Guile via WASM, that's a WASM interface, but I'm pretty sure it's not the one you were thinking of! (or a more serious example, quill only targets rust so far)

[koalazub]

I posted above, but would it be worth examining the wasm 2.0 announcement from a few days ago? Or was this included in your assessment.

Although, I do get the main point about wasm just being way too unstable to get any real progress done

[txtyash]

@kirawi Can we start a curated list of plugin ideas right away. It might seem too early but if we have a list right now then it might be useful for future devs to take suggestions from.

We could ask users to suggest neovim/emacs plugins they would like to see in helix and after a huge list gets created, we could put it in a poll where the devs can get a clear idea of what plugins or plugin ideas are necessary.

[CBenoit]

It would be useful. A few plugin ideas can be found in the issue tracker with the A-plugin label (some closed issues are interesting as well, some are simply converted to discussions)

[theherk]

This is what I was suggesting in this comment. For me, magit is unparalleled as a git tool, and I'd love to see integration that tight in helix. Perhaps in core. It is discussed in #227.

[txtyash]

Now we should think about the best way to do it. To me I thought we should first create an overloaded list and then put it in a poll because I think we can't edit polls to add new plugins(or plugin ideas) later. If we can then it's a nice thing. But since we probably cannot, this should have it's own place on the helix website where a plugin idea could be added to the list on user suggestions. I can full help to maintain such page. If authorization is an issue then I'll create a repo right away where I will add user suggested plugins as they suggest me in chats, discussions or issues.

Some of you guys might be concerned about my recent disrespectful behavior at element, for which I apologize again. I'll try to stay absolutely neutral with that page and put my full efforts in maintaining that page if you guys help me fix marksman for markdown in helix first ; )

[dead10ck]

This could be useful to kind of get a clearer picture of what features the community and maintainers would like to be part of the core experience vs what belongs as a community maintained plug-in.

But as for what to work on, I think once a plug-in system lands, this will largely be determined by the developers who put the effort into it. Devs will naturally pick the features that best motivate them. And what the most popular plugins will end up being determined naturally I think.

[txtyash]

Here is the list: https://github.com/zim0369/helix_plugins

[pascalkuthe]

Hi,
I have been passively observing the discussion for a while (read all the issues/PRs) and I have some thoughts.

Just as a quick preface to understand where I am coming from: to me, the most important benefit of a plugin system would be bringing some features that are critical to me in an editor to helix (most notably VCS integration, something like nvim-lightspeed and a filetree).
While I considered putting in the work myself to work on some/one of these, but reading through existing issues/PRs it seems to me these features are preferred outside core (I would personally love git integration in core, but I can also see why that might not be desirable).

It seems the current plan is to embed scheme, because WASM component models are not stable enough yet.
I think a big drawback of this approach is that scheme is not a super widely used language (just a quick language:scheme on GitHub does not turn up any major projects). While it's a simple enough language to learn, it does mean that people will be less likely to write plugins if they have to learn a new language first. However, more importantly, scheme does not have the same ecosystem that rust does. In rust there are a ton of really high quality crates (examples: git2-rs/gitoxide/pijul) easily distributed with cargo.
This means that scheme plugins would heavily rely on callbacks to core API provided by helix because any other required functionality would have to be implemented from scratch.

Therefore, I would like to propose a rust based plugin system:
Plugins are shared libraries that implement a certain C ABI interface and are loaded with libloading.
To make developing plugins more ergonomic, a helix-ffi create would contain the FFI definitions as well as a save wrapper around these definitions (similar to the implementation of the abi-stable crate but written by hand to ensure the C ABI stays stable).

This approach allows plugin developers to use the full rust ecosystem. Furthermore, plugin authors could easily write "library plugins". These would be crates that depend on the helix-ffi crate and offer some helix specific functionality as a library.
Neovim is actually often adding new API entirely written in lua to the core that is "just" used for plugin development.
By letting this kind of API life on crates.io the maintenance burden for the core developers could be reduced and code reuse among plugins could become more common.

The obvious drawback to rust based plugins would be the need for compiling these plugins.
I would envision this rust based plugin system being mainly used for large/complex plugins.
The plugins could be built in CI and uploaded as release artifacts (that are automatically downloaded by helix).
I think that for complex plugins the advantages of rust (good tooling, type checking, runtime performance, ecosystem) outweigh the higher barrier to entry to plugin development.

For people unfamiliar with rust/simple plugins, a scripting language could still be added to helix in the future (WASM, scheme or something else).
In fact, the API exposed by helix-ffi could be reused by this embedded language.
Therefore, this approach could be a stepping stone towards a more dynamic approach.

I have many more thoughts/ideas regarding the detailed design and comparisons to other approaches, but I just wanted to share my basic idea and gather your feedback. Could you see this as an approach being a good fit for helix?

Because this is a pretty long post, a short summary (with some added points):

• implement a C/System ABI interface for helix for plugins
• implement a rust wrapper around that ABI to enable ergonomic rust plugins
• potentially reuse this API in the future for an interpreted plugin API
• pro: plugins have access to the rust ecosystem
• pro: easily allow code reuse between plugins using crates.io
• pro: rust has great tooling
• pro: rust is more widespread than all scripting language options, so far
• pro: many other languages could also target C ABI (C++, Zig, Go, C to name a few)
• con: plugins must be compiled
• con: no sandboxing

[pascalkuthe]

I personally quite like the idea of rust-only WASM and we could definitely overcome the WASM limitations in that case but I am not sure if the maintainers are open to that.
It also doesn't address the specific use-case I have in mind:
I have implemented git integration based on gitoxide in #3890 and would love to also support other VCS (like pejul or mecurial) aswell.
The implementation is already completely decoupled and everything git specific could be trivially moved to a plugin.
Other VCS could be trivially implemented in the same way.
However VCS are very performance sensitive and rely heavily on memmap and other mechanisms that are not possible in WASM.
So moving the VCS integration to plugins requires native plugins.

Another example where native plugins are required is FFI.
There are some things which are simply too hard to reimplement from scratch and linking to an existing C implementation (which definitely doesn't compile to WASM) is simply easier (example hunspell for spellchecking).

Furthermore WASM needs to be compiled anyway.
While people could write WASM by hand, in practice, nobody does that.
So people will download precompiled WASM instead of the source-code anyway.
In that case its not that far of a stretch to download a precompiled dynamic lib (so compilation would be done by the plugin developer not by the user).
Yes the lib is platform specific but its honestly not that hard to just enforce a naming scheme like "-.hplugin" for example and then just let helix determine the correct URL automatically.

I do agree that WASM is a nicer solution than native in cases where native is not required . That's why I would like to see both options (with perhaps only a trusted few plugins available as native plugins like VCS integration) and native as an easier first step.

[SoraTenshi]

I personally quite like the idea of rust-only WASM and we could definitely overcome the WASM limitations in that case but I am not sure if the maintainers are open to that. It also doesn't address the specific use-case I have in mind: I have implemented git integration based on gitoxide in #3890 and would love to also support other VCS (like pejul or mecurial) aswell. The implementation is already completely decoupled and everything git specific could be trivially moved to a plugin. Other VCS could be trivially implemented in the same way. However VCS are very performance sensitive and rely heavily on memmap and other mechanisms that are not possible in WASM. So moving the VCS integration to plugins requires native plugins.

I have to agree with that, as a 'rust-only' approach completely eliminates the benefit wasm as a plugin system gives you.
Also; it would be not in any way better than just exposing a Rust api / just using a scripting language in general.

[yuuyins]

edit: in all this time using Emacs which crashes, blocks, races too much mainly when using third-party packages/extensions, I can say I prefer rust safety and thus a rust-only wasm over lisp/scheme. I agree with @pascalkuthe

Rust a compiled language; that means no iterative/REPL-driven, easy, fast development as allowed by something like lisp/scheme, which for me is a big plus for extending the system (imagine just if Helix ever implements a much needed for serious use proper client-server architecture, I want this thing uptime 24/7, we need this to be dynamic). Y'all should look into GNU Emacs and its incredible approach (C for underlying software, and Lisp for extending it). A major issue with it is the language in that people really want to make Emacs their total computing environment, so they want to access a modern Web browser on it and so on, which only ecosystems like javascript, python have libraries for

• https://github.com/emacs-eaf/emacs-application-framework (python)
• https://github.com/emacs-ng/emacs-ng (javascript)

also it is a big issue that in GNU Emacs Elisp is actually a bad language for today's concurrent computing world. So a language like Rust, would shine in providing a safe tooling, because at least for Emacs, at least 50% of its value comes from its ecosystem (packages/extensions).

[pascalkuthe]

Rust a compiled language; that means no iterative/REPL-driven, easy, fast development as allowed by something like lisp/scheme, which for me is a big plus for extending the system

For me the reason I want to implement extensions in Rust is precisely the opposite. REPL driven development look appealing at first but honestly I much prefer the reliability and speed that rust provides over dynamic languages. My experience with dynamically typed languages (python, lisp variants, lua, ...) is that you end up with a ton of bugs that a static type system would prevent unless you use copious amounts of testing (which lets be honest most people won't do for an editor plugin).

(imagine just if Helix ever implements a much needed for serious use #312 (comment), I want this thing uptime 24/7, we need this to be dynamic

Both shared libraries and WASM can be loaded and unloaded at runtime so that would not be a problem.

Y'all should look into GNU Emacs and its incredible approach (C for underlying software, and Lisp for extending it).

I think most people here are familiar with emacs. Both Emacs and vim both use C + extension language approach because at their inception no better approach was around. C/C++ were the only mainstream options for high performance code but writing anything in those languages is super tedious. Modern compiled languages (like rust and go) solve this problem by offering high speed, static typing and high productivity. I can definitely see a place for a lisp (or whatever other dynamic language) being available for configuration and simple user scripting. In fact I specifically mentioned that my approach as a basis for such a system in the future.

However I would hope that for complex plugins, that we are able to leverage existing statically typed programming languages to ensure that helix plugins can be just as reliable as the helix core code

[getreu]

@pascalkuthe

However I would hope that for complex plugins, that we are able to leverage existing statically typed programming languages to ensure that helix plugins can be just as reliable as the helix core code

similar to the implementation of the abi-stable

abi_stable promotes itself doing "load-time type-checking".

For Rust-to-Rust ffi, with a focus on creating libraries loaded at program startup, and with load-time type-checking.

Does this imply additional memory safety guarantees compared to a C ABI interface?

[hgkamath]

I'm Just ordinary user chiming in.

From the description from on https://github.com/bytecodealliance/cargo-component , the components model came up because they needed to wrap WebAssembly modules. It seems like there is some need to make a wit-bindgen like tool for the components model, so some solution should arrive soon.

Herein, I try collect in one place, summary of sub-discussions for quick reference, with links included inline to the actual sub-discussions.

Plugin language options

• wasm <- CBenoit WebAssembly plugins system #122
  • variants:
    • wasmtime, web
      • pro: implementation lang: rust
    • wasmer, web
      • pro: implementation lang: rust
    • wasmflow, web
      • pro: implementation lang: rust
    • assemblyscript, web
      • implementation lang: typescript compiled to wasm
    • grain, web
      • implementation lang: Reason transpiled to javascript
    • deno, web
      • pro: implementation lang: rust
  • pro: most languages with wasm target will compile or JIT-compile to it. (grain , assemblyscript)
  • pro: possibility of code-sharing with other editors/ides and porting of their already developed plugins
  • pro; wasm byte code for the wasm virtual machine is distributable across any architecture
  • pro: sandboxing
  • pro: bytecode is almost as fast as native, but much faster than other interpretted languages. The blazing speed of wasm runtime, can make the speed difference of feature implementation between in-core and via-plugin imperceptible.
  • con: cannot use native-binary system-local libraries
  • con: API/ABI not yet stable, development library/tools not yet available
• rust-itself <- pascalkuthe #3806 comment 3644259
  • con: compilation required which is impedes rapid-script-style development (not a scripting language)
  • con: either plugins will need to be distributed as binaries for each architecture, or user will need entire rust compiling toolchain
  • con: some super secure trusted computing environments may not allow users to download/run/link non-approved binaries, even in their own userspace
  • plus all of what pascalkuthe in prev-ans said above
• Interpreted rust <- hgkamath #3806 comment
  • implementation efforts towards rust-REPL include: evcxr, miri, papyrus, IRust, rusti, replit
  • con: no mature implementation, a dependable rust JIT/REPL does not exist as of today
  • con: rust is not known for fast compilation. So, startup and execution performance will also be slow
  • pro: can be a stepping stool towards compiled-rust-based-plugins
• lua, web/luajit web<- JHoanker #2949 comment 1191709989
  • pro: popular, simple easy to learn language
  • pro: fast
  • implementation lang: C
  • con: not many libraries
  • con: not suited for large codebase linking
  • con: problems seen from neovide's experience.
  • possible to support lisp via fennel-lisp, which is written in lua and compile lisp to lua
• lisp family
  • pro: perhaps easy to transpile scripts away from if required
  • con: sparsely used academic languages
  • rust-lisp
    • implementation lang: rust
  • scheme/vonuvoli <- bb010g #3806 comment 3777950
    • implementation lang: rust
    • R7RS
  • scheme/chibi <- archseer #2949 comment 1186159322
    • implementation lang: C, chibi-scheme
    • R7RS
  • scheme/GNU-guile <- archseer #2949 comment 1186159322
    • con: no bindings in rust
  • scheme/archseers-r7rs <- archseer #2949 comment 1186159322
  • GNU-elisp
  • common lisp
    • variants:
      • clasp
        • implementation lang: C++, common-lisp
        • llvm based
      • GNU-common-lisp, web
        • implementation lang: C, asm, common-lisp
    • con: specification larger than scheme archseer #2949 comment 1186159322
    • con: may cause bloat, increase compile-time and startup-time
• janet, web <- glyh #2949 comment 1186221704
  • implementation lang: C
  • pro: has rust binding janet-rs
  • con: rare language
  • con: too new, seldom used
• rhai, web <- stappersg #3806 comment 3629810
  • pro: implementation lang: rust
  • feature: customizable language
  • con: rare language
  • con: too new, seldom used
  • con: slow AST language
  • con: archseer above said it has bus of size 1 ( developed by 1 person)
• rune, web
  • pro: implementation lang: rust
  • con: rare language
  • con: too new, seldom used
• gluon, web
  • pro: implementation lang: rust
  • con: rare language
  • con: too new, seldom used
• javascript <- SoraTenshi #3806 comment 3634736
  • pro: very popular, ubiquitous language
  • pro: javascript engines have evolved to become very fast, within reasonable order of native.
  • con: javascript is a messily evolved language that makes it too easy to write equally messy code with bugs
  • con: huge language to embed, increases compile-time and startup-time
  • runtimes
    • v8: web
      • implementation lang: C++
      • pro: mature, backed by google
    • bun, web : <- gibbz00 #3806 comment 4345201
      • can also transpile typescript/javascript
      • implementation lang: zig, C++
• python
  • via
    • pyO3, web <- jlloh #3806 comment 4072099
    • RustPython, web <- xJonathanLEI #3806 comment 4072467
  • pro: very popular language with huge number of modules
  • con: slow
  • con: making sure compiled-in crate interops with system installed python libraries.
  • con: likely to require multiple virtualenv management for each plugin, which takes up user space disk-usage.
  • con: some pip installs pull in binaries or need local pre-compilation
  • con: huge language to embed, increases compile-time and startup-time
• no plugin system/all features in core <- robertkleene #3806 comment 4070879
  • pro: all most features usually needed by developers will work right out of the box
  • pro: core-devs relieved of need to maintain plugin system, plugin-api and its versioning
  • con: features will need to be maintained in lockstep with helix-core, cannot develop at their own pace. as no gradually evolving plugin-api
  • con: no parallel/competing implementations of features
  • con: raises barrier to entry for a new potential feature to make it to core
  • con: there can be many unanticipated special requirements.
  • con: user can only customize what existing features allow to customize
  • con: can lose ordinary-users with specialized niche-needs fulfill-able via plugins.
  • con: can also lose users, who want the contingency option to want a plugin if an unanticipated need ever arises.
  • con: can lose power-users who want plugin-grade power and control. Such users have potential to become future contributors.
  • con: base install may become bloated, may also increase startup-time
• exposed json api <- ksandvik #3806 comment 4081149
  • pro: any feature developer can use any tool they want
  • pro: would be suited for nushell type json processing
  • Essentially dev energy will move from embedding a scripting language to development and maintenance of a json-api.
  • con: can be very slow, as this will require starting external executables, json interconversion. So not suitable for reactive interaction
  • con: may require external programs/processes to keep running and maintain state
  • con: shell scripting is not the best way to write large programs
  • con: plugins that rely on shell or external tools may not be always be cross-platform and are certain to bring in platform dependencies. This case is like the polar opposite of sandboxing.
• Ref/Links
  • https://github.com/dbohdan/embedded-scripting-languages

The case for plugin-provided-feature

• Not possible to anticipate all special requirements.
• Never possible to claim all needs of all users are met. Plugin candidate features are infinite. A user, due to repetitive editing work, will feel the need to exchange/manipulate text from an out-of-process/network resource (ex website/etc) that could have been done faster with a plugin.
• A plugin is preferable when the feature depends on the existence of the web-resource such as a 3rd party provider of some service.
• Plugins allow for easier private changes. One cannot assume, all created plugins will be shared. One may create a for self-only/organization only plugin in cases where it makes no sense to share, like for a private resource. Without plugins facility, user has to make direct changes to core-code, which implies user is forced to recompile whole editor.
• Another class of plugins are for UI-customization, for which preferences are subjective.
• Plugins have a low barrier to entry for a user to contribute.
• Languages chosen for plugin are high-level, have REPL, suitable for prototyping, and are live code-able. They are hence relatively easier to learn and code, as compared to low level system programming languages.
• Attracts a special-feature-requiring-new-user, if plugin/feature desired by the user exists
• Attracts and retains power-users and users who want customizability and control via plugins
• Plugins decouple plugin development from core-development. If too many features are pushed into core, it increases the maintenance work on core-developer. If the core-feature developer fades away, increases work-load on present maintainers. Larger software has larger cognitive load on the developer, and can cause maintainer-fatigue. On the other hand, if a plugin is abandoned, it may get adopted and re-developed. It may be retired/quietly-dropped with less impact to core. Users may also switch to an alternative competing plugin.
• Plugins, by making life easy for core maintainers, allows for project longevity.
• Plugin installs are separate and self-contained, and hence bring no platform dependencies
• Plugins allow for parallel/competing implementations of features. The competition keeps plugin developers on their toes. Plugin competition, keeps alternative ideas and ways of doing things alive for those who need them, instead of forcing only one-way of doing things.
• A plugin-api forces a interface, which decouples the plugin-code from all of the core-code. The advantage of this is that changes within core and changes within the plugin are less likely to cause unintentional breaks as long as they conform to the api. This is sound, as opposed to monolithic development, where any call/access can be made to anywhere, and one has to work out if a change affects any callees/accessees. A integrated feature would require a proposals, reviews and more steps to be accepted. A plugin, being more independent, can be developed faster and upgraded.
• Script style development is faster to share and contribute. It also allows a user to make a custom tweak in the inner workings if necessary,
• Plugins can be downloaded and installed on need, avoiding bloat.
• Plugins, upon starting editor, can be minimally-initialized and lazily-loaded on need, saving startup-time.

The case for in-core-feature

• Core features work right out of the box
• A candidate feature for in-core feature is very likely one that is a must-have. Over the past three decades, there is some agreement that some editor features are must-haves, and that they might as well be integrated. ex vcs, ruler, file-explorer, file/buffer-diff, terminal-buffer, etc.
• A feature is suitable to be in-core when there aren't many alternate ways of implementing the feature.
• Possible to make-build by static-linking or dynamic-linking to any platform or bundled library in blazing fast native code. Expectedly libraries already exists for almost any feature one might ask for. pro: fast. con: bloats the core, creates build and install dependencies
• Speed: A core feature implemented in Rust will be blazing fast and will beat any plugin implementation. However, the blazing speed of wasm runtime for plugins, can make this difference negligible.

List of candidate plugins

• zim0369 is maintaining a list: https://github.com/zim0369/helix_plugins <- zim0369 #3806 comment 3637032

List of conventional-plugin-features gone in-core

• color themes
• VCS
  • vcs/git magit like VCS/git support #227
    • git-integration based diff gutter on gitoxide (Git) diff gutter implementation #3890 <- pascalkuthe v22.12 (2022-12-06)
• file-explorer File explorer and tree helper #2377
• file/buffer-diff
• syntax-coloring
• bufferline initial implementation of bufferline #2759
• inter-process client-server buffer-sharing Implement Kakoune-style client-server model for sharing buffers across windows in different terminals #312

Desirable characteristics of a plugin system

• gozzarda #3806 comment 4270848
  • even if helix supports dynamically typed plugin languages, desirable to have a statically typed interface
  • asynchronous wherever possible
  • sandboxing, plugin memory separation,
  • a way to marshall calls and return values
• xJonathanLEI #3806 comment 4270883
  • ability to do permissioning on sandboxed plugin
• kirawi #806 comment 3629638
  • Interpreted
  • Reliable
  • Popular
  • Small footprint
• blazing fast byte code virtual machine
• any language should be transpilable to the byte code
• rust implementation

Projects that are noteworthy for their plugin system implementation

• extism, web: <- yogeshdhamia #3806 comment 4285371
  • A universal plugin system
  • wasm based
  • implementation lang: rust
• graph-node, web: <- xJonathanLEI #3806 comment 3988059
  • has a wasm runtime
  • target language assembly script
  • implementation lang: rust
• Zellij, web:
  • bundled plugins <- xJonathanLEI #3806 comment 4071890
  • con: uses stdin/stdout to marshall events <- pascalkuthe #3806 comment 4275006
  • implementation lang: rust
  • wasmer and wasmer-wasi
• lapce , web: <- archseer #3806 comment 4713353
  • Lapce's Plugin Server Protocol, a lapce editor specific Editor Plugin Protocol
  • https://lapce.dev/plugin-server-protocol/specifications/psp/0.1/specification/
• Blender, web: <- Z2Up1UwcaYOyZq #3806 comment 4343127
  • ships almost-all plugins at once with main installer
  • of them, some are enabled by default, whilst others are disabled by default
  • clear distinction between official plugins and non-official 3rd-party plugins
  • seems to be many 3rd-party add-on/script markets.
• Emacs <- jgarvin #3806 comment 4079704
  • Emacs has plugins to avoid/manage RSI
  • Emacs has pushed what plugins can do the most
• VScode, web:
  • VScode compatibility may be desirable
  • In the beginning, extensions were written to restrictive contribution points. Advanced contribution points came later <- spion #3806 comment 4071144
  • More and more developer tool companies will release extensions for vscode, but this sentiment can only go so far <- tubba #3806 comment 4491845
• nushell, web <- getreu #3806 comment 5314935
  • MessagePack (a small binary serialization of JSON) and JSON since Aug-2022 <- fdncred #3806 comment 5860594
• Vplugin : <- tukannoidd #3806 comment 5703649
  • pro: Rust based, low-level block, is its own plugin-API with format,
  • con: still in development, other language bindings not yet there.

Other discussions

• wasmer vs wasmtime
  • WASI Interface Type proposal vs Components Model Proposal <- kirawi #3806 comment 3629471
  • Attempt at helix plugin based on wasmer WASI <- CBenoit WebAssembly plugins system #122
  • The WASI group's plan is to build WASI interation preview 2 over the component model <- tshepang #3806 comment 4496058
  • wasmtime released, quick releases <- ath3 #3806 comment 3781782
• On sandboxing <- xJonathanLEI #3806 comment 3988059
  • Are editor plugins trusted or need sandboxing?
• Performance impact of interface that uses marshalling in function calls
  • serialization and byte buffers<- pascalkuthe #3806 comment 4286540
  • due to memory isolation of WASI, arguments and return values need to be serialized
• On RPC/IPC
  • gRPC, jRPC, tarpc, rpclib <- lukepighetti #3806 comment 4712349
  • RPC are too slow, can stall render loop <- archseer #3806 comment 4713346
    • Xi editor Ralph Levin creator says not a good idea link
  • ability to connect to a remote plugin <- gozzarda #3806 comment 4270848
  • IPC <- drsensor #3806 comment 4781273
• On initial configuration file format vs minimal scripting language for configuration <- amano-kenji #3806 comment 4285371
  • A scripting-language based configuration-file allows for customizing settings dynamically inferred from the environment
    OTOH, a simple standard-file-format based configuration-file allows for quick loading and management by idempotent configuration tools
  • KDL : alternative to XML/TOML/JSON/YAML<- NLKNGuyen #3806 comment 488645
  • dhall-lang : a programmable lightweight configuration language optimized for maintainability <- TheOptimal #3806 comment 4848546
• On a multi-editor interoperable editor plugin protocol <- lukepighetti #3806 comment 4712349
• Determining use cases for plugins <- QuinnFreedman #3806 comment 4672472
  • Goal: to investigate whether a plugin-language is at all required / find out what features are better implemented in-core.
• Articles on writing plugins for rust <- getreu #3806 comment 4876311
• git-branches or initiatives where volunteer developers can contribute <- AngeloChecked #3806 comment 4495169
• redox-OS inclusion <- luccahuguet #3806 comment 4276410
• meta-repos and plugin vetting <- gyreas #3806 comment5842368
  • pro: security, quality
  • con: devs are few, and dev-time too limited to spend on repo, Users sourcing sourcing a plugin directly from its git repo much easier
  • con: barrier to share, centralization, bureaucracy,

so... where we at?

Q1) Is choosing a plugin language now, just a temporary stop gap for small simple plugins until wasm matures, just to buy time for an eventual later migration?

Q2) neovim has both a lua and vimscript engine. Would helix be able to have multiple simultaneous plugin language engines, perhaps aiding transition if the afore-said stop-gap were to be retired?

Q3) Would the plugin api be designed to be so that it mimics vscode so that as many plugins from vscode eco-system be easily ported?

Q4) If lapce (a gui code-editor like vscode) is going with wasm, wouldn't that also allow easy script sharing and porting with helix (and also kakoune if it gets a wasm plugin engine)?

Q5) Would plugin system allow for dynamic load/unload or one time load on startup?

notes

• kakoune is a modal editor implemented in C++, has minimal scripting with kakscript and also in plugin-engine selection-limbo with no official plugin system.

Helix holds much promise. thx to the developers

P.S.

1. This post only serves to collect links to sub-discussions that have been made elsewhere. Credit of content goes to the knowledgeable comment-posters. Any limitation/bias/omission in this post is my fault. Any new information pointed-to or contained-within is thanks to contributors. So, please don't post comment to this post, unless you are alerting me to a needed correction.
2. Make your posts/comments by replying to sub-discussion/subtopic specific comments where the sub-discussions/subtopics are made. The links to the posts/comments can be found above. As the discussion is now closed, do exercise restraint before posting a new thought. Please search/do Ctrl-F to see if the thought is already posted.
3. kirawi has called for end of discussion and need for implementation. Any general closing comments to the discussion are best made to kirawi's post of 20221222 kirawi #3806 comment 4404346

[askreet]

You mention JSON API as an option and that it would be slow to convert JSON constant for message processing. What about a binary protocol? The plugins could be applications started by Helix that meet some binary protocol specification on a unix domain socket. Do we think there are interactions that plugins would want that are orders of magnitude more chatty than LSP, which uses a similar approach (and JSON)?

I just think it's interesting from a feature isolation standpoint. If a plugin crashes, my editor doesn't freeze or crash (assuming it handles timeouts and read delays with grace).

[zachdaniel]

I say this with zero interest in attempting to rush or apply pressure on any of the maintainers of this project, only as someone who is considering using helix as my primary editor(because its really cool, and of all of the things I've tried is the only one that "just works"): is there any kind of rough estimate on when a plugin system might emerge? I don't mean dates, just the general scope of it. Are we talking weeks/months/years?

[gavlooth]

For the sake of completeness, I am adding picolisp here. Its extremely small and has a very powerful ffi and can interact with everything. Without knowing much I suspect it would be very easy to embed.

[LeoniePhiline]

Looks like years is the right answer.

[archseer]

If only we used Cloud Build! Then we'd be up and running in a weekend.

[zachdaniel]

Also: stick with WASM plugins, IMO :) If we want to take Helix with us into the future, we should lean into WASM.

[workingj]

true

[lukepighetti]

100%

[archseer]

... #3806 (comment)

[pianocomposer321]

I have a question...

I don't have a terribly strong opinion about what language helix ends up using long-term. However, it appears to me that no matter what ends up happening, it will take a significant amount of time. Is there any possibility for any sort of short-term solution? In particular one idea I had was to be able to write something in rust that has access to all of the helix commands as functions, and just be able to compile that to some target and put it in the .config/helix directory. I'm far from an expert on rust development in general or helix's development specifically, but since the codebase is written in rust it doesn't seem like much work would have to be done to expose these functions. No need to write a whole api spec or anything like that. It definitely wouldn't be the best idea in the long run, but the idea would just be to give the users some way to script until a more formal solution comes about.

It obviously wouldn't have to be that idea either. I just would like to know if there's any possibility of something to give us the ability to script something until the plugin system is ready. Because it looks like it may be a long time before that happens.

[diktomat]

Well, you could always go the Suckless way and write patches to apply after pulling, before compiling. No support from Helix itself needed, but you would obviously have to keep up with internal code changes (but that would be the case with modules in .config too)

[casey]

This could be a great short-term solution. Xmonad, a window manager written in Haskell, uses essentially this model. The ~/.xmonad config file imports xmonad as a library, and configures it using Haskell, before running it. The xmonad command is a wrapper that checks for changes in ~/.xmonad, compiles it to a binary, and runs the binary.

This would get something working right away, and wouldn't require a different language or a VM.

[ath3]

https://bytecodealliance.org/articles/wasmtime-1-0-fast-safe-and-production-ready

[arxra]

https://github.com/bytecodealliance/wasmtime/releases/tag/v2.0.0 and now there's 2.0

[ghostbutter-games]

https://github.com/bytecodealliance/wasmtime/releases/tag/v2.0.0 and now there's 2.0

They jumped a whole version number between September and now?? How did that happen? 😅

[sunfishcode]

The release process is described here.

[ghostbutter-games]

EDIT: I seem to have come across as trolling or hurtful to some people and for that I apologize. I did not mean to sound overly critical, and should not have implied that overall progress has stagnated, which is obviously wrong: What I meant was that progress on this particular issue seems to be slow, and I can't wait to find out what direction this will take and in what time frame it will happen.

---

First of all: I love the Helix project and wish it nothing but the best. I personally haven't yet invested that much time into learning vim (only dabbled so far), but I really want to learn one modal editor to use for most of/all of my programming in the future.

I have been following Helix since the beginning of this year, and really like what it already is!
However: I feel that this unresolved plugin situation is really hindering the growth of the project.
I don't want to rush anyone or step on anyone's toes, and if @archseer decides that this is not what Helix is all about, I am totally fine with that. I already like it for what it is, but I don't see a mainstream future for Helix unless it has a great plugin ecosystem, to be able to compete with the likes of Neovim.

Totally fine if "competing" with Neovim is not a goal of this project, but I would really like to know if that's the case and what the vision for this project really is.
Growth seems to have stagnated a bit in the past months and many people seem to be wanting this feature, or at least a clear statement on where this is heading.

If a plugin system is something that's far off, like potentially years away, I would like that to be clarified as that would mean that this is not the right fit for me, personally.

ThePrimeagen pointed out in one of his recent videos that one of the great points about learning vim is its ubiquity on all platforms. Of course that grew out of it having been used for decades, and helix might never reach that point – but I'd like to know if helix even aims that high or if this is just kind of an experimental project that's not really intended for mainstream use. Fine either way, but I'd really like to know where the project's core developers stand with regards to this.

[stappersg]

Sorry for coming across a bit harsh, I really really love Helix and its vision, thanks for all your hard work!

Find ways to make Helix possible. If you don't know how, make sure that you are not standing in the way.

[hgkamath]

As an ordinary user, in this comment, i just want to

1. Try help cool the plugin-anxiety in the community.
2. Recap a little on what I think is going on in the editor evolution space
3. Convince reader why there is no hurry and why just using any other modal editor for now, until helix is ready, even if that say is 2 years from now.

1: I am guessing the comment-OP, and other people like me, who want plugins are not a plugin developer, but a plugin user. Possibility of great plugin system is not the only lure of helix. I want to herein convince reader that its present unavailability of plugin-system is an insignificant negative.
Comment-OP's main intent is to learn a modal editor and its scripting language, but assumes that learning how to use either is a big hurdle, and so wants to make a one-time decision. Perhaps comment-OP assumes it will be difficult to switch later and fears that one will be stuck with a current decision.
First, let's recap some reasons:
The main advantage of terminal editors is that they are not restricted to desktop usage. (However, increased network speeds now allow gui-editors to do fast remote-fs as well as direct network editing/remote-development. )
The main reason for terminal editors like vim and emacs, is to do featureful-editing (unlike pico) in a terminal.
The main reason for learning a key-stroke terminal editor is because, as keystrokes get committed to muscle memory.
The main advantage of vi/vim/neovim over emacs, is that vim-style modal editing avoids using the ctrl-key, which is suitable for remote-ttys and tty-multiplexers. (with due respect, emacs has other advantages, pls noone wants to enter this editor-war).
The main advantage of modal editor is that it separates of edit-mode and command-mode and allows for a style of keystrokes.
The main advantage of minimal-vi modal editor is that it can be packaged onto a lightweight minimal system. like an initrd or embedded system.
The reason for kakoune is that for its new selection-first key-bindings, allowing for visual operation.
The reason for helix is that it uses kakoune's key-bindings and visual-selection-feature, and is implemented in rust, which is thought to be blazing fast, has modern libraries (crates), and promotes faster development with lesser bugs.

When one chooses an editor, one wants to use the same editor everywhere as possible, to get best use of muscle memory.
Learning to use one modal editor is not as big a task, and is also a 1-time task. Switching to another one, when a newly developed editor (such as kakoune or helix) matures is not as big a task as one might think. Once, one is sure that the new editor fits all your use-cases, a switch at a later time is easy to do.
First, If one's intention was to make a contribution to helix as a plugin developer/contributor, I see your point why one is in a hurry, can't argue there. But, if one just want to use plugins, then one is better off with neovim/vim. which already have mature plugins, and for now, just watch how helix develops.
The vim/emacs documentation is large. and one might be tempted to think one will need to invest a lot of time learning. But in reality, one uses only a small table-ful of commands that often fit on a 1-page-pdf-cheatsheat. Even as a seasoned user, one never rarely if ever needs to use all of the editors features. The human mind cannot remember rarely used features. When one encounters a situation needing one, for one time-use its often expedient just to manually-repeat. One only takes the time to lookup the advanced tricks, only when one finds oneself doing manual-repeating-work more frequently.
If and when you choose to switch, one will in short order of time, learn to forget the muscle-memory keybindings of vi/vim/neovim and learn that of kakoune/helix. When plugin-system and plugins are ready, Switching rc-scripts, custom defaults and plugin customization-s, are one time tasks, and will be least of the problems. Once, you get your scripts to work the way you want, one seldom sees them again, so one forgets and there is re-learning each time one dabbles into it. Its not like one knows/remembers vimscript the same way one knows/learns/remembers bash/pwsh/python/C++/etc.
tbh, I anticipate one may never fully let go of vi and its key-bindings, its too ubiquitous, one will run into it as default editor in many systems, and one will find oneself in situations requiring its use, even after switching to say helix.

2: editor evolution
Original vi was proprietary unix but very bare-bones.
vim is written in C, had more features, was much more usable and is open-source.
neovim brought a different, but more modern codebase/architecture, that was more conducive to faster community development, supported LSP and lua-engine.
Xi editor, no longer developed, introduced a concept called the rope-data-structure for large files.
kakoune is written in C++, but departs from vim's verb-first keybinding to selection-first keybinding.
tree-sitter was a new parsing idea that improved how syntax was highlighted.
helix is a rust implementation using tree-sitter and the kakoune-style key-bindings.
anyone wants to add here?

3: no hurry
So as mentioned in point 1. it will be easy to switch when the time comes.
For now, one can just watch how helix develops for now, try it occasionally and use neovim/something-else for most purposes.
Its more important to make good design and implementation decisions in well thought way, than in a hurry, pass the tests of time, and not end up in editor graveyard like Xi.
So let developers who are thinking and contributing their free time, take their time to make the best decisions under the circumstances.
It will not be that difficult to make a switch, if and when helix is ready.

[ksandvik]

I think the charm with Helix is that it just works from the box, and I don't need to spend time fixing .init files like with Neovim. So the more we could put in as default behavior in hx without making it bloated, the better. For example, if we had a nice way to just call command line tools and get the output into a temporary shown buffer, maybe even in structured format (similar to nushell or jc), that would work fine for me. Then I could write various custom shell scripts and use those in my environment not putting a burden on custom plugins and a plugin runtime.

Git integration could/ should be part of the default hx editor, for example.

[stappersg]

if we had a nice way to just call command line tools and get the output into a temporary shown buffer

Link to https://github.com/fregante/GhostText/blob/main/PROTOCOL.md an #3478

[theherk]

While I do think a fully fleshed out plugin system is a fine idea, I also love how much works immediately upon installation, @ksandvik. In fact, think it is probably a good design goal, even if not stated, to have a robust, usable, and feature-rich editing experience out-of-the-box, and reserve plugins for unique or resource intense components.

I am excited to see where all this lands and am so pleased that the editor is a perfectly capable joy to use in the meantime.

[xJonathanLEI]

[Summarizing a conversation from Matrix for informational purposes]

Unless I misinterpreted something, it looks like the decision to not use wasm was mainly due to the lack of a standardized interface. Using wasm only opens up the possibility of multi-language support, but it's not automatic due to the lack of the interface types.

So I shared what looks like a workaround of this issue that I've seen in the past.

graph-node has a wasm runtime. It's not exactly a plugin system, but it does expose a wasm API and runs user wasm code (so sandboxing is a hard requirement). Instead of defining the interface in Rust, they're actually doing it in their target language AssemblyScript here in graph-ts, and in turn use a custom codegen tool to turn it back to Rust for use in the wasm host. This way AssemblyScript gets direct access to the API, and for any other language, they just have to implement another codegen tool for bridging the gap.

Generalizing this approach further, we can declare the interface in a format like toml, yaml, or even json, which are easy to parse, and maintain codegen tools for Rust and our primary plugin langauge. This is essentially a low-budget interface type implementation.

Obviously with this approach there's gonna be extra effort for supporting more languages. For that, @archseer acknowledged the value of settling on a single plugin language, and I agree. However, wasm has a lot more benefits other than just "multi-language". Performance and sandboxing (and thus a permission system) are extremely important in a plugin system too. We can totally settle on a single "officially supported" plugin language, and only maintain the codegen tool for that language. It's also future-proof if the plugin author community decides that another language is better - no backward compatibility concern at all.

[lazytanuki]

Nice !
Can't stress enough how important a sandboxing/permission system would be for editor plugins, this seems like a good way towards that goal !

[hgkamath]

In a discussion below, it seemed like sandboxing, in some implementations, may force byte-array ser-de for call arguments and return values across host-guest interface. It may be that ser-de does not affect performance if if such funtion-calls into host are rare.

The general vibe I got while reading all comments is there are those who are enthusiastic about sandboxing, and wasm being a web technology has a selling point of easy to sandbox.

But, is Sandboxing necessary?

Sandboxing

• implies memory isolation
• prevents unintentional or malicious access to something that not asked for.
• allows for a permissioning system.
• wiki

Sandboxing is important in use-cases like

• browsers, where privacy and security needs are very high
• mobile phone apps, where every business entity wants user to install an app, but shouldn't do more than what a user allows
• virtual-machines/emulators, where a guest system should not access any host resources
• interception and firewalling apps, where one wants to monitor or control over what a guest requests.

Editors and IDEs are usually trusted applications that pretty much have access to anything. Users are also economic with plugin choices and they are also trusted, ergo the question: So what if editor plugins do too? Does sandboxing make sense for editor plugins?

For completeness of discussion sake, could one try expand on all the future reasons/use-cases sand-boxing brings to editor plugins. Perhaps also list some known cases in other projects where lack of sandboxing caused difficulties. (just trying to learn here)

Some reasons I could think of

• permission granularity, just because the main-prog requests location-access, one shouldn't assume all plugins should automatically get location-access
• one day if there is a need to port to a system/platform that disallows native calls
• one day if there is a need to port into a browser/webapp which forces sandboxing by design.
• one day if something super virus/malicious untrusted plugin comes along

The above to me seem far fetched and don't feel compelling enough to want them in editor plugins.

[robenkleene]

I'm going to make a suggestion that maybe Helix doesn't need an extension system at all.

The problem with extension systems is straightforward: They reduce velocity because they increase the consequences of breaking backwards compatibility (since existing extensions will be impacted by changes that aren't backwards compatible). In specific terms, they direct text editor developers efforts towards extension API maintenance, and away from feature development.

What about the fact that most (all?) successful text editors have extension systems? I've written about the rise of extension-based editors (see "the Text Editor as Platform"). In that piece, I make the case that VS Code has pushed the extension-based editor as far as it can go. In short, as always happens with innovation, the design space of extension-based editors has now been exhausted. An extension-based system in Helix will put it up against VS Code in design approach. This is a match that Helix will always lose, because one, Helix wasn't formed around extensions at its inception like VS Code was (and creations always bear the characteristics of their inception in perpetuity). But also simply because of the constraints of the format itself, since a web-based editor has a lingua franca of GUI customization (HTML/JS) with no equivalent for a TUI editor.

It's always a stronger approach, especially at the beginning, to focus on your strengths, over shoring up your weaknesses (a later concern). In my opinion, Helix's strongest advantages today is that it achieves a tremendous quality and power while not yet taking on the baggage of an extension system.

Another reason all other successful editors have historically had extension systems is because, until recently, they needed them! There was no consolidation about how languages were supported in an editor. So extension systems needed to exist to implement the glue code between how various linters, autocompletion systems, and other tooling interfaced with the editor. Now that language server protocols have largely consolidated how those features are implemented, the case for an extension system is a lot weaker.

When a paradigm shifts arises, in this case, text editors moving beyond the need for extension systems, it's because the technology landscape has changed in a fundamental way. LSP is that change.

Beat your enemies, not by playing their game, but by changing the rules of the game (just having fun here, I certainly don't think anyone should think of VS Code as an enemy). In this case that means continuing to build new core features at the same amazing quality level of Helix's existing features. VS Code can't do this themselves, because the VS Code team has to maintain their extension system. Zig when they zag.

[robenkleene]

I am also using gitui (similar to tig) for that but to me there is value in having it right in the editor. The context switch of having to look at a seperate window to figure out what changed, read the line number and to type that into the editor feels much more cumbersome then just pressing a keybinding in my editor or using a picker (yes you can technically edit in tic/gitui but I much prefer to keep my editing in helix, especially for complex changes).

tig does not work the same way as gitui (and lazygit). And, if I may editorialize for a minute, I consider gutui / lazygit to be bad TUIs for this reason. There's a historical precedent of delegating text editing tasks from a TUI to your $EDITOR variable. In fact, historically that's been one of the value-adds of using TUI software: One consistent editing environment regardless of task. So you'd have a dedicate interface for browsing email message (e.g., mutt), but when you're actually editing text, you used your $EDITOR (e.g., vim). So mutt opens vim with the appropriate content (e.g., message being replied to). When you read historical documents on Unix, this is just an assumption of how things work, so I always find it odd these lessons seems to have been lost in the sands of time (GitHub's gh tool is another example of a TUI I find unusable for this reason).

tig follows this pattern correctly: Just hit enter on a diff in tig and your brought to the exact line number of the change in your $EDITOR (see editor-line-number (bool) in man tigrc note this doesn't work-out-of-the-box yet with Helix, so I made a little wrapper for myself to support it). Quit your editor and your brought right back to the tig interface where you left off. This works so well, I actually find it's actually, somewhat unintuitively, more efficient than integrated VCS in a text editor / IDE. This is because the key binding space is so exhausted in text editors / IDEs that adding navigating diffs to an editor is usually more keystrokes than just delegating that task to tig.

With that said, I do think dedicated VSC integration could be better than tig if it's done correctly. And I think you're diff is the right approach! I think it would have to be in the form of updates to Helix core, and it should use a built-in key-binding system that allows navigating and operating on hunks efficiently, while co-existing nicely with Helix's other key bindings).

Supporting a workflow like that is exactly my long term goal. Getting a vimdiff like functionality is a bit more complicated because it requires virtual text (but we are also working towards making that happen in the future) but setting the commit for comparison should be fairly easy to do and with the gutter/jumping to the change (and potentially a picker for diffs) is already quite useful

❤️❤️❤️!

[nekodjin]

I think that Helix really does need a plugin/extension API. There some languages such as Idris and Clojure that are really meant to be written in an interactive manner, and unless it is possible to write extensions to support these languages, Helix will fundamentally lack support for them in a way that no other popular editor does. LSP is delightful, but neither it nor any other protocol will ever be able to support specialized, language-specific needs such as these interactive editing modes.

[tubbo]

As more developer tools companies begin introducing VSCode extensions along with their products, as well as some popular libraries used for development, this sentiment can only go so far.

Given that Helix decides to not go with a plugin system at all, these new products will have no way of integrating themselves into the editor. Let's consider this: if Vim adopted LSP when it came out, would NeoVim have ever taken off? Would there need to be this whole effort to "port" LSP into Vim just to get it on-par with VSCode? CoC.vim is basically a fully-fledged JavaScript workspace in your Vim, including stuff like package management and configuration abilities, and this is just to get VSCode plugins working, because Vim itself (well, at least earlier versions) has/had absolutely no way of supporting this kind of functionality.

So, I would say that if Helix doesn't build a plugin system, then we are digging our own grave here, IMO. That said, I don't think it needs to be all that crazy. As others have said in this discussion, most editor plugins these days really don't need much: call a CLI tool of some kind, read its response and massage it into the data the editor needs to show feedback. Not really that complicated, not sure if we need an enterprise-scale common runtime system just for that. LSP and code formatters are really good enough for my needs, personally.

[dead10ck]

call a CLI tool of some kind, read its response and massage it into the data the editor needs to show feedback. Not really that complicated, not sure if we need an enterprise-scale common runtime system just for that

This is exactly what kakoune does, and it doesn't scale well even to simple use cases. All the shelling out is extremely expensive and makes using even a few plugins at the same time extremely sluggish.

[amano-kenji]

Writing a shell script is not a deal breaker, but having to write a shell script in a string literal in a kakoune script was a deal breaker. There is no good way to indent shell script in a string literal in a kakoune script. There is no good way to activate syntax highlighting and automatic indentation for blocks of external programming languages in a kakoune script.

Thus, kakoune sucks at scripting. It doesn't have a clean way to deal with external languages. Because kakoune script doesn't have control structures like if statement, it has to embed shell script in a string literal. It is a failed attempt at an orthogonal plugin system.

It's also difficult to keep IPC(interprocess communication) fast. Making IPC fast is possible, but it requires a careful design and optimization.

I think lua as the official plugin language is a good starting point, but the real deal would be to have one official scripting/plugin language that is run by an interpreter running on sandboxed webassembly runtime. Helix can include the interpreter. Users can configure helix with the official scripting language running on a webassembly interpreter. Then, you would have orthogonality through other compiled webassembly languages. We just need to wait for a good webassembly interpreter.

[jlloh]

Has anybody considered Python? I've dabbled with pyO3 and the interop with Rust is fairly painless. It's seamless interop is one of the things I like about Rust. Also, python is more popular than Lua and more mature than wasm. Although caveats of course is you're still limited by the GIL, and it uses your local python interpreter, so varying local python setups might make the user experience inconsistent, but I suspect that can be circumvented somewhat with virtual environments.

[xJonathanLEI]

Let's not lol.

Using a local Python interpreter would introduce all sorts of problem like you already mentioned. No I don't want to set up virtual Python environment just for using some plugins... I need things to just work like they do in say VS Code.

What about shipping an interpreter like RustPython? I think that's too much. It increases Helix compile time by a lot and it's likely not very portable.

[xJonathanLEI]

Actually, same energy as: #3806 (comment)

[jlloh]

Hmm fair points. But I guess the purpose of plugins should be that it's easy to author, and it's pluggable and it's optional to the core experience. Hence a scripting language is often more successful.

Agree that RustPython is probably overkill (although TBF I don't really know how big it is, nor the compile time).

Throwing ideas out there, how about somehow limiting people to core python3? I don't know enough to comment whether core python would suffice for writing plugins. Cursorily looks like what neovim does for Lua? Although it seems to ship the Lua interpreter.

And also is it a reasonable assumption/pre-requisite to make sure people have Python 3.7+ installed on their machines (my initial thoughts are it is, because most distros ship with Python, excluding windows).

[emilyyyylime]

I don't think shipping only core will suffice because some plugins like discord presence may need more more than that

[ksandvik]

My Mac brew environment now has python 3.7, 3.8, 3.9 and even 3.10. It's a pain to keep a stable python environment around as the scripting system for Helix.
a) best no plugin system and core functionality and if possible easy integration to trigger command line tools and scripts with output to a buffer.
b) Or then a self-contained scripting solution.

[xJonathanLEI]

I think you meant to reply to a thread above, but anyways:

IMO:

No plugins but features well integrated ≈ Contained and sandboxed plugin system (wasm) >>> Any other plugin solution

[LeoniePhiline]

Cloud Build - pure Rust helix plugins

Not sure if Cloud Build has been suggested yet.

Prior art

Both BetaFlight (v4.4) and ExpressLRS (v3.3) have recently introduced Cloud Build.

Both projects deal with MCUs:

• BetaFlight is a flight controller firmware for FPV multicopters
• and ExpressLRS is a radio control link.

Their motivation was overcoming flash space limitations on cheaper MCUs, while being able to add features (= more bytes) for top notch hardware as well as support a wider variety of features such as new models of microcontrollers and peripherals, such as gyroscopes, accelerometers and GNSS systems as well as new protocols for communicating between the MCU and the various hardware peripherals.

Applicability to helix

Like helix plugins, BetaFlight and ExpressLRS features must have the lowest possible latency and be baked into the core software as tightly as possible.

In this discussion, many proposals had to be rejected due to size, RPC overhead, serialize/unserialize/marshal/unmarshal costs and the lack of a stable ABI.

Other options involve more or less esoteric programming languages, where obviously Rust would be the best choice if just we could circumvent the need for users to have toolchains available and compile helix with their own set of plugins.

How it works

In BetaFlight and ExpressLRS, it works like this:

• You select your target (MCU, vendor).
• Optionally: Select a release, branch name, PR ID or git commit ref.
• You select the features you want - in Helix this could be plugins - and a binary is fetched for you from CI.
• If no binary with the selected set of features (helix plugins) has not been built yet, the CI runs, the artifact is stored and sent to you at the same time.

The CI costs are covered by donations.

Bootstrapping

BetaFlight and ExpressLRS both use a "configurator", which is either an Electron app or a PWA with USB access for flashing hardware.

For helix, I would imagine this to be either a plain web app, or (probably not?) part of the helix binary.

As a third option, a tiny bootstrapper CLI can be published, which is only used to pick plugins and then download helix for you from Cloud Build.

But using a small web app appears to me to be the best solution. It would provide UI for the following steps:

• Select a target (Windows, Linux+ABI, MacOS).
• Optionally ("Export mode"?): Select a release (latest release is preselected) or branch name, PR ID, git commit ref.
• Tick boxes for the plugins you want baked in.
• And then just hit the "Download helix" button which will
  • either download a prebuilt binary if one has already been cached,
  • or build and cache a fresh one if needed.

Benefits for helix

A Cloud Build solution would allow for helix plugins to be written in pure Rust, without requiring users to have toolchains installed.

[LeoniePhiline]

@archseer

This now requires us to set up and maintain custom build infrastructure

You already have one.

(not to mention we need to compile per architecture).

You already do that.

You're also implicitly trusting binary blobs from some server that has to be properly secured (and if the server ever gets hacked it would be very easy to deliver malicious payloads to end users).

You’ve got the same issue right now with offering precompiled binaries in the github releases.

Nothing changes. It’s just the same CI with an added parameter (list of plugins).

Tamper-proofing is a solved issue. It is common practice to even distribute binaries through third party mirrors as long as the checksums are published by the primary source.

@pascalkuthe

Creating a plugin API in rust is still needed.
So you need a fixed and versioned API and despite some confusion from some people in this thread that's not that hard to implement with the C ABI and then you can just dynanically load the plugins at runtime.

Of course you need a versioned API! Any possible solution (including lisp) needs a versioned API.

Now imagine you could do all that without the need for unsafe external C bridges, in plain safe rust.

see the selected answer to this thread

The selected answer is outdated; mentioning wasm, an idea which is said to have been discarded.

Exactly like what I propose, the selected answer mentions rust-based plugins for performance. You writing "It doesn't really solve anything either" is just rude. It does in fact solve the problem of using rust-based plugins while not relying on wasm, and neither on people having to have a rust toolchain ready.

But this doesn't solve the primary usecase for the plugin system: A more powerful scriptable config. Which can only be covered by a scripting language (lisp)

The primary use case for any plugin system is extensible functionality. No scriptable config.

A more powerful scriptable config can only ever be secondary.

A scriptable config is a separate issue that should receive a separate solution. That’s a config problem, not an editor functionality problem.

It would help seeing those two realms more clearly as separated.

If you will, a scriptable config system could be implemented as one of many native rust plugins.

[gagbo]

The use case for any plugin system is extensible functionality.

The use case of any plugin system is that you can plug components in.

What you're proposing right now is something where, if I wanted to add a new plugin to test it/see if I like it, I'd need to go to a configurator page, to tick all the boxes I had before plus one, and then wait for a CI process to run and build the editor, and then download it.

And do that for each architecture if I use helix on a macbook and a linux desktop for example.

I really don't like that UX for a plugin system.

[LeoniePhiline]

This is absolutely acceptable if you can have both safe and and fast plugins.

Choose C bindings and you will have unsafe pointer handling, memory leaks, segfaults, and vulnerability hell in the plugin system.

Choose a scripting language (for the plugin functionality, not the configuration, mind you) and you get a soggy laggy editor that calls into foreign scripting runtimes on each keystroke.

What you're proposing right now is something where, if I wanted to add a new plugin to test it/see if I like it, I'd need to go to a configurator page, to tick all the boxes I had before plus one, and then wait for a CI process to run and build the editor, and then download it.
And do that for each architecture if I use helix on a macbook and a linux desktop for example.

Web app:

In a web app, this is easily solved with a cookies / local storage. You will never need to tick boxes for plugins you already have installed - this is a typical task computers are supposed to do for us.

CLI app:

If you have the Cloud Build system API client implemented as CLI tool instead of web app, then it already knows which plugins you use, and it can just add the one you like to try. Same if plugin management was integrated into helix itself. (Just run hx plugin add <plugin-name>.)

Build time:

If you are not running plugins nobody else uses, then your binaries will already have been built and cached, so there is no waiting time at all. You get your binary directly and can start trying the plugin.

On the other hand, if you are experimenting with your own plugin or something new and unknown, then you would need to compile it anyway.

Except if, as said above, the plugin system is based on slow scripting runtimes which would make you quit using helix anyway, due to bad runtime UX.

The image you are painting is unrealistic: You are not changing your set of plugins constantly. Plugging something in does not mean that you rip it back out and shove it in again multiple times a day. It just means composability and extensibility. It means being able to make a connection, an integration happen.

If you require a dynamic system, use something written in a dynamic language, or something static with a stable ABI which is also safe (hint: does not exist).

[archseer]

Go with C bindings

Now imagine you could do all that without the need for unsafe external C bridges, in plain safe rust.

Ironically, Rust does not have a stable ABI so we'd have to compile to C ABI.

[xJonathanLEI]

Ah list of "plugins" to extend the core functionality to be specified at compile time? Sounds like a pending PR rebaser lmao:

https://github.com/xJonathanLEI/helix/blob/35d8be1d03174fa7cdc138b37e8067f8f8457b3d/update_batteries.sh

Welcome to my plugin system :)

[getreu]

Who are your potential plug-in writers? What would be the easiest solution form them?

I do not know if a Helix user survey exist, but I suppose most of us have a modal editing background. Very likely we come from Vim, then Neovim and end up with Helix (for good reasons). Probably most of us have some Lua experience we could easily build on. At the same time I think we have very few users coming from Emacs mastering Lisp. Furthermore, I can imagine many of us have some notion of Rust.

As these are just my personal assumptions, shouldn't we make a survey what plug-in language our users are actually familiar with? The plug-in extension is planned for some audience; shouldn't we know more about them? Once we know what is the easiest solution for our target audience, the chances are high to motivate more people to write fancy plug-ins. A simple survey could be realized in a "discussion" thread.

[filipdutescu]

Thats the point @pascalkuthe was making. Helix "does not care" about this aspect. It solves problems for a subset of people. If that subset is large (as is today) that is a side-effect, not a main drive/goal. Hence, Helix will for the most part prioritise itself, not what a large(r) user-base would like.

If it made technical sense to have an Assembly-based plugin system, that would be the choice, even if the majority of its users would only know Python. The maintainers are already working on this issue anyways, with their own approach. They read opinions, but are not bound by any other than theirs. Afterall, they are responsible for developing, designing and maintaining such a system. They will go with what makes sense and is easy, since they already have a hard, time consuming job as-is.

[getreu]

The art (and pleasure) of our developer job, is to design interfaces, that are easy to consume. This does not question authority in any way.

[ni-be]

True, but an open source project also stays alife from how easy it is for others to join. Since this is all done using private time for free, maintainers change since life can change. So if something is a PITA it might not be that good of an idea in regards to the future of the project.

That being said, with my comment I am not critizing any position, choosen stack or anything, just a general observation and thought.

[getreu]

but an open source project also stays alife from how easy it is for others to join.

This could argue in favour of Rust as plug-in language. A plug-in writer could easily become a (core) developer.
More in general, the advantage of choosing whatever wide spread plug-in language speaks for itself. It lowers the hurdle for new developers to get involved.

BTW: When the Neovim project chose Lua some years ago, Lua had still a niche status. Nevertheless, it was a very smart choice: Lua is extremely simple and easy to learn, flexible, fast and resource efficient. That said, other (new) languages might have similar properties and should be considered.

[getreu]

BTW: I just learned that Lua was embedded as scripting language into a TeX engine already in 2007!

LuaTEX is similar to XETEX in its native Unicode and modern font support. It
also embeds the Lua scripting language into the engine, exposing an interface for
package and document authors. It first appeared in 2007, developed by a core
team of Hans Hagen, Hartmut Henkel, Taco Hoekwater, and Luigi Scarso.1⁸

Modern LaTeX - modern-latex.pdf, page 47.

[nferhat]

Hi!

Did you take a look at what Kirottu/anyrun does for its plugin system? It's quite limited compared to what you'd want to have when builtin plugins for a text editor, rather than an application launcher, but is still quite a good start.

This would still assume that most people know how to use Rust extensively, and quite opposed to the idea of @getreu where we could poll potential plugin developers (perhaps we could use FFIs to integrate with more languages like Lua?)

[banocean]

Btw deno is also written in rust and it's ready for embedding in other projects. Plugins in js/ts will be good, soydev friendly addition 🙃

[hemedani]

helix not fully support denols at all. you offer to write plugin in deno!

[nxy7]

@hemedani I think if more people thought that deno is good choice, then making deno support would be quickly achieved.
In my opinion deno would be great choice. It can use vast npm package ecosystem now, can use TS directly (without explicit compile step) and would be welcoming to most devs. If you ask me TS is much better than Lua (I'm comparing to lua as it's used in neovim). The only issue would be size of the runtime. I believe one of the advantages of Lua is that it's really light and I'm not sure how big is deno runtime compared to nodejs.

[hemedani]

I'm agree with you, and I working on several project based on deno like Lesan

[SomeGuyNamedMay]

is nickel a good fit for this use case?

[nxy7]

I believe nickel is configuration language and would be close to useless when it comes to making plugins. Non existent ecosystem + language barriers (I don't think it's turing complete) makes it a bad choice here imao. Nickel could be used instead of toml for Helix configuration, but tbh toml is plenty for simple turning options on or off.
Probably better use case for nickel are big configuration files (something like nix, maybe some kubernetes solutions could be based off it).

[CaesarXInsanium]

Even Nix itself is a programming language.

[nxy7]

@CaesarXInsanium yea, it is programming language (and I think I've read that Nix is turing complete), but it's crafted toward solving specific goal (just like nickel). I'm active Nix enjoyer (writing this comment from NixOS) and it would be horrible to do any real programming in nix.
Because of strong coupling of Nix language with the niche it operates in I don't think it would be far stretch to call it domain specific langauge (or configuration language) and not a programming language.
My whole point was that plugin language should probably be one of general-purpose languages, as they're better equipped for dealing with wide range of problems.

[win845]

While a lot has been said already, want to throw Mojo on the radar.

Coming from neovim, i really value they have added lua as its perfect to be embedded and has a jit to be fast.
But i really dread coding something in it and rather let it be. Picking a language which is mainstream has a huge boost to the community.

Mojo wants to address pythons performance , while being a superset of python.
Its from the creator of MLIR and SWIFT and targetetet at ML devs who spend most of time making sure
their python code actually does not call any python :)). It was so far only released as preview awaiting a full open source
release.

Read more here.
https://gadgetadvisor.com/ai/mojo-is-a-new-programming-language-from-the-creator-of-swift

[khimaros]

it looks like Mojo can be targeted to WASM, so supporting WASM would pave the way for Mojo support.

[koalazub]

W3C recently released their draft for Wasm 2.0. Could be worth taking another look into

[noorwachid]

its seem no one suggest scripting language like dascript or quirrel (the fork squirrel)

both are made by a company who uses it in their products, not some random guy writing scripting language in the weekends

and no lua please, i have a nightmare writing neovim plugin because 1 based array indexing.

[nxy7]

I think there are 3 'good' ways maintainers can go forward with plugin system.

1. Pick whatever they like (I think lisp is not liked among Helix users, but maintainers want to use it)
2. Pick widely used language (so that writing plugins is easier)
3. Maybe WASM once it matures, because as far as I understand it's not there yet

Anyway I think that dascript and quirrel are such a niche languages, that picking them would lower the value of plugin system. Despite it's flaws lua is lightweight and relatively easy to use. I'd much rather use 'bad' widely used language such as Lua than learn dascript/quirrel. Of course it's just my opinion and there's a chance that maintainers are big fans of those languages in which case 1) applies.

[banocean]

WASM is there, WASI is not there with all system-related api's. Good solution for this kind of things might be WASIX.

[getreu]

Can we use this / learn from it? Plugin User Experience (UX) | IntelliJ Platform Plugin (Link corrected).

[ModProg]

I think those are based on the JVM, something helix probably doesn't want to include.

[zbin]

Yes, IntelliJ Platform Plugin System includes a large number of JVM features. So does Jenkins.

[getreu]

I posted the wrong link, sorry. It should be Plugin User Experience (UX) | IntelliJ Platform Plugin. The document describes general UX requirements for plugin developers.

I also find the pages Plugin Structure and Plugin Dependencies interesting. It categorizes use cases.

[CaesarXInsanium]

Might I ask, what sort of interface would these hypothetical plugin system use? If we assume that the language is chosen, what API would it be able to access?

Should plugins be for language specific code, things that the helix maintainer do not want to provide themselves? Would it allow creation of custom UI's

• allow to request a helix custom plugin buffer to draw on: maybe Git integration plugin like Magit
• functions to place certain characters in certain buffer positions: drawing
• read user buffer contents
• provide completion sources
• modify and maybe override Helix's UI
• call external commands and read their output in order to determine actions/behavior
• access filesystem: maybe restrict to specified directories and git repo.

I really like this editor. If it had parinfer or something like it, I would be using it now instead of Neovim. These are just some ideas. Personally I would like to use Helix to edit some lisp code, as I am reading SICP

[Ericson2314]

IMO plugins systems are tarpit. Have people considered just "write more Rust, recompile the app?" as an approach? I.e. think of the editor as a library, combine with other libraries, and then just slap on a very small boilerplate main function.

[Ericson2314]

And if breaking changes in the Rust are an issue, could go full mono-repo and have a /contrib with a bunch of crates.

[MatrixManAtYrService]

think of the editor as a library

I've been thinking about using this method, not to write helix plugins, but to reference helix as part of a zsh plugin (or maybe a fork). The goal would be that unlike the existing vi and emacs modes, it's not a partial implementation--it's the real thing, and it respects your existing config.

Does anybody know of projects that already interface with helix in this way? I can perhaps use them as a guide.

[mitchmindtree]

@Ericson2314 totally - would love to see some API / crates / docs for a compile-time extension approach first - then if folks want higher-level extension options like WASM or scheme, they could always be implemented on top, potentially out-of-tree.

[rishflab]

Yeh I think this is probably the best bet. This would be the fastest way to explore the concept and figure out a plugin API. It could be extended to support WASM once a concrete API is determined.

You could also load the plugins as dynamic libs.

I don't think the complexity of rust or language choice is much of issue when it is confined to a Plugin API.

[cloudhan]

IMHO, continue discussing on this thread will not have a consensus. Why not just take the python community routine, Benevolent Dictator For Life, the helix creators decide what to use. If the scheme is the one to go, then that's it. In the future API design, expose the eval function with some additional data manipulation API will make other langauges also possible.

[pascalkuthe]

IMHO, continue discussing on this thread will not have a consensus. Why not just take the python community routine, Benevolent Dictator For Life, the helix creators decide what to use. If the scheme is the one to go, then that's it.

this is exactly what's happening. @archseer is our BDFL. As we said often in the matrix we already decided on using a scheme. Its also why we marked this discussion as anwsersd (and posted exactly that information in the response). People just keep necro-bumping it

[kirawi]

Maybe the title should be changed to "We chose Scheme as the plugin language!" so people won't overlook it anymore, :P

[banocean]

What about using wasm/wasi/wasix, it have good build-in support and for other languages. For things like js/lua/python it would be possible to just embed runtimes like deno into plugin using wasix (so for example all scripts written in js would require a plugin to run and provide access to helix apis).

Also wasmer and wasmtime crates allow to easily embed runtimes in other rust projects.

[banocean]

Nvm it looks like there is already selected similar approach, the only thing that I think is needed is the wasi/wasix support to provide access to the system apis that aren't present in normal wasm.

[khanra17]

Hell no
Stay true to Rust and Lua. Let's not introduce JS into the mix – it's best to avoid that poison.

[rishflab]

As I said in the earlier thread, I think the implementation details(language/runtime) of the plugin system are not important as this stage.

First we need to figure out what sort of features we would like to enable with this plugin system, then define a plugin API that allows these features to be implemented and then finally the implementation details(language/runtime etc) can be considered. A plugin system is a huge undertaking which is why I think it is best to pick one feature, for example source control, and implement a plugin system just for that feature OR pick an existing feature and make it modifiable using a plugin.

I feel like both of the plugin threads have been unintentionally derailed by people listing their favourite scripting languages and/or runtimes. The difficult part is going to be deciding the scope and capabilities of the plugin system and how it will affect the vibe of the editor. A lot of people have praised helix for its ease-of-setup compared to neovim or emacs. Something to keep in mind.

[archseer]

This thread has devolved into language popularity discussions and most options have been mentioned multiple times already (and no, we won't pick XYZ because it's your favourite language).

I've kept the discussion open because there's occasionally been an interesting comment or project mention. But as the discussion grew older ideas would get buried in the backlog and brought up again several times. Since each post on this thread pings a couple hundred of us subscribed to the discussion I'm going to lock the issue for the time being. If you'd like to have a technical discussion on the implementation we're always available on Matrix.

---

I plan to write a longer post that discusses alternatives we've considered (including WASM) and the design constraints chosen but we're currently leaning towards a Scheme-like implementation (in fact @mattwparas in #3806 (comment) has already implemented a nice prototype that even covers a vim-dirvish-like file tree, the furthest any of the plugin system attempts has gotten).

• From maintenance perspective, Scheme is a very small language (you can read the full spec in one sitting) where we can own the compiler and the VM while adding a very small footprint to the editor. I'd like to maintain Helix for a long time so using an implementation we fully understand and can hack on is a big benefit (we also don't have to worry about the language dying).
• The syntax is not to everyone's liking, but it's very easy to parse, unambiguous and could easily be a compile target from a higher level language.
• The macro system is flexible and expressive. elisp is one of the main reasons emacs is so popular and extensible. (why should emacs be the only editor with a window manager! :)

Helix is a pragmatic editor: it should behave as you'd expect out of the box, and I don't expect users to write significant amounts of Scheme unless they're plugin authors (yak shaving over thousands of lines of config files is precisely why the average user has switched to helix!). In fact the core has been extensible enough that we haven't seriously needed a plugin system yet. But I'd still like to provide a plugin system flexible enough to extend the editor for less common usecases

Why not WASM: WebAssembly is popular but it's not the magic solution to every problem. We need to expose a very large ABI to the language and since there isn't a cross language compatible memory layout we'd still end up being locked into a single language running on top of WASM (or the maintenance burden of multiple shims to support other languages). We'd only be getting the benefits of WASM's VM and we'd be importing projects orders of magnitude bigger than the editor itself. It's not clear to me if the tradeoff would be worth it.

Even if we were able to support multiple languages via WASM, it seems better to me to focus on a single language so the ecosystem doesn't fragment into smaller parts where each plugin I'd be running could be in a different language, making contribution a lot harder.

Why not compiled plugins: The scripting language should also be used for configuration. While there's software that does this (e.g. dwm) it puts a hurdle on the end user: Helix would then require Rust knowledge, and switching a setting could result in several minutes of compile time.