Plugins and middleware approach

[liabru]

I'm investigating patterns for a plugin / extension approach for matter.js. Ultimately I'd like to maintain a collection of small module packages named matter-* that follow a common pattern (e.g. like the grunt / gulp ecosystem) rather than a monolithic library.

This should hopefully:

• make the core lighter and simpler
• allow new features without feature creep worries
• encourage community contributions without needing pull requests

Some goals for a plugin approach are:

• a common pattern
• simple to define
• simple to use
• forwards and backwards compatibility
• compatibility with other plugins
• promote low coupling

To start we can say that plugins should be:

• CommonJS module format
• Built with browserify / webpack for browser use

One of the more difficult concerns for plugins is the need to extend existing methods like Engine.update or Body.create with new features. To do this plugins will need to hook these module methods so they can apply their own operations at the right moment. We could use events for this, but this requires definitions of lots of new events and there is little control over order of execution.

My proposal is that plugins can patch any module method through function composition.
It could work like this:

MyPlugin.js

var MyPlugin = module.exports = {};

MyPlugin.patch = function(base) {
    // check dependencies
    assert(base.Engine);
    assert(Matter.Plugin.has(base, 'matter-another-plugin'));

    // or we can load our own plugin dependencies directly
    Matter.Plugin.use(base, MyPluginDep); // or 'matter-my-plugin-dep'

    // use function composition to patch and extend
    var engineUpdate = base.Engine.update;
    base.Engine.update = function(engine, delta, correction) {
        MyPlugin.Engine.update(engine, delta, correction);
        return engineUpdate(engine, delta, correction);
    }

    // more patched methods here
}

MyPlugin.somethingNew = function(things) {
    // doing something new has no need for patching
}

MyPlugin.Engine = {};

MyPlugin.Engine.update = function(engine, delta, correction) {
    // do plugin stuff on the engine
}

App.js

var Matter = require('matter-js');
var MyPlugin = require('matter-my-plugin');
var AnotherPlugin = require('matter-another-plugin');

Matter.use(MyPlugin);
Matter.use(AnotherPlugin);

// Matter.use is sugar for Matter.Plugin.use(Matter, MyPlugin) e.g. 
// (base, plugin) => { if (!Matter.Plugin.has(base, plugin)) { plugin.patch(base); base.plugins.push('matter-my-plugin'); } }

// we can now use Matter.* as normal, with additional features :)

var newThings = MyPlugin.somethingNew([Matter.Body.Rectangle(...)]);

I realise that monkey-patching like this might make some people wince. But it's extremely powerful.

It's also optional! As it's entirely possible to use plugins defined like this without using Matter.use (obviously takes more work on the user's part). Documentation of plugins might be a little tricky...

Candidates for plugins so far:

• Matter.Render
• Matter.RenderPixi
• Matter.Runner
• Matter.Svg

If anybody has any comments, ideas or examples of other good approaches I'd be interested to hear.

Edit: Looks like Vue.js is using a very similar approach.

[roboshoes]

I like the idea. This is a very powerful technique, and (partly thanks to JavaScript) would allow contributors to make almost any kind of modulation, without having to alter the core.

[Aralun]

This is a very exciting idea for many reasons I won't detail because reading is boring!

I do have a question. Should there be some kind of feedback/awareness of Matter.js of who changes what? To follow your examples, something that would say:

Matter.Engine.update has been altered by matter-my-plugin

(I know this could be done using various strategies, I am asking if it should be)

[liabru]

@Aralun yeah there should be a way of tracking changes made by plugins, particularly for debugging or for making special cases. I've started an implementation, it's looking good.

I'm currently trying to decide whether I should go for a flat or recursive plugin dependency structure. At first most plugins will likely have no other plugin dependencies, but I like to think forward. It would be cool to build small plugins that build on each other.

A flat dependency structure would mean that all plugins must be installed on the Matter namespace (simpler). Recursive would mean that plugins can be installed on each other (less simple, but more powerful).

[Aralun]

If someone builds a plugin to, say, detect collisions of a specific body or some kind of render engine or multi-client synchronizations or whatever, people will want to build upon them (use-case: a video game...)

But isn't this already possible with npm packages and maybe some ES6 magic? Do you actually need to do anything in that matter, or just provide some guidelines as to how to make a plugin pluginable?

[liabru]

I see what you mean, let me clarify for everyone how I'm defining plugins vs just modules as the terms overlap:

Modules alone are intentionally generic, they don't normally extend existing modules directly. They tend to create new functionality but expect the user to implement it into their application directly (e.g. must use new constructors, add event hooks or update things manually on every tick). They may not be compatible with each other and there are limitations on what can be achieved because implementations are often hidden.

Plugins are also modules but they are much more specialised for a particular application. They are designed to fit in to (or modify) an existing pipeline, in a way that remains highly compatible and composable with other plugins (both forwards and backwards). They should require very little work to use.

So to implement plugins requires some sort of special work from the host framework to accommodate closer integration and flexibility than you could get with modules alone.

I guess a good example of what I'm trying to describe would be the concept of middleware in web frameworks, for example express middleware.

Does that make sense?

The closest thing the OOP world has to plugins would maybe be mixins (or multiple inheritence) and dependency injection. But these concepts alone aren't really for creating chains of actions like we want. You'd have to combine them with events unless your APIs are already based on streaming. Promises are a neat idea here but they are designed for asynchronous operations.

I did a quick look for examples of mixins in JS and came across an article titled Mixins Are Dead. Long Live Composition. It advocates an approach related to what I'm considering:

How do we solve this without a mixin?
A higher-order component is just a function that takes an existing component and returns another component that wraps it.

As for any ES6 magic with regards to composition, there are Proxies which would be a possible way to patch things but they are way more heavyweight than needed.

[liabru]

It turns out that the Redux project provides a great article on how their middleware is implemented. It's similar to what I've been playing with. They also take it a step further than monkey patching but it ends up being less easy to understand and requires the patched function to implement the pipeline (I think). The approach I'm working on allows anything to be patched. They also have a requirement for async, which matter.js generally doesn't (but it would be nice).

[liabru]

@antirez just posted his ideas about a module system for redis. In particular he mentions about maintaining back compatibility while at the same time allowing access to core internals. This seems like an ideal scenario, but I'm not sure how it can be pulled off easily for most libraries. He suggests that plugins specify a target version and that a compatibility layer handles changes. This seems pretty reasonable, although may prove tricky in some cases depending on the nature of the changes.

[liabru]

Just an update on this, I'm very close to having finalised the plugins implementation and it will be published in a branch soon!

[liabru]

The plugins branch is now pushed! Check it out!

Take a look at the code for Matter.Plugin to see the implementation.

There are also two new wiki pages on Using plugins and Creating plugins that go in to detail on the approach and the design of the plugin system.

Here are three simple plugins you can try:

• matter-attractors
• matter-gravity
• matter-world-wrap

You can see them in action by running the attractors example.

I'd appreciate it if you guys got back to me with your thoughts!

[Swendude]

Any idea when this will be merged into the stable version? I am currently unable to build the release myself. Really interested! Great job 👍

[liabru]

This has now been merged into master and I've updated the edge build which now includes Matter.Plugin. This should make its way into a release soon!

[liabru]

The plugin system is now available in release 0.11.0.
See the Creating plugins and Using plugins pages for more information. See also the docs for Matter.Plugin and the Attractors demo.

A plugin boilerplate will be released soon to make it easier to get started!

[liabru]

It's been a pretty huge task to get this done how I wanted, but now these plugins are now fully packaged!

• matter-attractors
• matter-wrap

Also check out the matter-plugin-boilerplate.

I'm going to close this thread for now, any issues with plugins should be opened in a new issue (on the appropriate repo). If anyone still would generally like to discuss the approach some more, feel free to post here.

Thank you everyone for your input!