Compiler plugins

[mathisonian]

This adds support for custom compile-time plugins. This feature has been discussed in #70 quite a bit, and should enable fun things like references, tables of contents, etc. Some other use cases could be shelling out to other languages like r/python (or DSLs like vega-lite, penrose) to generate charts at compile time, fetching dynamic data, code linting, spellchecking, etc.

I've added a new module, idyll-ast, that will be useful for plugin authors who want to manipulate the AST somehow. This module is factored out of the compiler, which now depends on it, so it should be pretty stable.

Here's how it works: you define a plugin and the plugin receives the AST as input. It can modify this however it wants, and the results will be passed on to the runtime. A very simple example:

const AST = require('idyll-ast');

module.exports = (ast) => {
  return AST.appendNode(ast, AST.createNode('div', {}, 'Hello World!') );
};

That is all. Right now it is expected that each of the plugins will be a standalone JS module. You can configure Idyll to use plugins in the following ways:

via package.json

"idyll": {
  "compiler": {
    "postProcessors": ['my-idyll-postprocessor']
  }
}

via API

var Idyll = require('idyll');

var idyll = Idyll({
    compiler: {
       postProcessors: ['my-idyll-postprocessor']
    }
});

in browser (manually)

const compile = require('idyll-compiler');
const myPostprocessor = require('my-idyll-postprocessor');
const IdyllDocument = require('idyll-document');

const ast = compile(idyllMarkup, { postProcessors: [ myPostprocessor ] });

return <IdyllDocument ast={ast} />

I'm happy to make API changes if people have preferences. I'll be testing this a bit more before merge.

Notifying folks who participated on the previous discussion:

/cc @rreusser @dundalek @heathermiller @bclinkinbeard

[mathisonian]

/cc @sirinath

[rreusser]

Main question is about whether async behavior is a natural extension since there are a number of use-cases. (fetching remote data, inlining base64 images, more?) It's fine if the answer is "no", but just curious.

[rreusser]

(Oh, and also this is awesome! <3)

[mathisonian]

Great point - we probably do want to support async. One option would be to determine if things are async would be to check if the plugin takes a callback, e.g.

async callbacks

const AST = require('idyll-ast');

module.exports = (ast, cb) => {
  cb(AST.appendNode(ast, AST.createNode('div', {}, 'Hello World!') ));
};

or just check if the function provided is async? This feels like a little fancy but maybe people are used to the syntax by now?

async functions

const AST = require('idyll-ast');

module.exports = async (ast) => {
  const myValue = await fetch(...);
  return AST.appendNode(ast, AST.createNode('div', {}, myValue) ));
};

[rreusser]

TBH I've never actually used async, but I have nothing in particular against it except for the fact that it's nothing you can't accomplish with callbacks. 😄

[mathisonian]

The last commit adds support for asynchronous plugins via callbacks:

module.exports = (ast, cb) => {
  cb(err, updatedAST);
};

The compiler is completely synchronous ATM, so this change required updating the API to support asynchronous usage - I'll update the dependents to reflect this.

[mathisonian]

This is published in idyll@2.7.0, and I'll be adding a section to the docs site to match.