Static analysis plugins

[rreusser]

Prior art: remark

Goal: Table of contents. References. Other similar things.

Proposed interface for plugins that are applied to the AST after parsing and before use:

module.exports = {
  name: 'tableOfContents',
  plugin: function (ast) {
    return {headings: ...}
  }
};

Presumably it can do the cowboy thing and even modify the AST. Like to add anchors to the headings. The data could get registered globally as:

{pluginData: {tableOfContents: {headings: ...}}}

So that you could write:

[TableOfContents/]

Which would know enough to grab the tableOfContents data and render it to a list. This would also work great for references. You could write:

As seen in [Reference author:"Smith"/]

And then write at the end of the document

[References/]

And again…………… magic?

The other component to this would be a couple visitor style helper functions to make it easier to traverse the AST. Presumably to use these you'd write idyll index.idl --plugin table-of-contents which would have to get fancy about finding and requiring it via node_modules. Seems possible though.

[rreusser]

Or just thinking out loud, you there could be a visitor interface where you're write:

visitors: {
  'h1': function sectionVisitor (node) {
    var name = node[2];
    var anchor = parameterize(name);
    node[1].push(['id', anchor])
    this.sections.push({name, anchor});
  }
}

Don't want to make it too elaborate, but seems like with a bit of glue code could enable a lot.

[mathisonian]

I like this a lot, it seems like it could be pretty powerful. What I haven't quite figured out is where this code would live, and how the user would interact with it. Take the example of a table of contents plugin - the plugin code itself presumably exists as a module that users can install, does that module also include a component that users then include in the markup?

Counterpoint - you could probably get a lot of this functionality by making smarter components, e.g. a References component that knows every time a Cite component is invoked

[dundalek]

I like this project a lot and a support for plugins would be needed for my particular use case.

Will share some thoughts out loud too. I imagine the plugins would be able to hook into the build pipeline. There could be hooks for different stages like pre-parse, AST traversal, post-output, etc. User could include and configure plugins maybe via markdown front matter config.

Besides extending the pipeline I think it would be cool to be able to extend syntax rules. Good example is remarkable parser where rules can be extended. There are three kinds of rules: document-level (core), block-level, and inline-level. Plus there are render plugins to transform newly introduced AST types to markup.

For example I want to have special component for definitions. A component might look like this:

[Definition name:"Newton (unit)"]
One newton is the force needed to accelerate one kilogram of mass at the rate of one metre per second squared in direction of the applied force.
[/Definition]

But it gets quite chatty after a while. So more concise and pretty widely accepted extension of markdown for definitions looks like this:

Newton (unit)
: One newton is the force needed to accelerate one kilogram of mass at the rate of one metre per second squared in direction of the applied force.

I think syntax extensions would be killer. I think success of markdown is because it is a DSL for markup that does not get into ones way as the underlying markup does. So being able to extend the vocabulary for a particular domain seems like a great feature for me.

And now for my particular use case. I am working on a project kmdoc which is a system for writing down knowledge and presenting it in a interactive way. It also uses markdown with some extensions. However, it is quite old from the pre-React era, so I am thinking about revamping it. Building it on top of Idyll seems like a good fit. It would benefit from the toolchain and also better support of interactive widgets.

[mathisonian]

Thanks @dundalek, kmdoc looks like a cool project. I agree with the thoughts above that transformation plugins could enable a lot of cool things and we should probably add them.

Syntax extensions would probably require a bit of restructuring of the compiler, and I'm a little hesitant to add a sort of fragmentation issue to an already new syntax. I think in the short term its probably more likely that we would include something like definitions which are already well defined in some flavors of markdown directly in the compiler.

[rreusser]

One other realization: plugins would need to be async-capable. Use-cases abound.

• reading/inlining data files
• browserifying input js
• inlining image previews as low-res blurred data uri
• fetching remote data (e.g. fetching + inlining static tweet text, then hydrating with the twitter api on load, which seems to be the standard way people do it.)
• tons of others. basically any processing that can't be done synchronously in the SSR rendering of a react component

[mathisonian]

With the spellchecker being removed idyll-lang/idyll-compiler#32, it would be a cool first test case for this to re-implement it as a plugin

[mathisonian]

Added in #280.