Syntax Highlighting

[bryphe]

This is a collection of some thoughts around syntax highlighting - since Oni2 controls the render/view layer, it's important for it to be able to get handle syntax highlighting!

Since Oni2 will be compatible with VSCode plugins - having compatible highlighting with VSCode is important. In other words, if you install a VSCode language plugin for Python, Go, etc, we should have the same syntax highlighting experience as VSCode for those languages.

At this time - this necessitates using textmate highlighting. (Thanks @CrossR for pointing out though that tree-sitter is on VSCode's roadmap: microsoft/vscode#585 microsoft/vscode#50140)

We implemented an initial approach to this in Oni1, based on vscode-textmate - so a near-term strategy is to leverage that syntax highlighting piece. The downside is that it is node-based, so would have to run asynchronously as a separate node process. I think this is a reasonable tradeoff at the moment to get syntax highlighting, but down the road, we should pursue a native strategy (tree-sitter would be perfect for that - or a native version of vscode-textmate, as it is a wrapper around a native regex library).

Part 1: Client interface for Syntax Highlighting

We should design an initial API for our syntax highlighting that can support either an in-proc or out-of-proc model, and both synchronous and asynchronous modes.

A potential interface for this could be:

module type SyntaxHighlighter {
     notifyBufferActive(bufferInfo);
     notifyBufferUpdate(bufferUpdate);
  
     onHighlightTokens: Event.t(SyntaxHighlightInfo.t);

     getHighlightAt(bufferInfo, bufferLine, bufferColumn);
}

As we get buffer updates - we'd broadcast them to the SyntaxHighlighter via notifyBufferUpdate. As the tokens are parsed and colorized (asynchronously), we'd get onHighlightTokens events that we could use to notify the UI to re-render.

The interface could also provide a way to query the latest highlight information, which we could use for rendering to pick the right tokens.

Part 2: Integrate TextMate highlighting service

We could extract out the Oni service responsible for parsing buffers based on textmate grammars and bring it over as a project like src/textmate-service. We'd also need to vendor a node binary like we do for nvim today.

We could actually simplify what we did in Oni - since we didn't use a webworker at the time, we try to spread the syntax highlighting work over multiple frames via a complex job concept. We could streamline and remove that - the perk of it being its own process is that it won't block rendering or the front-end.

Part 3: Native TextMate highlighting / tree-sitter?

[bryphe]

Closing out in lieu of #161 , which is tracking remaining work.