Add ES|QL syntax highlighting to EuiCodeBlock and EuiMarkdown components

[drewdaemon]

It would be great to support syntax highlighting for ES|QL in EUI.

ES|QL is key to our strategy as a company. Users interact with ES|QL primarily through the ES|QL editor:

However there are a couple of prominent places we are displaying ES|QL queries in Kibana using EUI. One is the inline documentation. Another is the AI assistants.

Example from the inline docs (uses EuiMarkdown, no highlighting):

Full support in EUI would improve the appearance of these occurrences and encourage the use of both ES|QL and EUI.

ES|QL technical background

ES|QL syntax highlighting in the editor is supported by the Monaco and ANTLR libraries (editor, and parser-generator). Our lexer generates a list of tokens given an ES|QL query. Then, these tokens are passed to Monaco, along with a set of coloring rules that are applied based on the token identifier.

The lexer currently lives in Kibana itself. It is generated code so it is automatically updated on a weekly basis from any changes made in the ANTLR grammar on the Elasticsearch side. ES|QL is an evolving language, so this is a plus.

EUI technical background

EUI’s syntax highlighting is tightly coupled with the refractor library and the Prism library which it wraps. Prism allows custom languages to be registered, but they are based on regex rules. Simple example:

Prism.languages.csv = {
    value: /[^\r\n,"]+|"(?:[^"]|"")*"(?!")/,
    punctuation: /,/
}

As I understand it, refractor generates an AST from the Prism highlighting and then EUI renders that AST into the code blocks.

Questions/technical thinking

First I thought about how we might integrate ES|QL seamlessly into EUI’s current system. Prism does not support imperatively-defined language definitions, so leveraging our existing lexer is not an option. We would essentially have to come up with regex based rules for ES|QL. We would not reap the benefits of our automatic updates to the lexer, since the regex rules would need to updated as a separate step. However, this approach has the benefit of not requiring any refactor on EUI’s side, and offering ES|QL support in a popular library like Prism might be a nice thing strategically.

Another option I thought of was to build a refractor-like API that uses the existing lexer under-the-hood to generate a refractor AST. EUI would have the responsibility of routing ES|QL queries to this API instead of refactor, but very little would have to change in the downstream rendering system. This custom AST generator could either be injected into the relevant EUI components from Kibana, or the lexer could be separated from Kibana, becoming a shared dependency between Kibana and EUI. There are pros and cons to either strategy and one might be a stepping stone to another.

Custom theming

There is also the question of custom theming. We have defined custom theming for the ES|QL editor (though using EUI theme colors). We might like to keep that same theme consistent for all appearances of ES|QL in Kibana. This opens the question of code theme overrides. Again, here we have the option to have these injected from Kibana, or we could house them in EUI and import them into Kibana.

Conclusion

I am thinking broadly here and hoping to get technical feedback and ideas from the EUI team. No need to view this as anything more than a discussion about the technical landscape. Thanks in advance!

[stratoula]

Thanx Drew for the investigation!

I am assuming that more and more places in kibana would want to show an ES|QL query as a code block / markdown so it would be awesome to support it in EUI. If not, we could handle it at the kibana side.

So interested in seeing what you think @JasonStoltz. It would help us to prioritize accordingly.

[stratoula]

Thanx Jason, this answers my question. As ES|QL becomes step by step the language of the stack I think it should be supported by the EUI components. I am expecting that the language will start becoming popular so ex-kibana users would want to use our components to render ES|QL queries.

So for me the ideal implementation is that ES|QL is official supported by CodeBlock and other components and can be used everywhere (not only kibana). We will figure out the technical implementation but being plugged in by every consumer doesnt seem ideal to me.

[JasonStoltz]

being plugged in by every consumer doesnt seem ideal to me

I'm sure there are ways we could make this less onerous. If we have to we could make the parser/lexer a dependency of EUI, maintained elsewhere.

[stratoula]

This is what we also had in mind. It is not about you be responsible for this but for easily supporting ES|QL in the EUI components for all the consumers (kibana or not)

[drewdaemon]

@JasonStoltz could you make sure one of your devs sees this issue and can comment from a technical angle?

[JasonStoltz]

@tkajtoch

[drewdaemon]

I did notice that we are currently using regex-based rules, similar to what Prism uses for our docs ES|QL highlighting. A "quick fix" would be to copy those. However, we'd have to solve the problem of keeping them up-to-date with new commands and syntax.

[drewdaemon]

cc @nik9000 I heard you were behind these regex rules

[drewdaemon]

Thanks @leemthompo for pointing me to these.

[nik9000]

I am! It's an old, old story. The docs use a fairly ancient version of prettify. I think that's even out of date.

In an ideal world, we'd standardize on something else. Something that's shared and maintained. But I'm not sure who has the time to move off of prettify on the docs side. On the other hand, building a new regex here or there is something we've had time for.

That's the story behind the regex stuff. I'm not proud of it. I'm not really proud of any regex I write. But it get's the job done for now.

[smith]

VSCode uses TextMate grammars for syntax highlighting (example: https://github.com/rosshamish/kuskus/blob/master/kusto-syntax-highlighting/syntaxes/kusto.tmLanguage.json)

If there were a way to generate a grammar like this from what's happening in Kibana that could drive VSCode syntax highlighting.

[drewdaemon]

Thanks for pointing this out, @smith ! Just to be clear, are you saying this is relevant to the EUI discussion, or are you broadening the discussion to include IDE syntax highlighting support?

[smith]

This is not about the EUI component, but I just wanted everybody to know how VSCode does it because I and others would like to have accurate ES|QL highlighting in VSCode and hoping whatever solutions we come up with leave that open as a future possibility.

[drewdaemon]

Okay thanks for the clarification.

I and others would like to have accurate ES|QL highlighting in VSCode and hoping whatever solutions we come up with leave that open as a future possibility.

100% something we want to do, and 100% a door we are not going to shut!

[tkajtoch]

I've looked into kbn-esql-ast, kbn-esql-editor, and our components including the EuiSearchBar. I'd love to see a universal solution that works with all usages across Kibana and EUI, and I think a custom EUI API for passing an AST processor would be the best and most flexible solution.

This work could also decrease the overall bundle size by eliminating the need to ship multiple tokenizers/parsers/regex-based processors to clients and instead provide a single solution to support them all. This would be pretty complicated to pull off, though...

My general take on this issue is that I don't fully love having the DSL for EuiSearchBar in the EUI repo since it's so tightly coupled with Elasticsearch and Kibana instead. The same should apply to ESQL processing - its definition should live outside our repo, and we should only provide an API to render the data nicely.

As an immediate solution, I'd suggest adding a lightweight, regex-based ESQL parser to cover 99% of its syntax and have it correctly highlighted. Without a (somewhat heavy) language processor, there's no way to support 100% of the language features anyway, and it usually isn't needed just for syntax highlighting.

[drewdaemon]

As an immediate solution, I'd suggest adding a lightweight, regex-based ESQL parser to cover 99% of its syntax and have it correctly highlighted

You mean a registering a Prism language, right?