Allow external lexer to be configured with Live Preview

[itegebo]

Live Preview is a great feature; it'd be even more useful if you could configure an existing lexer rather than approximating an existing one within the BNF.

This issue was prompted by:
http://devnet.jetbrains.com/message/5521947#5521947

[gregsh]

"Existing lexer" is a program with any side-effects and 3rd party dependencies possible.
So if the project compiles it can be run in a separate process in the module class path.

[itegebo]

To clarify, I meant a lexer created by JFlex using a separate (hand-made) *.flex file rather than one generated from Grammar-Kit's BNF.

[gregsh]

The point is that *.flex can contain arbitrary java code, states, imports...
It should be possible to hack and load the generated lexer as groovy script in the same VM but with rather unpredictable consequences.

[itegebo]

Naively, I would have thought it possible for Grammar-Kit to expose the requirements of any lexer that's going to work with it. Thus, if someone has a separately defined lexer then they could adapt it as appropriate.

Perhaps you could provide some guidance on how I might explore this in code. If I had to guess, I'd start by extracting an interface from LivePreviewLexer and then I'd implement a class that delegates to a JFlex lexer.

[gregsh]

"JFlex lexer" can stand for 3 different entities: a flex file, a java file, a class file. How exactly the delegation to a JFlex lexer could look like in LivePreviewLexer.java ?

The requirement for a lexer is already in IntelliJ Platform, it is Lexer interface (com.intellij.lexer).

[itegebo]

"JFlex lexer" can stand for 3 different entities: a flex file, a java file, a class file. How exactly the delegation to a JFlex lexer could look like in LivePreviewLexer.java ?

I don't know, that's why I was asking for guidance. I'd assume that one would prefer to develop a lexer via the flex file, and then be able to configure Grammar-Kit/LivePreview to use the generated java class; I'm confused about why you'd make a distinction between the java source and the compiled class file - are you asking about some more exotic reflection/bytecode/aspect approach?

Anyway, I didn't expect this be a confusing feature request because on https://github.com/JetBrains/Grammar-Kit, I read:

"Lexer can be provided separately or one can use the generated *.flex file as a base."

Thus, I assumed it would be reasonable to ask that LivePreview be usable with such a provided lexer.

[clojj]

Any news on this ?
Is back-porting an external lexer into BNF the only workaround ?

live-preview is a mighty tool, and it'd be unfortunate to not be using it becaus of this limitation