Contextual tokens

[ericvergnaud]

Some lexer patterns change meaning depending on the context.

As an example, in C# value is a keyword in a property setter, and un mundane identifier elsewhere.

In typescript, const is a keyword, but is also a valid property identifier:

type A = { const: string }
const a = { const: 12};

At present, grammar developers need to go though painful gymnastics in order to support lexer patterns that have a different meaning depending on the context.

Disclaimer: implementing this requires revisiting significantly the current antlr architecture.

[ericvergnaud]

A minimal improvement might be achieved using Token value comparators

[KvanTTT]

Disclaimer: implementing this requires revisiting significantly the current antlr architecture.

I'm not sure it's the significant revisiting ANTLR architecture, I suspect token value can be treated just as token on the parser side. I'll try to experiment with it the original repository (separated branch) at first.

[ftomassetti]

I think there are a lot of languages which (unfortunately) use soft keywords, and it can be tricky to support them, so I think this would be very useful

[KvanTTT]

I can't say it's "unfortunately" because it's the best solution of adding new syntax without loosing backward compatibility.

[KvanTTT]

FYI, I'm working on prototype (the best activity for Christmas vacation). I hope I'll publish first steps at the beginning of the January, it looks promising.

[KvanTTT]

Initial prototype is ready! KvanTTT/antlr4#8

[ericvergnaud]

I'm not convinced that the soft token feature addresses all contextual token scenarios, but it certainly addresses most of them in a very simple way. So I'd suggest closing this topic for now.

[KvanTTT]

I agree, the topics seem similar.