Oliver claims expectedTokens() doesn't work on these examples

[parrt]

http://code.google.com/p/antlr4-ast-visitor-sandbox/source/browse/trunk/test/ParserTest.java

[sharwell]

Is there an ANTLR bug involved? This appears to be a unit test for another application which uses ANTLR but isn't testing the ANTLR library itself. The expectedTokens test doesn't contain any assertions, so it's unclear what the intent actually is.

[DJCordhose]

Tried to clarify the code a bit, please re-check. The test concerned is expectedTokens().

Unfortunately, the stuff lies in the output to stdout. I spit out what I have parsed so far and then what tokens are to be expected using Parser#getExpectedTokens(). All I see as the expected token is the token I just parsed.

Maybe I just do not grok how this is supposed to work.

[sharwell]

Parser.getExpectedTokens() uses the current state of the current context (parser._ctx.s) determine the set of expected tokens. As soon as the parser changes state, the set of expected tokens changes. Right before a call to match (which produces the call to visitTerminal), the state is set to the source state of the non-epsilon transition matching the symbol. If you call getExpectedTokens() from within visitTerminal of a parser listener, you'll get the set of tokens for the single transition that matched the terminal (which is exactly what you saw).

[DJCordhose]

Hey, Sam, thanks for being patient with me. I actually mixed up stuff here and what you say makes sense. For I actually want to do: Finding out the set of valid tokens that can follow on a given position inside a buffer. Like for an editor with auto-completion. Any pointers?

[sharwell]

In ANTLRWorks 2 (currently closed source due to licensing details I'm working on), to get this information I needed to extend ParserATNSimulator, CommonToken, ANTLRErrorStrategy, TokenSource, and use a special RuntimeException to bail out of the internals while preserving the interpreter state, as well as controlling the beast with a special controller. In addition, I had to add the -Xforce-atn option to the tool (and use it for any parser where I intend to derive this information).

It's gruesome, but what's possible here is without a doubt the most powerful solution to this problem I've ever seen. Proof of this should be clear in what I was able to accomplish in GoWorks in under 1 month.