Python parser generator for R grammar

[TDiazT]

I wanted to use the R grammar (https://github.com/antlr/grammars-v4/tree/master/r), so I changed the actions (in RFilter.g4) to target Python but the generated parser has some errors which I had to change manually. I'm not sure if maybe I set something wrong in the actions or if it's something with the generation.

My Rfilter.g4 version (I omitted some sections which remained the same) :

/*

parser grammar RFilter;

options { tokenVocab=R; }

@members {
    self.curlies = 0
}

// TODO: MAKE THIS GET ONE COMMAND ONLY
stream : (elem|NL|';')* EOF ;

eat :   (NL {$NL.channel = Token.HIDDEN_CHANNEL})+ ;

elem: 
    [...]
    |   '{' eat? {self.curlies += 1} (elem|NL|';')* {self.curlies -= 1 } '}'
    [...]
    |   'else'
        {
tok = self._input.LT(-2)
if self.curlies > 0 and tok.type == self.NL:
   tok.channel = Token.HIDDEN_CHANNEL
    }
    ;

[...]

In particular:

1. eat : (NL {$NL.channel = Token.HIDDEN_CHANNEL})+ ; becomes (0 if (localctx._NL is None else localctx._NL.channel) = Token.HIDDEN_CHANNEL, which I manually changed to localctx._NL.channel = Token.HIDDEN_CHANNEL.
2. The action in the else rule adds extra indentation. I mean, currently as shown above it works, but otherwise if adding a bit more indentation (as I did when changing from the Java version to this one), then it adds more indentation than needed. Not sure if this is expected or not...
3. There is an error in the importing of a module (it still works but thought I'd mention it). from typing.io import TextIO can't find 'io' in 'typing.pyi'.

Target: Python 3
Antlr: 4.7.1

[ericvergnaud]

Re 1: This is an issue in the parser generation template. I just fixed it Re 2: ANTLR does not make guesses about indentation, so best is to move the body in a method and just call it from the grammar Re 3: this probably relates to your setup. On my box, using Python 3.5.1, there is no error.

…

Le 2 janv. 2018 à 22:53, Tomás Díaz ***@***.***> a écrit : I wanted to use the R grammar (https://github.com/antlr/grammars-v4/tree/master/r <https://github.com/antlr/grammars-v4/tree/master/r>), so I changed the actions (in RFilter.g4 <https://github.com/antlr/grammars-v4/blob/master/r/RFilter.g4>) to target Python but the generated parser has some errors which I had to change manually. I'm not sure if maybe I set something wrong in the actions or if it's something with the generation. My Rfilter.g4 version (I omitted some sections which remained the same) : /* parser grammar RFilter; options { tokenVocab=R; } @members { self.curlies = 0 } // TODO: MAKE THIS GET ONE COMMAND ONLY stream : (elem|NL|';')* EOF ; eat : (NL {$NL.channel = Token.HIDDEN_CHANNEL})+ ; elem: [...] | '{' eat? {self.curlies += 1} (elem|NL|';')* {self.curlies -= 1 } '}' [...] | 'else' { tok = self._input.LT(-2) if self.curlies > 0 and tok.type == self.NL: tok.channel = Token.HIDDEN_CHANNEL } ; [...] In particular: eat : (NL {$NL.channel = Token.HIDDEN_CHANNEL})+ ; becomes (0 if (localctx._NL is None else localctx._NL.channel) = Token.HIDDEN_CHANNEL, which I manually changed to localctx._NL.channel = Token.HIDDEN_CHANNEL. The action in the else rule adds extra indentation. I mean, currently as shown above it works, but otherwise if adding a bit more indentation (as I did when changing from the Java version to this one), then it adds more indentation than needed. Not sure if this is expected or not... There is an error in the importing of a module (it still works but thought I'd mention it). from typing.io import TextIO can't find 'io' in 'typing.pyi'. Target: Python 3 Antlr: 4.7.1 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2193>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADLYJN2HJfNAPrzRswcWHg1Gk-Dlc_B6ks5tGkLSgaJpZM4RQsVO>.

[TDiazT]

1. Thanks!
2. Ohh okay, sounds reasonable :p
3. Interesting... I'll check on that then (I have Python 3.5.0)

[kenahoo]

I have the same error about from typing.io import TextIO can't find 'io' in 'typing.pyi'. I have to go into the generated parser file and change it to from typing import TextIO, then everything else goes as expected. I'm running Python 3.6.2.

[ericvergnaud]

That would be a bug in the R grammar not in Antlr Envoyé de mon iPhone

…

Le 2 mars 2018 à 00:07, Ken Williams ***@***.***> a écrit : I have the same error about from typing.io import TextIO can't find 'io' in 'typing.pyi'. I have to go into the generated parser file and change it to from typing import TextIO, then everything else goes as expected. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[kenahoo]

@ericvergnaud - that can't be the case, I'm not using the R grammar, I'm writing my own grammar and using the Python3 output.

[ericvergnaud]

Can you explore https://docs.python.org/3/library/typing.html
?
Seems to state that typing.io/TextIO is part of the runtime

[brizjin]

I have the same error: typing.io import TextIO can't find 'io' in 'typing.pyi'