Parse error with multi-line expression wrapped in parenthesis without line continuation

[ye]

This was first discovered and reported in flask_ext_migrate.

I believe this is the right place to report and get it fixed. See example below and the original issue I filed above for more details. pallets/flask-ext-migrate#24

$ less test.py 
      1 class A(object):
      2     prop2 = 'This should work OK!'
      3 
      4 class B(object):
      5     prop1 = {'prop': A()}
      6 
      7 x = (B().prop1
      8     ['prop'].prop2)
      9 print(x)
$ python3 test.py 
This should work OK!
$ flask_ext_migrate test.py 
Traceback (most recent call last):
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/baron/baron.py", line 20, in _parse
    return parser(tokens)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/baron/grammator.py", line 695, in parse
    return parser.parse(iter(tokens))
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/baron/parser.py", line 168, in parse
    raise ParsingError(debug_output)
baron.parser.ParsingError: Error, got an unexpected token PRINT here:

   2     prop2 = 'This should work OK!'
   3 
   4 class B(object):
   5     prop1 = {'prop': A()}
   6 
   7 x = (B().prop1    
   8 ['prop'].prop2)
   9 print<---- here

The token PRINT should be one of those: AMPER, AMPER_EQUAL, AND, AS, BACKQUOTE, BINARY, BINARY_RAW_STRING, BINARY_STRING, CIRCUMFLEX, CIRCUMFLEX_EQUAL, COLON, COMMA, COMMENT, COMPLEX, DOT, DOUBLE_SLASH, DOUBLE_SLASH_EQUAL, DOUBLE_STAR, DOUBLE_STAR_EQUAL, ELSE, ENDL, EQUAL, EQUAL_EQUAL, FLOAT, FLOAT_EXPONANT, FLOAT_EXPONANT_COMPLEX, FOR, GREATER, GREATER_EQUAL, HEXA, IF, IN, INT, IS, LAMBDA, LEFT_BRACKET, LEFT_PARENTHESIS, LEFT_SHIFT, LEFT_SHIFT_EQUAL, LEFT_SQUARE_BRACKET, LESS, LESS_EQUAL, LONG, MINUS, MINUS_EQUAL, NAME, NOT, NOT_EQUAL, OCTA, OR, PERCENT, PERCENT_EQUAL, PLUS, PLUS_EQUAL, RAW_STRING, RIGHT_BRACKET, RIGHT_PARENTHESIS, RIGHT_SHIFT, RIGHT_SHIFT_EQUAL, RIGHT_SQUARE_BRACKET, SEMICOLON, SLASH, SLASH_EQUAL, STAR, STAR_EQUAL, STRING, TILDE, UNICODE_RAW_STRING, UNICODE_STRING, VBAR, VBAR_EQUAL

It is not normal that you see this error, it means that Baron has failed to parse valid Python code. It would be kind if you can extract the snippet of your code that makes Baron fail and open a bug here: https://github.com/Psycojoker/baron/issues

Sorry for the inconvenience.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ye/.envs/py35/bin/flask_ext_migrate", line 11, in <module>
    sys.exit(startup())
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/flask_ext_migrate/startup.py", line 21, in startup
    fix(filepath)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/flask_ext_migrate/__init__.py", line 140, in fix
    ast = read_source(input_file)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/flask_ext_migrate/__init__.py", line 7, in read_source
    red = RedBaron(source_code.read())
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/redbaron/redbaron.py", line 36, in __init__
    self.node_list = base_nodes.NodeList.from_fst(baron.parse(source_code), parent=self, on_attribute="root")
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/baron/baron.py", line 57, in parse
    to_return = _parse(tokens, print_function)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/baron/baron.py", line 26, in _parse
    raise e
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/baron/baron.py", line 24, in _parse
    return parser(tokens)
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/baron/grammator.py", line 695, in parse
    return parser.parse(iter(tokens))
  File "/Users/ye/.envs/py35/lib/python3.5/site-packages/baron/parser.py", line 168, in parse
    raise ParsingError(debug_output)
baron.parser.ParsingError: Error, got an unexpected token NAME here:

   2     prop2 = 'This should work OK!'
   3 
   4 class B(object):
   5     prop1 = {'prop': A()}
   6 
   7 x = (B().prop1    
   8 ['prop'].prop2)
   9 print<---- here

The token NAME should be one of those: AMPER_EQUAL, CIRCUMFLEX_EQUAL, COLON, COMMA, COMMENT, DOUBLE_SLASH_EQUAL, DOUBLE_STAR_EQUAL, ENDL, EQUAL, LEFT_SHIFT_EQUAL, MINUS_EQUAL, PERCENT_EQUAL, PLUS_EQUAL, RIGHT_PARENTHESIS, RIGHT_SHIFT_EQUAL, SEMICOLON, SLASH_EQUAL, STAR_EQUAL, VBAR_EQUAL

It is not normal that you see this error, it means that Baron has failed to parse valid Python code. It would be kind if you can extract the snippet of your code that makes Baron fail and open a bug here: https://github.com/Psycojoker/baron/issues

Sorry for the inconvenience.

$ cat test.py | sed 's/\.prop1/.prop1\\/' > test_with_slash.py
$ flask_ext_migrate test_with_slash.py
$ python3 test_with_slash.py
This should work OK!
$ diff test_with_slash.py test.py 
7c7
< x = (B().prop1\
---
> x = (B().prop1

Update: added the exact exception print out from the original ticket to clarify the issue.

[bootandy]

This seems to work in baron:
baron.dumps( baron.parse("\nx=(1+\n2)\n") )

This doesn't:
baron.dumps( baron.parse("x = (dictionary\n['prop'])") )

So it looks like baron can't handle the dictionary spanning new lines. It seems to handle function calls on new lines though.

This works:
baron.dumps( baron.parse("x = (clazz\n.method)") )

[ye]

@bootandy that's different. What you had there is a expression with split lines wrapped in parenthesis. What I encountered was accessing dictionary values via key but in a different line, python doesn't complaint but baron does.

[bootandy]

Yes I agree, I was trying to pin down the cause of this.

Accessing a dictionary value with the key on the next line breaks baron.

The problem seems to be in baron/parser.py BaronLRParser.parse() function which looks like it was copy pasted from the rply library's parser.py. Something in self.lr_table.lr_action causes the 't' value to not end up as 0. I don't currently know what the self.lr_table dictionary is doing.