Improve parser to accept complicated block of code under interactive REPL

[hsfzxjy]

Feature or enhancement

Proposal:

Previously the Python REPL only accepts either a single compound statement or a line of simple statements. For example, we can either do:

>>> def f():
...     pass
... 
>>> 

or do:

>>> a = 1; a *= 3; print(a)
3
>>> 

However, the REPL doesn't accept multiple consecutive compound statements, or any simple statement followed soon after a compound statement:

>>> def f():
...     pass
... def g():
  File "<stdin>", line 3
    def g():
    ^^^
SyntaxError: invalid syntax
>>> def f():
...     pass
... a = 1
  File "<stdin>", line 3
    a = 1
    ^
SyntaxError: invalid syntax
>>> 

This will cause a lot of frustration when, for example, a user pastes snippet directly into the REPL and sees unexpected syntax error.

Hence, I propose to improve the parser, specifically the statement_newline rule to accept the abovementioned cases.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-111658: Improve parser to accept complicated block of code under interactive REPL #111659

[terryjreedy]

This is not a minor feature. Terminating multiple-logical-line compound statements with a blank line is a core feature of interactive mode. Properly formatted 'interactive' code with blank lines after all such statements and never within such statements can be pasted today in the REPL. First physical lines of such statements will be marked with a primary prompt and and subsequent lines with secondary prompts. Output for a statement will appear after the statement and before the next statement. Blank lines tell the interpreter to execute the statement and then send a new primary prompt to the terminal before it displays another statement.

[hsfzxjy]

@terryjreedy Thanks for your reply. My current implementation seems flawed and I would correct it later, but before this I would like to clarify the desired effect of this proposal:

1. All patterns that were accepted by previous versions will continue to be accepted. For example, compound statements with a blank line will still terminate the input.
2. In addition to those patterns, the proposal introduces some new patterns to allow statements appearing tightly after a compound statement, as described in the issue body. In my opinion, this should not break any existing code/doctests and will improve quality of life.

I was wondering whether this proposal would be accepted given the behavior described above?

[ericvsmith]

@pablogsal is working on the REPL in #111201, so this feature would have to build on that work, if it's not already included there (I haven't been following it very closely).

[pablogsal]

Yeah, the parser is not the correct way to fix this IMHO, the repl is. Currently, unfortunately both are tangled but after the work in #111201 they won't be so we can have a better place to include this behavior and make it easier to address the underlying problem which seems to allow users to paste code.