3.11 exec raises SystemError instead of SyntaxError on char after line continuation

[pewscorner]

BPO	45738
Nosy	@aroberge, @Carreau, @lysnikolaou, @Zac-HD, @pablogsal, @miss-islington, @isidentical, @pewscorner
PRs	bpo-45738: Fix computation of error location for invalid continuation characters in the parser #29550 [3.10] bpo-45738: Fix computation of error location for invalid continuation (GH-29550) #29551 [3.9] bpo-45738: Fix computation of error location for invalid continuation (GH-29550) #29552

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-11-14.13:59:48.259>
created_at = <Date 2021-11-06.13:10:51.997>
labels = ['type-bug', '3.9', '3.11']
title = '3.11 exec raises SystemError instead of SyntaxError on char after line continuation'
updated_at = <Date 2021-11-14.18:39:21.628>
user = 'https://github.com/pewscorner'

bugs.python.org fields:

activity = <Date 2021-11-14.18:39:21.628>
actor = 'mbussonn'
assignee = 'none'
closed = True
closed_date = <Date 2021-11-14.13:59:48.259>
closer = 'pablogsal'
components = []
creation = <Date 2021-11-06.13:10:51.997>
creator = 'pewscorner'
dependencies = []
files = []
hgrepos = []
issue_num = 45738
keywords = ['patch']
message_count = 12.0
messages = ['405860', '406188', '406223', '406224', '406235', '406236', '406301', '406303', '406304', '406305', '406322', '406329']
nosy_count = 8.0
nosy_names = ['aroberge', 'mbussonn', 'lys.nikolaou', 'Zac Hatfield-Dodds', 'pablogsal', 'miss-islington', 'BTaskaya', 'pewscorner']
pr_nums = ['29550', '29551', '29552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45738'
versions = ['Python 3.9', 'Python 3.11']

[pewscorner]

When exec'ing code with an illegal character after the line continuation character '\', python 3.11.0a2 raises this strange exception:

SystemError: Negative size passed to PyUnicode_New

Here's an example where '#' is the illegal character:

  exec('1,\\#\n2')

I expected the following exception (which is raised by Python 3.10.0, and also by Python 3.11.0a2 when using eval instead of exec, or when the string content is put into its own file and run as a normal script):

SyntaxError: unexpected character after line continuation character

[Zac-HD]

It looks like this also affects Python 3.9.8, which makes me very suspicious of https://bugs.python.org/issue45494 as the probable cause.

See Zac-HD/hypothesmith#16 and psf/black#2592 (comment) for reports.

[Carreau]

Yep, this also break IPython command completions when users press tab in some places in multiline code.

88f4ec8 is the first offending commit for me in the 3.9 branch:

[3.9] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) (bpo-29071)

There are two errors that this commit fixes:

* The parser was not correctly computing the offset and the string
  source for E_LINECONT errors due to the incorrect usage of strtok().
* The parser was not correctly unwinding the call stack when a tokenizer
  exception happened in rules involving optionals ('?', [...]) as we
  always make them return valid results by using the comma operator. We
  need to check first if we don't have an error before continuing..
(cherry picked from commit a106343f632a99c8ebb0136fa140cf189b4a6a57)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

NOTE: unlike the cherry-picked original, this commit points at a crazy location
due to a bug in the tokenizer that required a big refactor in 3.10 to fix.
We are leaving as-is for 3.9.

[Carreau]

Note that with the following example, exec and CommandCompiler return a different error:

    from codeop import CommandCompiler

# identical errors
#exec('1,\\#\n2') SystemError
#CommandCompiler()('1,\\#\n2', symbol='exec') SystemError

    # one is syntax the other is System.
    exec('a \ \n')  # SyntaxError
    CommandCompiler()('a \ \n', symbol='exec') # SystemError

[pablogsal]

I will be able to take a look later in the weekend, so if someone is able to give it a go before, that would be great.

[Carreau]

I will be able to take a look later in the weekend

Please take some time for yourself, I added you as you were the original author of the patch to make you aware. I believe most projects can workaround that for now.

[pablogsal]

New changeset 25835c5 by Pablo Galindo Salgado in branch 'main':
bpo-45738: Fix computation of error location for invalid continuation (GH-29550)
25835c5

[miss-islington]

New changeset bf26a6d by Miss Islington (bot) in branch '3.10':
bpo-45738: Fix computation of error location for invalid continuation (GH-29550)
bf26a6d

[pablogsal]

This should be fixed by now, but please, check if everything is in order in your respective test suites and confirm here.

[pablogsal]

New changeset 142fcb4 by Pablo Galindo Salgado in branch '3.9':
bpo-45738: Fix computation of error location for invalid continuation characters in the parser (GH-29550) (GH-29552)
142fcb4

[pewscorner]

Seems to work. Thanks!