bpo-40612: Fix SyntaxError edge cases in traceback formatting

[gvanrossum]

This fixes both the traceback.py module and the C code for formatting syntax errors (in Python/pythonrun.c). They now both consistently do the following:

• Suppress caret if it points left of text
• Allow caret pointing just past end of line
• If caret points past end of line, clip to just past end of line

The syntax error formatting code in traceback.py was mostly rewritten; small, subtle changes were applied to the C code in pythonrun.c.

There's still a difference when the text contains embedded newlines. Neither handles these very well, and I don't think the case occurs in practice.

https://bugs.python.org/issue40612

Automerge-Triggered-By: @gvanrossum

[pablogsal]

LGTM

I left some suggestions for PEP7

[ammaraskar]

Just fyi this code right here is the one that I mentioned in we-like-parsers#121 (comment)

Whereby a SyntaxError with an offset relative to the start of the file will end up pointing to the right place. I'm tempted to say we should eventually just remove it since the new parser will always provide line-relative offsets.

[gvanrossum]

Hm... Why would the offset in the SyntaxError object ever end up being file-relative? Do you know of any code that produces such SyntaxErrors?

[ammaraskar]

In the old parser it was this code:

cpython/Parser/parsetok.c

Lines 425 to 431 in 4a12d12

	err_ret->offset = col_offset != -1 ? col_offset + 1 : ((int)(tok->cur - tok->buf));
	len = tok->inp - tok->buf;
	err_ret->text = (char *) PyObject_MALLOC(len + 1);
	if (err_ret->text != NULL) {
	if (len > 0)
	strncpy(err_ret->text, tok->buf, len);
	err_ret->text[len] = '\0';

tok->cur is the current read index of the tokenizer and tok->buf is the start of the file. Also see how it copies the entire file up till the error into the SyntaxError.text field.

Try this out with the old parser as a quick example:

code = """\
a = \\
    \\
    \\?"""

try:
  compile(code, '<stdin>', 'exec')
except SyntaxError as e:
  print(e)
  print(e.lineno, e.offset)

[gvanrossum]

Oh, you mean the opposite of file. :-) It occurs when it's not read from a file.

Also it doesn't occur all the time -- perhaps only when there's a continuation line? E.g. here all is good:

>>> try: compile("def f():\n 1+\n", "", "exec")
... except SyntaxError as e: e
... 
SyntaxError('invalid syntax', ('', 2, 4, ' 1+\n'))
>>> 

[gvanrossum]

FWIW, your example does show up weird in the old parser:

>>> code = """\
a = \\
    \\
    \\?"""
... ... ... >>> code
'a = \\\n    \\\n    \\?'
>>> try: compile(code, "", "exec")
... except SyntaxError as e: e
... 
SyntaxError('unexpected character after line continuation character', ('', 3, 19, 'a = \\\n    \\\n    \\?\n'))
>>> 

The new parser seems to solve the dilemma by suppressing the source text (also the offset is set to zero, meaning unknown):

>>> code = """\
a = \\
    \\
    \\?"""
... ... ... >>> 
>>> try: compile(code, "", "exec")
... except Exception as e: e; e.lineno, e.offset, e.text
... 
SyntaxError('unexpected character after line continuation character')
(3, 0, None)
>>> 

[ammaraskar]

Aah yes, it seems like it's just an issue with line continuations. I thought I had another example but it must have been a misunderstanding because tok->buf gets advanced with newlines.

So I guess this code here in pythonrun.c is just for this one case?

[gvanrossum]

Okay, so I have a serious question here. Is there a regression due to this PR for people using -X oldparser? I have tried to research this and cannot find a regression. The C code in pythonrun.c still skips through newlines. The traceback.py code doesn't, but it never did.

Here is what I did for research. First Python 3.8:

>>> import traceback; traceback.print_exception(None, SyntaxError("msg", ("f.py", 3, 10, "aaa\nbbb\nccc\n")), None)
  File "f.py", line 3
    aaa
bbb
ccc
       
   
 ^
SyntaxError: msg
>>> raise SyntaxError("msg", ("f.py", 3, 10, "aaa\nbbb\nccc\n"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "f.py", line 3
    ccc
     ^
SyntaxError: msg
>>> 

Then the master branch:

>>> import traceback; traceback.print_exception(None, SyntaxError("msg", ("f.py", 3, 10, "aaa\nbbb\nccc\n")), None)
  File "f.py", line 3
    aaa
bbb
ccc
       
   
 ^
SyntaxError: msg
>>> raise SyntaxError("msg", ("f.py", 3, 10, "aaa\nbbb\nccc\n"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "f.py", line 3
    ccc
     ^
SyntaxError: msg
>>> 

To me that looks like in both versions, the C formatter (invoked by raise) does the right thing, while traceback.py messes up the output.

[miss-islington]

@gvanrossum: Status check is done, and it's a failure ❌ .

[miss-islington]

@gvanrossum: Status check is done, and it's a success ✅ .