codeop: inconsistent handling of invalid escapes

[kevinushey]

Bug report

It seems like the codeop module is inconsistent with regards to handling of invalid escapes. For example, '\(' is accepted at the top level, but apparently not within indented blocks.

Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import codeop
>>> codeop.compile_command("'\('")
<code object <module> at 0x00000244A5BE6290, file "<input>", line 1>
>>> codeop.compile_command("while False:\n\t'\('")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\kevin\AppData\Local\Programs\Python\Python310\lib\codeop.py", line 107, in compile_command
    return _maybe_compile(_compile, source, filename, symbol)
  File "C:\Users\kevin\AppData\Local\Programs\Python\Python310\lib\codeop.py", line 70, in _maybe_compile
    compiler(source + "\n", filename, symbol)
  File "C:\Users\kevin\AppData\Local\Programs\Python\Python310\lib\codeop.py", line 86, in _compile
    return compile(source, filename, symbol, PyCF_DONT_IMPLY_DEDENT | PyCF_ALLOW_INCOMPLETE_INPUT)
  File "<input>", line 2
    '\('
    ^^^^
SyntaxError: invalid escape sequence '\('

This can also be seen when submitting code via IDLE; e.g.

However, such code is accepted by a "plain" Python terminal:

Given the discrepancy, it seems like invalid escapes should perhaps be accepted by codeop here?

Your environment

Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32

[serhiy-storchaka]

It is because the codeop turns SyntaxWarning into SyntaxError when call compile() second time.

When run Python with the -Wa option you see warnings:

>>> import codeop
>>> codeop.compile_command("while False:\n\t'\('")
<stdin>:1: DeprecationWarning: invalid escape sequence '\('
<input>:2: DeprecationWarning: invalid escape sequence '\('
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/codeop.py", line 107, in compile_command
    return _maybe_compile(_compile, source, filename, symbol)
  File "/usr/lib/python3.10/codeop.py", line 70, in _maybe_compile
    compiler(source + "\n", filename, symbol)
  File "/usr/lib/python3.10/codeop.py", line 86, in _compile
    return compile(source, filename, symbol, PyCF_DONT_IMPLY_DEDENT | PyCF_ALLOW_INCOMPLETE_INPUT)
  File "<input>", line 2
    '\('
    ^^^^
SyntaxError: invalid escape sequence '\('

The first warning (starting with ":1:") is emitted by the REPL. The second one (starting with ":2:"). And finally you get an error when compile() is called after warnings.simplefilter("error"). Without that filter you would get a doubled warning.

Duplicated warnings were silenced in bpo-40807, and turned into errors in bpo-41520. @vstinner

[vstinner]

I'm no longer sure if codeop should display SyntaxError or raise an exception by default (see issue #85692).

But I dislike the fact that warnings depends if the first or the second compile() function succeeded. We can to treat warnings the same way for all compile() calls.

[vstinner]

@serhiy-storchaka: Is this issue fixed or not? It's still open.

Issue #97562 can be closed as a duplicate if I understood correctly.

[serhiy-storchaka]

Yes, this issue has been fixed.

[kevinushey]

Should I see the fix in 3.11.0rc2 / 3.10.7?

[serhiy-storchaka]

No, this change was not included in 3.11rc2 and 3.10.7. You have to wait the next bugfix releases.