autopep8 formatOnSave breaks f-strings on long lines, causing "unterminated string literal" or "EOL while scanning string literal"

[afarah1]

Type: Bug

Behaviour

Expected vs. Actual

Expected: Upon saving the file, f-strings on long lines are not broken at string literals, or have \ added to the EOL.

Actual: Upon saving the file, f-strings on long lines get incorrect line breaks, causing "unterminated string literal" or "EOL while scanning string literal". If one attempts to fix the line break with a \, autoFormat removes that upon save. It's impossible to have a file with an f-string on a long line, autoFormat has to be disabled.

Steps to reproduce:

1. Enable formatOnSave with ms-python.autopep8. On settings.json:

    "[python]": {
        "editor.defaultFormatter": "ms-python.autopep8",
        "editor.formatOnSave": true,
    },

2. Edit and save the following file, triggering auto-format:

mystr = 'abc'
path = f'aVeryLargeFStringLineThatBreak{mystr}aVeryLargeFStringLineThatBreaksWithaPep8'

3. The file gets formated to:

mystr = 'abc'
path = f'aVeryLargeFStringLineThatBreak{
    mystr}aVeryLargeFStringLineThatBreaksWithaPep8'

4. Attempt to fix with the following and save again:

mystr = 'abc'
path = f'aVeryLargeFStringLineThatBreak{\
    mystr}aVeryLargeFStringLineThatBreaksWithaPep8'

5. The modifications are overwritten by autoformat.
6. Run the file, getting:

$ python3 test.py
  File "/Users/me/test.py", line 2
    path = f'aVeryLargeFStringLineThatBreak{
                                            ^
SyntaxError: EOL while scanning string literal

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.0
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Poetry
• Value of the python.languageServer setting: Default

Extension version: 2023.8.0
VS Code version: Code 1.78.0 (252e5463d60e63238250799aef7375787f68b4ee, 2023-05-03T20:15:29.774Z)
OS version: Darwin arm64 23.1.0
Modes:
Sandboxed: No

[karthiknadig]

Can you test this by:

cat /Users/me/test.py | python -m autopep8 -

This seems like autopep8 itself is returning this. See issue on autopep8 with f-strings: hhatto/autopep8#712

[afarah1]

You're correct, I didn't know about that issue. Please disregard my report.

[afarah1]

Actually, strike that... The file was already formatted in vscode when I tested directly with autopep8... Sorry for the confusion.

It seems like autopep8 doesn't break the line at all:

$ cat test.py 
mystr = 'abc'
path = f'aVeryLargeFStringLineThatBreak{mystr}aVeryLargeFStringLineThatBreaksWithaPep8'

$ cat test.py | python3 -m autopep8 -
mystr = 'abc'
path = f'aVeryLargeFStringLineThatBreak{mystr}aVeryLargeFStringLineThatBreaksWithaPep8'

$ cat test2.py 
mystr = 'abc'
path = f'aVeryLargeFStringLineThatBreak{mystr}aVeryLargeFStringLineThatBreaksWithaPep8'
long_breakable_line = 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str'

$ cat test2.py | python3 -m autopep8 -
mystr = 'abc'
path = f'aVeryLargeFStringLineThatBreak{mystr}aVeryLargeFStringLineThatBreaksWithaPep8'
long_breakable_line = 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + 'str' + \
    'str' + 'str' + 'str' + 'str' + 'str' + 'str' + \
    'str' + 'str' + 'str' + 'str' + 'str' + 'str'

[karthiknadig]

This is autopep8 issue, here is the output from autopep8 that the extension sees:

This is what we return back to the editor. As you can see there is a \n in the result from autopep8. Note that autopep8 seems to have different results based on python versions and versions of packages it depends on. So, when you ran it from the terminal it might have used a version where it worked. In my case it repro-ed on python 3.12. which is where most people are seeing the issue.