Stop overriding locations for expressions within f-strings

[harupy]

RustPython/RustPython#4384 fixed the location of FormattedValue. Now, expressions within f-strings should have the correct locations. Here's a quick demo.

Screen.Recording.2022-12-31.at.13.03.53.mov

> cargo run -- --show-source --no-cache --select C,F a.py
a.py:1:4: C408 Unnecessary `list` call (rewrite as a literal)
  |
1 | f"{list()}"
  |    ^^^^^^ C408
  |
a.py:2:6: F821 Undefined name `x`
  |
2 | f"{  x   }"
  |      ^ F821
  |
a.py:4:6: C408 Unnecessary `dict` call (rewrite as a literal)
  |
4 | f"""{dict()}"""
  |      ^^^^^^ C408
  |
a.py:6:5: F821 Undefined name `x`
  |
6 | rf"{x}"
  |     ^ F821
  |
a.py:8:7: F821 Undefined name `x`
  |
8 | rf"""{x}"""
  |       ^ F821
  |
a.py:11:2: F821 Undefined name `x`
   |
11 | {x}
   |  ^ F821
   |
a.py:14:6: F821 Undefined name `x`
   |
14 | f"\n{x}"
   |      ^ F821
   |
a.py:16:10: F821 Undefined name `x`
   |
16 | f"\u1234{x}"
   |          ^ F821
   |
a.py:18:6: F821 Undefined name `x`
   |
18 | f"\\{x}"
   |      ^ F821
   |
a.py:20:14: F821 Undefined name `x`
   |
20 | f"\\\\\\\\\\{x}"
   |              ^ F821
   |
a.py:23:2: F821 Undefined name `x`
   |
23 | {x}"
   |  ^ F821
   |
Found 11 error(s).
2 potentially fixable with the --fix option.

[charliermarsh]

What?!?!? Noooo way!

[charliermarsh]

How the heck did you do this? That looks like a really impressive change.

[harupy]

What I did was the following:

1. Use the right offset when parsing expressions within f-strings.
2. Fix the lexer to preserve the original string to compute the right offset.

[harupy]

Tok::Bytes has been removed. The new lexer only uses Tok::String.

Old lexer:

# Implicitly concatenated string
b"" ""  # Tok::Bytes, Tok::String

New lexer:

b"" ""   # Tok::String (with kind=Bytes), Tok::String (kind=String)

[harupy]

Duplicate function arguments like def f(a, a): pass now yield SyntaxError ( RustPython/RustPython#4380).

[harupy]

[harupy]

Maybe we should remove F831 because it's never raised.

[harupy]

or we can just keep it as it's not harmful.

[charliermarsh]

Ah yeah, we can remove in a separate PR.

(Is this true in all Python versions 3.7+? If you know.)

[harupy]

@charliermarsh

> docker run python:3.7 python -c "def f(a, a): pass"
  File "<string>", line 1
SyntaxError: duplicate argument 'a' in function definition

> docker run python:3.8 python -c "def f(a, a): pass"
  File "<string>", line 1
SyntaxError: duplicate argument 'a' in function definition

> docker run python:3.9 python -c "def f(a, a): pass"
  File "<string>", line 1
SyntaxError: duplicate argument 'a' in function definition

> docker run python:3.10 python -c "def f(a, a): pass"
  File "<string>", line 1
SyntaxError: duplicate argument 'a' in function definition

[harupy]

@charliermarsh

we can remove in a separate PR.

I'll file a PR.

[charliermarsh]

This also seems to be a little bit faster...

[charliermarsh]

Honestly this makes me so happy. This is great. Thank you.