Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Python 3.12 compatibility in multi-line string scanning #496

Merged
merged 2 commits into from
May 30, 2023

Conversation

akaihola
Copy link
Owner

@akaihola akaihola commented May 30, 2023

Partially fixes #495.

Starting from Python 3.12, tokenizer emits an f-string and a triple-quoted f-string as a sequence of tokens starting with FSTRING_START and ending with FSTRING_END, instead of a single STRING token.

Note that as expected, we're still getting the 25 other test failures which will be fixed by #497:

FAILED action/tests/test_main.py::test_creates_virtualenv - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_installs_packages[run_main_env0-expect0] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_installs_packages[run_main_env1-expect1] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_installs_packages[run_main_env2-expect2] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_installs_packages[run_main_env3-expect3] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_installs_packages[run_main_env4-expect4] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_installs_packages[run_main_env5-expect5] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_installs_packages[run_main_env6-expect6] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_installs_packages[run_main_env7-expect7] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_wont_install_unknown_packages[foo] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_wont_install_unknown_packages[  foo  ] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_wont_install_unknown_packages[foo==2.0,bar] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_wont_install_unknown_packages[  foo>1.0  ,  bar  ] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_wont_install_unknown_packages[pylint,foo] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env0-expect0] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env1-expect1] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env2-expect2] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env3-expect3] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env4-expect4] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env5-expect5] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env6-expect6] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env7-expect7] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_runs_darker[env8-expect8] - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_error_if_pip_fails - ModuleNotFoundError: No module named 'pkg_resources'
FAILED action/tests/test_main.py::test_exits - ModuleNotFoundError: No module named 'pkg_resources'

@akaihola akaihola added the bug Something isn't working label May 30, 2023
@akaihola akaihola added this to the Darker 1.7.2 milestone May 30, 2023
@akaihola akaihola requested a review from my-tien May 30, 2023 06:14
@akaihola akaihola self-assigned this May 30, 2023
Copy link
Collaborator

@my-tien my-tien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine, just one question: Why is it necessary to test if token end > token start?

@akaihola
Copy link
Owner Author

Looks fine, just one question: Why is it necessary to test if token end > token start?

Because the function is looking for multi-line strings like these:

"""line 1
line 2"""

b'''line 1
line 2
'''

f"""multi-line
{x} f-string"""

Strings which fit on one line have the same line number for token start and end, for example:

"one-line string"
'one-line single-quoted string'
f"one-line {x} f-string"
b"one-line bytestring"

akaihola added 2 commits May 30, 2023 23:58
Starting from Python 3.12, `tokenizer` emits an f-string and a
triple-quoted f-string as a sequence of tokens starting with
`FSTRING_START` and ending with `FSTRING_END`, instead of a single
`STRING` token.
@akaihola akaihola merged commit e34932f into master May 30, 2023
@akaihola akaihola deleted the multiline-py312 branch May 30, 2023 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

Successfully merging this pull request may close these issues.

Multiple breakages with Python 3.12.dev
2 participants