Formatting produces invalid code on 3.13.4

[MeGaGiGaGon]

Summary

With the release of 3.13.4, python/cpython#129958 was fixed, which now makes formatting produce invalid code on 3.13.4
playground link
Input:

f"{
11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111:}"

Output:

f"{
    11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111:
}"

The input is valid code on 3.13.4, the output is not

$ uv run --python 3.13.4 python -c $'f"{\n11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111:}"'

$ uv run --python 3.13.4 python -c $'f"{\n    11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111:\n}"'
  File "<string>", line 2
    1111111111111111111111111111111111111111111111111111111111111111111111111111
1111111111111:

              ^
SyntaxError: f-string: newlines are not allowed in format specifiers for single
quoted f-strings

Version

No response

[ntBre]

It looks like the PR linked in that issue modifed the CPython lexer, so we may need to update that as well.

[ntBre]

Thanks for the report!

[MeGaGiGaGon]

(@AlexWaygood I think the label should be python313 instead of python314, since the affected version is 3.13.4 (Yes it's confusing, I also typed 3.14 on accident a bunch while writing the issue))

[AlexWaygood]

Oh wow, sorry, I misread!

[MichaReiser]

Hmm, it's unfortunate that this changes in a patch release.

But yes. We'll have to update the formatter. We may want to wait with updating the parser until the formatter is rolled out so that the formatter can fix incorrectly formatted code.

[MichaReiser]

I'm still trying to understand the extend of the bug here. Reading through python/cpython#110259 it seems that format strings followed by a newline are fine.

But it seems that python/cpython#129958 now goes further and disallows any format spec that's followed by a newline, even if, what comes after, is only whitespace. I think this is very unfortunate for formatters. I can understand the motivation in python/cpython#110259 but the motivation for disallowing any newline followed only by whitespace isn't clear to me. It seems an unnecessary restriction IMO.

Either way. It seems we can no longer split f-string parts with format specifiers 🤷

[MeGaGiGaGon]

At least the change itself makes sense to me, since as I understand it the allowing newlines was causing the lexer to reset to normal mode in the format spec, which causes issues down the line. It also behaves very strangely, ie using

class A:
    def __format__(self, *args):
        print(args)
        return str(self)

Under 3.12 this

f"{A(): }"

outputs (' ',)
while this

f"{A():
 }"

outputs ('',)

The argument also makes a sort of sense to me, since the format spec is basically just a string without quotes that gets passed to __format__, so the logic being used is that the unquoted string acts like the string it's in, and since single quoted strings can't have newlines then their format specs can't either.

[MichaReiser]

This might be easier to fix than expected. It seems we have already implemented the same behavior for triple quoted strings. We just need to apply it not to all strings

ruff/crates/ruff_python_formatter/src/other/interpolated_string_element.rs

Lines 98 to 119 in c9dff5c

	self.parent_context.layout().is_multiline()
	// For a triple-quoted f-string, the element can't be formatted into multiline if it
	// has a format specifier because otherwise the newline would be treated as part of the
	// format specifier.
	//
	// Given the following f-string:
	// ```python
	// f"""aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbb ccccccccccc {variable:.3f} ddddddddddddddd eeeeeeee"""
	// ```
	//
	// We can't format it as:
	// ```python
	// f"""aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbb ccccccccccc {
	// variable:.3f
	// } ddddddddddddddd eeeeeeee"""
	// ```
	//
	// Here, the format specifier string would become ".3f\n", which is not what we want.
	// But, if the original source code already contained a newline, they'll be preserved.
	//
	// The Python version is irrelevant in this case.
	&& !(self.parent_context.flags().is_triple_quoted() && self.has_format_spec)

[MichaReiser]

And we probably need to revert #7787

What I understand so far is that it's fine for a format-spec to contain line breaks inside of interpolated elements. It's just not allowed inside of string elements

[MeGaGiGaGon]

After some messing around, I was able to come up with this

rf"{1:\
\}"

formats to

rf"{
    1:\
\
}"

So even without the changes in 3.13.4 it was still technically possible to have behavior change as a result of the reformatting

>>> rf"{1:\
... \}"
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    rf"{1:\
       ^^^^
    \}"
    ^^
ValueError: Unknown format code '\' for object of type 'int'
>>> rf"{
...     1:\
... \
... }"
'1'