NULL in lexical analysis of f-string

[chepner]

Documentation

This is somewhat related to #116580. There are two references to NULL in the description of f-strings that don't have a clear meaning.

format_spec       ::=  (literal_char | NULL | replacement_field)*
literal_char      ::=  <any code point except "{", "}" or NULL>

In both cases (but especially literal_char), it could refer to U+0000, but I'm unaware that a null character is allowed anywhere in Python source. At least, my attempt to inject one failed:

>>> print(ast.dump(ast.parse("f'{x:"'\x00'"}'"), indent=2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ast.py", line 52, in parse
    return compile(source, filename, mode, flags,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: source code string cannot contain null bytes

For format_spec, it could refer to an empty specification (f'{x:}'), but (literal_char | replacement_field)* would cover that just as well.

Linked PRs

• gh-116881: Remove erroneous or redundant grammar NULL #116885
• [3.12] gh-116881: Remove erroneous or redundant grammar NULL (GH-116885) #116951
• [3.11] gh-116881: Remove erroneous or redundant grammar NULL (GH-116885) #116952

[ericvsmith]

Could you point to where in the documentation you're referring? It seems to me that literal_char in your description already says NULL is excluded.

And is there any practical problem this causes?

[terryjreedy]

format_spec       ::=  (literal_char | NULL | replacement_field)*
literal_char      ::=  <any code point except "{", "}" or NULL>

are from string literals

In the literal_char token description, NULL has the normal meaning of \0' (\x00'). In the format_spec grammar production, NULL is wrong if taken in the same sense and is confusing since it must be interpreted to mean , which I literally cannot show between backticks because a zero-width background change is invisible. It is redundant because multiple nothings are nothing and that is already included in the production with | NULL removed.

The full grammar has the following, without the unneeded .

fstring_format_spec:
    | FSTRING_MIDDLE 
    | fstring_replacement_field

where FSTRING_MIDDLE is a token. The repetition * is attached to fstring_format_spec in another production.

fstring_full_format_spec:
    | ':' fstring_format_spec*

I am making a PR.

[terryjreedy]

@encukou Petr, assigning to you after reading your comments on #116580. I will submit the trivial PR but leave it to you to apply now, leave open until later, or close. But it later is months away, please consider correcting what seems to be an overt error now.

[encukou]

Thanks for the PR, and for pinging me!
Yup, my PR for lexical_analysis is months away. I'd like to be aware of work in this area, not to block it :)