`pygettext` does not work with new f-strings (same quotes)

[Obstbeeren]

Bug report

Bug description:

When using the new f-strings in 3.12 with the same quotes, pygettext failes.

from gettext import gettext as _

print(_(f"Hello, {_("Test")}!"))

Running pygettext returns the following error:

*** test.py:3: Seen unexpected token "f""

CPython versions tested on:

3.12

Operating systems tested on:

Linux, Windows

[terryjreedy]

The print works, at least on Windows 3.13.0a2, but there does not seem to be a pygettext module.

[serhiy-storchaka]

pygettext is not a module. It is a script in Tools/i18n.

It fails because it currently does not support new tokens for f-strings: FSTRING_START, FSTRING_MIDDLE and FSTRING_END. Previously the tokenizer produced a single STRING token for f-string. It have to be parsed to the AST and processed recursively.

[serhiy-storchaka]

It seems to me, that it works correct in 3.12 with this example.

1. It is not related to using same quotes. You get the same result with different quotes.
2. It complains, because it expects a literal string as argument of _(). You get a similar complain about _(x) and _('a'+'b'). It is not error. Perhaps it should be more clear about this.
3. It correctly finds "Test" in the nested _("Test"). But it does not find it in 3.11, this is a bug.
4. It does not find nested calls in some other examples, e.g. _(_('a')), class A(f(_('a'))):, def f(x=_('a')):, etc.
5. It can find false docstrings, e.g. in def f() -> lambda: '''not docstring''':.

So, there are two issues:

1. pygettext should be more clean about what is error and what is not error. _() can be used with a non-literal string, and it is in general not error, but it can be programming error if f-string is used unintentionally, so it should emit a warning.
2. In some corner cases pygettext cannot find nested _() calls or found false docstrings. It is desirable to make it more reliable, although processing class and def declarations can be too complicated. Using AST (pygettext: use an AST parser instead of a tokenizer #104400) can make it simpler, but it can have other drawbacks.

Also, the code for handling f-strings in 3.11 is no longer used and can be deleted.

[tomasr8]

Fixed in #104402