Skip to content

Flag starred expressions in return and yield #16520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ntBre opened this issue Mar 5, 2025 · 6 comments · Fixed by #17134
Closed

Flag starred expressions in return and yield #16520

ntBre opened this issue Mar 5, 2025 · 6 comments · Fixed by #17134
Labels
bug Something isn't working parser Related to the parser

Comments

@ntBre
Copy link
Contributor

ntBre commented Mar 5, 2025

Both of these are invalid on every Python version I've tried and should be flagged as ParseErrors in the parser:

def f(): yield *rest
def f(): return *rest

They result in SyntaxError: can't use starred expression here on 3.8 and 3.13 and the less helpful SyntaxError: invalid syntax on 3.7.

This is separate from the related case in #16485 where these are part of a tuple, which is allowed after 3.8.

Originally posted by @ntBre in #16485 (comment)

@ntBre ntBre added the parser Related to the parser label Mar 5, 2025
@MichaReiser MichaReiser added the bug Something isn't working label Mar 5, 2025
@AlexWaygood
Copy link
Member

Python's ast module parses these:

Python 3.11.4 (main, Sep 30 2023, 10:54:38) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse('def f(): return *x')
<ast.Module object at 0x7d5c2433d0>

which indicates to me that this syntax error is detected by CPython's compiler rather than CPython's parser. That means that this issue is a sub-issue of #11934.

It doesn't mean that we shouldn't detect the syntax error (we absolutely should!). But it does mean that we may want a way of suppressing the diagnostic internally so that we can continue to test our parser directly against CPython's parser using e.g. the py-fuzzer script (which uses Python's ast module to generate random code).

@MichaReiser
Copy link
Member

Thanks @AlexWaygood That would suggest that this should be implemented as a semantic syntax error, outside the parser.

@ntBre
Copy link
Contributor Author

ntBre commented Mar 5, 2025

I'll start moving some of the other errors from #6591 to #11934 too. I think the global declarations in try nodes are another example:

Python 3.12.9 (main, Feb 12 2025, 14:50:50) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse("""
... a=5
...
... def f():
...     try:
...         pass
...     except:
...         global a
...     else:
...         print(a)
... """)
<ast.Module object at 0x79c0c0e06d10>

@AlexWaygood
Copy link
Member

Good call. Some of the existing errors listed in #11934 and its sub-issues may also have been introduced in recent versions of CPython; we'll need to check when implementing those semantic syntax errors whether they should be emitted for all target versions or not

@dhruvmanila
Copy link
Member

(I think from Python 3.9, you can test the CPython parser directly using python -m ast path/to/file.py.)

@ntBre
Copy link
Contributor Author

ntBre commented Mar 7, 2025

This issue may also include for statements as added in 3.9 (BPO).

Python 3.13.1 (main, Dec  4 2024, 18:05:56) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse("for _ in *rest:...")
<ast.Module object at 0x7cd78d9bb450>
>>> for _ in *rest: ...
  File "<python-input-2>", line 1
    for _ in *rest: ...
             ^^^^^
SyntaxError: can't use starred expression here
>>>

ntBre added a commit that referenced this issue Mar 7, 2025
before Python 3.9

Summary
--

This PR reuses a slightly modified version of the `check_tuple_unpacking` method
added for detecting unpacking in `return` and `yield` statements to detect the
same issue in the iterator clause of `for` loops.

I ran into the same issue with a bare `for x in *rest: ...` example (invalid on
every Python) and added it as a comment on
#16520.

I considered just making this an additional `StarTupleKind` variant as well, but
this change was in a different version of Python, so I kept it separate.

Test Plan
--

New inline tests.
ntBre added a commit that referenced this issue Mar 13, 2025
…fore Python 3.9 (#16558)

Summary
--

This PR reuses a slightly modified version of the
`check_tuple_unpacking` method added for detecting unpacking in `return`
and `yield` statements to detect the same issue in the iterator clause
of `for` loops.

I ran into the same issue with a bare `for x in *rest: ...` example
(invalid even on Python 3.13) and added it as a comment on
#16520.

I considered just making this an additional `StarTupleKind` variant as
well, but this change was in a different version of Python, so I kept it
separate.

Test Plan
--

New inline tests.
ntBre added a commit that referenced this issue Apr 1, 2025
Summary
--

Fixes #16520 by flagging single, starred expressions in `return`, `yield`, and
`for` statements.

I thought `yield from` would also be included here, but that error is emitted by
the CPython parser:

```pycon
>>> ast.parse("def f(): yield from *x")
Traceback (most recent call last):
  File "<python-input-214>", line 1, in <module>
    ast.parse("def f(): yield from *x")
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/ast.py", line 54, in parse
    return compile(source, filename, mode, flags,
                   _feature_version=feature_version, optimize=optimize)
  File "<unknown>", line 1
    def f(): yield from *x
                        ^
SyntaxError: invalid syntax
```

And we also already catch it in our parser.

Test Plan
--

New inline tests.
@ntBre ntBre closed this as completed in d455932 Apr 2, 2025
maxmynter pushed a commit to maxmynter/ruff that referenced this issue Apr 3, 2025
…-sh#17134)

Summary
--

Fixes astral-sh#16520 by flagging single,
starred expressions in `return`, `yield`, and
`for` statements.

I thought `yield from` would also be included here, but that error is
emitted by
the CPython parser:

```pycon
>>> ast.parse("def f(): yield from *x")
Traceback (most recent call last):
  File "<python-input-214>", line 1, in <module>
    ast.parse("def f(): yield from *x")
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/ast.py", line 54, in parse
    return compile(source, filename, mode, flags,
                   _feature_version=feature_version, optimize=optimize)
  File "<unknown>", line 1
    def f(): yield from *x
                        ^
SyntaxError: invalid syntax
```

And we also already catch it in our parser.

Test Plan
--

New inline tests and updates to existing tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working parser Related to the parser
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants