-
-
Notifications
You must be signed in to change notification settings - Fork 30.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-104357: fix inlined comprehensions that close over iteration var #104368
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
carljm
changed the title
gh-104357: fix inlined comprehensions with cells
gh-104357: fix inlined comprehensions that close over iteration var
May 10, 2023
JelleZijlstra
approved these changes
May 11, 2023
carljm
added a commit
to carljm/cpython
that referenced
this pull request
May 11, 2023
* main: (27 commits) pythongh-87849: fix SEND specialization family definition (pythonGH-104268) pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384) pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367) pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369) pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383) pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287) pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185) pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368) pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141) pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182) pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364) pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539) pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292) pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202) pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013) pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355) pythongh-103960: Dark mode: invert image brightness (python#103983) pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253) pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354) pythongh-101819: Harden _io init (python#104352) ...
carljm
added a commit
to carljm/cpython
that referenced
this pull request
May 11, 2023
* main: pythongh-87849: fix SEND specialization family definition (pythonGH-104268) pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384) pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367) pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369) pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383) pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287) pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185) pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368) pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes the case where an inlined comprehension has an internal cellvar (i.e. it creates lambdas that close over the comprehension iteration variable), but the same name is also used a fast-local in the outer scope, and is read but never written within the body of the function (so e.g. an argument).
In code like this:
because
x
is a cell inside the comprehension, after inlining it would appear inu_cellvars
forf
, and thus the compilation off
would insert aMAKE_CELL
forx
at the start of the function. But we would still compile the non-comprehension part of the function treatingx
as a fast-local (emittingLOAD_FAST
for it), sof
would return a cell object containing the value ofx
, instead of returningx
directly.Similar cases were already tested, but only with a write (compiled as
STORE_FAST
) tox
before it was referenced, which stored the value directly and overwrote the wrongly-created cell.One approach to fix this would be to track which names are cells only inside an inlined comprehension, and prevent the compiler from emitting
MAKE_CELL
for those names (the inlined comprehension will itself emitMAKE_CELL
where it should, within the comprehension's isolated scope.) But this fix would add additional complexity to the compiler.A simpler fix (implemented here) is to ensure that we always treat such a variable as a cell throughout the outer function too (emitting
LOAD_DEREF
andSTORE_DEREF
for it). This maintains clearer semantics forco_cellvars
(if a name is in there, it is a cell in the main function) and keeps the compiler simpler. The downside is that cells are slightly slower than fast locals, but comprehensions creating lambdas that close over the iteration variable are rare in real code (they don't behave as people usually want them to, given that all the created lambda functions share the same final value in their closure), so I don't think performance is a concern in this edge case.Fixes #104357