Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-74929: Implement PEP 667 #115153

Merged
merged 44 commits into from
May 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
42d7186
Basic prototype for frame proxy
gaogaotiantian Feb 7, 2024
7eeab1b
Fix some lint and remove oprun check
gaogaotiantian Feb 8, 2024
60e70e7
Not entirely work yet
gaogaotiantian Feb 8, 2024
1454ce4
Fix a bug
gaogaotiantian Feb 8, 2024
0045274
Change code style and add GC
gaogaotiantian Feb 13, 2024
de73bc9
Clean up code
gaogaotiantian Feb 13, 2024
ca92393
Disable all fast/local functions
gaogaotiantian Mar 1, 2024
ff886ff
Update tests for the new f_locals
gaogaotiantian Mar 2, 2024
9690a2d
Comment out the pop for now
gaogaotiantian Mar 2, 2024
6e9848a
Convert f_locals to dict first
gaogaotiantian Mar 2, 2024
b84b0df
Add static to static functions, add interface for new C API
gaogaotiantian Apr 24, 2024
bebff28
Add some tests and a few methods
gaogaotiantian Apr 26, 2024
d846de9
Implement all methods
gaogaotiantian Apr 27, 2024
d00a742
Make f_extra_locals extra lazy
gaogaotiantian Apr 27, 2024
1a4344d
Merge branch 'main' into pep667
gaogaotiantian Apr 27, 2024
f720e12
Fix typo
gaogaotiantian Apr 27, 2024
64d3772
Remove print debugging
gaogaotiantian Apr 27, 2024
2eadbf0
Fix some styling issue
gaogaotiantian Apr 27, 2024
cbae199
Update generated files for cAPI
gaogaotiantian Apr 27, 2024
9e7edf8
Remove f_fast_as_locals and useless calls for sys.settrace
gaogaotiantian Apr 27, 2024
523cb75
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 27, 2024
4b83311
Add the new type to static types
gaogaotiantian Apr 27, 2024
ae2db7c
Remove internal APIs for fast locals
gaogaotiantian Apr 27, 2024
bf45c02
Add extra tests for closure
gaogaotiantian Apr 27, 2024
026e15e
Add the type to globals-to-fix
gaogaotiantian Apr 27, 2024
f42980d
Add CAapi test
gaogaotiantian Apr 27, 2024
e693ad0
Polish lint
gaogaotiantian Apr 27, 2024
5dd045b
Apply some simple changes
gaogaotiantian Apr 28, 2024
30ecd4d
Update Misc/NEWS.d/next/Core and Builtins/2024-04-27-21-44-40.gh-issu…
gaogaotiantian Apr 28, 2024
f35c5e3
Abstract the key index part
gaogaotiantian Apr 28, 2024
5844fb4
Fix error handling
gaogaotiantian Apr 28, 2024
06277f9
Make key index work better
gaogaotiantian Apr 28, 2024
e1c3f56
Add comments for GetHiddenLocals
gaogaotiantian Apr 30, 2024
b672d84
Add global test
gaogaotiantian Apr 30, 2024
3e32572
Remove unsupported methods
gaogaotiantian May 2, 2024
8dc4664
Support non-string keys
gaogaotiantian May 2, 2024
652f641
Use static function for setitem
gaogaotiantian May 2, 2024
f29e6a3
Fix the list comp
gaogaotiantian May 3, 2024
e0ca4fe
Fix mapping check
gaogaotiantian May 3, 2024
f78156a
Fix frame_getlocals
gaogaotiantian May 3, 2024
4503145
Fix test error
gaogaotiantian May 3, 2024
cdac22c
Change the new ref for getcode
gaogaotiantian May 3, 2024
49287ff
Avoid creating the frame object if possible
gaogaotiantian May 3, 2024
378aacf
Remove a single blank line
gaogaotiantian May 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Doc/data/stable_abi.dat

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions Include/ceval.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ PyAPI_FUNC(PyObject *) PyEval_GetGlobals(void);
PyAPI_FUNC(PyObject *) PyEval_GetLocals(void);
PyAPI_FUNC(PyFrameObject *) PyEval_GetFrame(void);

PyAPI_FUNC(PyObject *) PyEval_GetFrameBuiltins(void);
PyAPI_FUNC(PyObject *) PyEval_GetFrameGlobals(void);
PyAPI_FUNC(PyObject *) PyEval_GetFrameLocals(void);

PyAPI_FUNC(int) Py_AddPendingCall(int (*func)(void *), void *arg);
PyAPI_FUNC(int) Py_MakePendingCalls(void);

Expand Down
6 changes: 6 additions & 0 deletions Include/cpython/frameobject.h
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,9 @@ PyAPI_FUNC(int) _PyFrame_IsEntryFrame(PyFrameObject *frame);

PyAPI_FUNC(int) PyFrame_FastToLocalsWithError(PyFrameObject *f);
PyAPI_FUNC(void) PyFrame_FastToLocals(PyFrameObject *);
ncoghlan marked this conversation as resolved.
Show resolved Hide resolved


typedef struct {
PyObject_HEAD
PyFrameObject* frame;
} PyFrameLocalsProxyObject;
2 changes: 2 additions & 0 deletions Include/cpython/pyframe.h
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
#endif

PyAPI_DATA(PyTypeObject) PyFrame_Type;
PyAPI_DATA(PyTypeObject) PyFrameLocalsProxy_Type;

#define PyFrame_Check(op) Py_IS_TYPE((op), &PyFrame_Type)
#define PyFrameLocalsProxy_Check(op) Py_IS_TYPE((op), &PyFrameLocalsProxy_Type)

PyAPI_FUNC(PyFrameObject *) PyFrame_GetBack(PyFrameObject *frame);
PyAPI_FUNC(PyObject *) PyFrame_GetLocals(PyFrameObject *frame);
Expand Down
13 changes: 5 additions & 8 deletions Include/internal/pycore_frame.h
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ struct _frame {
int f_lineno; /* Current line number. Only valid if non-zero */
char f_trace_lines; /* Emit per-line trace events? */
char f_trace_opcodes; /* Emit per-opcode trace events? */
char f_fast_as_locals; /* Have the fast locals of this frame been converted to a dict? */
PyObject *f_extra_locals; /* Dict for locals set by users using f_locals, could be NULL */
/* The frame data, if this frame object owns the frame */
PyObject *_f_frame_data[1];
};
Expand Down Expand Up @@ -242,14 +242,11 @@ _PyFrame_ClearExceptCode(_PyInterpreterFrame * frame);
int
_PyFrame_Traverse(_PyInterpreterFrame *frame, visitproc visit, void *arg);

PyObject *
_PyFrame_GetLocals(_PyInterpreterFrame *frame, int include_hidden);

int
_PyFrame_FastToLocalsWithError(_PyInterpreterFrame *frame);
bool
_PyFrame_HasHiddenLocals(_PyInterpreterFrame *frame);

void
_PyFrame_LocalsToFast(_PyInterpreterFrame *frame, int clear);
PyObject *
_PyFrame_GetLocals(_PyInterpreterFrame *frame);

static inline bool
_PyThreadState_HasStackSpace(PyThreadState *tstate, int size)
Expand Down
186 changes: 175 additions & 11 deletions Lib/test/test_frame.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import copy
import gc
import operator
import re
Expand All @@ -13,7 +14,7 @@
_testcapi = None

from test import support
from test.support import threading_helper, Py_GIL_DISABLED
from test.support import import_helper, threading_helper, Py_GIL_DISABLED
from test.support.script_helper import assert_python_ok


Expand Down Expand Up @@ -198,14 +199,6 @@ def inner():
tb = tb.tb_next
return frames

def test_locals(self):
f, outer, inner = self.make_frames()
outer_locals = outer.f_locals
self.assertIsInstance(outer_locals.pop('inner'), types.FunctionType)
self.assertEqual(outer_locals, {'x': 5, 'y': 6})
inner_locals = inner.f_locals
self.assertEqual(inner_locals, {'x': 5, 'z': 7})

def test_clear_locals(self):
# Test f_locals after clear() (issue #21897)
f, outer, inner = self.make_frames()
Expand All @@ -217,8 +210,8 @@ def test_clear_locals(self):
def test_locals_clear_locals(self):
# Test f_locals before and after clear() (to exercise caching)
f, outer, inner = self.make_frames()
outer.f_locals
inner.f_locals
self.assertNotEqual(outer.f_locals, {})
self.assertNotEqual(inner.f_locals, {})
outer.clear()
inner.clear()
self.assertEqual(outer.f_locals, {})
Expand Down Expand Up @@ -269,6 +262,177 @@ def inner():
r"^<frame at 0x[0-9a-fA-F]+, file %s, line %d, code inner>$"
% (file_repr, offset + 5))

class TestFrameLocals(unittest.TestCase):
def test_scope(self):
class A:
x = 1
sys._getframe().f_locals['x'] = 2
sys._getframe().f_locals['y'] = 2

self.assertEqual(A.x, 2)
self.assertEqual(A.y, 2)

def f():
x = 1
sys._getframe().f_locals['x'] = 2
sys._getframe().f_locals['y'] = 2
self.assertEqual(x, 2)
self.assertEqual(locals()['y'], 2)
f()

def test_closure(self):
x = 1
y = 2

def f():
z = x + y
d = sys._getframe().f_locals
self.assertEqual(d['x'], 1)
self.assertEqual(d['y'], 2)
d['x'] = 2
d['y'] = 3

f()
self.assertEqual(x, 2)
self.assertEqual(y, 3)

def test_as_dict(self):
x = 1
y = 2
d = sys._getframe().f_locals
# self, x, y, d
self.assertEqual(len(d), 4)
self.assertIs(d['d'], d)
self.assertEqual(set(d.keys()), set(['x', 'y', 'd', 'self']))
self.assertEqual(len(d.values()), 4)
self.assertIn(1, d.values())
self.assertEqual(len(d.items()), 4)
self.assertIn(('x', 1), d.items())
self.assertEqual(d.__getitem__('x'), 1)
d.__setitem__('x', 2)
self.assertEqual(d['x'], 2)
self.assertEqual(d.get('x'), 2)
self.assertIs(d.get('non_exist', None), None)
self.assertEqual(d.__len__(), 4)
self.assertEqual(set([key for key in d]), set(['x', 'y', 'd', 'self']))
self.assertIn('x', d)
self.assertTrue(d.__contains__('x'))

gaogaotiantian marked this conversation as resolved.
Show resolved Hide resolved
self.assertEqual(reversed(d), list(reversed(d.keys())))

d.update({'x': 3, 'z': 4})
self.assertEqual(d['x'], 3)
self.assertEqual(d['z'], 4)

with self.assertRaises(TypeError):
d.update([1, 2])

self.assertEqual(d.setdefault('x', 5), 3)
self.assertEqual(d.setdefault('new', 5), 5)
self.assertEqual(d['new'], 5)

with self.assertRaises(KeyError):
d['non_exist']

def test_as_number(self):
x = 1
y = 2
d = sys._getframe().f_locals
self.assertIn('z', d | {'z': 3})
d |= {'z': 3}
self.assertEqual(d['z'], 3)
d |= {'y': 3}
self.assertEqual(d['y'], 3)
with self.assertRaises(TypeError):
d |= 3
with self.assertRaises(TypeError):
_ = d | [3]

def test_non_string_key(self):
d = sys._getframe().f_locals
d[1] = 2
self.assertEqual(d[1], 2)

def test_write_with_hidden(self):
def f():
f_locals = [sys._getframe().f_locals for b in [0]][0]
f_locals['b'] = 2
f_locals['c'] = 3
self.assertEqual(b, 2)
self.assertEqual(c, 3)
b = 0
c = 0
f()

def test_repr(self):
x = 1
# Introduce a reference cycle
frame = sys._getframe()
self.assertEqual(repr(frame.f_locals), repr(dict(frame.f_locals)))

def test_delete(self):
x = 1
d = sys._getframe().f_locals
with self.assertRaises(TypeError):
del d['x']

with self.assertRaises(AttributeError):
d.clear()

with self.assertRaises(AttributeError):
d.pop('x')

@support.cpython_only
def test_sizeof(self):
proxy = sys._getframe().f_locals
support.check_sizeof(self, proxy, support.calcobjsize("P"))

def test_unsupport(self):
x = 1
d = sys._getframe().f_locals
with self.assertRaises(AttributeError):
d.copy()

with self.assertRaises(TypeError):
copy.copy(d)

with self.assertRaises(TypeError):
copy.deepcopy(d)


class TestFrameCApi(unittest.TestCase):
def test_basic(self):
x = 1
ctypes = import_helper.import_module('ctypes')
PyEval_GetFrameLocals = ctypes.pythonapi.PyEval_GetFrameLocals
PyEval_GetFrameLocals.restype = ctypes.py_object
frame_locals = PyEval_GetFrameLocals()
self.assertTrue(type(frame_locals), dict)
self.assertEqual(frame_locals['x'], 1)
frame_locals['x'] = 2
self.assertEqual(x, 1)

PyEval_GetFrameGlobals = ctypes.pythonapi.PyEval_GetFrameGlobals
PyEval_GetFrameGlobals.restype = ctypes.py_object
frame_globals = PyEval_GetFrameGlobals()
self.assertTrue(type(frame_globals), dict)
gaogaotiantian marked this conversation as resolved.
Show resolved Hide resolved
self.assertIs(frame_globals, globals())

PyEval_GetFrameBuiltins = ctypes.pythonapi.PyEval_GetFrameBuiltins
PyEval_GetFrameBuiltins.restype = ctypes.py_object
frame_builtins = PyEval_GetFrameBuiltins()
self.assertEqual(frame_builtins, __builtins__)

PyFrame_GetLocals = ctypes.pythonapi.PyFrame_GetLocals
PyFrame_GetLocals.argtypes = [ctypes.py_object]
PyFrame_GetLocals.restype = ctypes.py_object
frame = sys._getframe()
f_locals = PyFrame_GetLocals(frame)
self.assertTrue(f_locals['x'], 1)
f_locals['x'] = 2
self.assertEqual(x, 2)


class TestIncompleteFrameAreInvisible(unittest.TestCase):

def test_issue95818(self):
Expand Down
7 changes: 6 additions & 1 deletion Lib/test/test_listcomps.py
Original file line number Diff line number Diff line change
Expand Up @@ -622,9 +622,14 @@ def test_exception_in_post_comp_call(self):

def test_frame_locals(self):
code = """
val = [sys._getframe().f_locals for a in [0]][0]["a"]
val = "a" in [sys._getframe().f_locals for a in [0]][0]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we've changed the intent of the test here. It was previously checking that a was added to the f_locals snapshot during the listcomp, and now it's checking that it is not visible through the proxy afterwards.

To cover both aspects, I think we need two test cases (first one shows that "a" is accessible in the f_locals while the listcomp is running, second shows that it is gone afterwards):

    def test_frame_locals_in_listcomp(self):
        # Iteration variable is accessible via f_locals proxy while the listcomp is running
        code = """
            val = [sys._getframe().f_locals["a"] for a in [0]][0]
        """
        import sys
        self._check_in_scopes(code, {"val": 0}, ns={"sys": sys})

    def test_frame_locals_after_listcomp(self):
        # Iteration variable is no longer accessible via f_locals proxy after listcomp finishes
        code = """
            val = "a" in [sys._getframe().f_locals for a in [0]][0]
        """
        import sys
        self._check_in_scopes(code, {"val": False}, ns={"sys": sys})

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a very interesting point because it introduced a new issue - should we include the hidden fast local in f_locals when it's module/class scope?

val = [sys._getframe().f_locals["a"] for a in [0]][0]

What should the sys._getframe().f_locals inside the list comprehension return? It's it a dict or a proxy? Do we consider that "within" the function scope and return a proxy? Will that include the module-level variables as well? Do we return a dict? Should we include variable a? If so, writing to the key 'a' won't have any effects because it's a fast variable, how do we deal with that inconsistency?

@markshannon

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hidden fast-locals were designed to mimic the prior behavior when comprehensions were previously implemented as one-shot functions. So as much as is feasible, the behavior of f_locals with hidden fast-locals should mirror how it would behave if the comprehension were a function.

Copy link
Member

@carljm carljm Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second proposed test in @ncoghlan 's comment is not correct. Nor is the test as currently modified in the PR.

If you access f_locals on a frame you got while inside the comprehension, it should include the hidden fast-local (i.e. the comprehension iteration variable), and that variable should still be present (in the f_locals of the frame you got while inside the comprehension) even after the comprehension is done. This behavior is the same as getting a frame (or its f_locals) inside a function and returning it from the function. This is the current behavior in main, and this behavior is important to keep in PEP 667.

Of course if you access the frame outside the comprehension, the comprehension's hidden fast locals should not be present in its f_locals.

Currently in main, module globals are present in the f_locals of a frame accessed inside a module-level comprehension. This differs from a function, but this difference was called out in PEP 709 and determined to be OK, and I think it's still OK.

writing to the key 'a' won't have any effects because it's a fast variable

I suspect this is also not a problem in practice, though if we can reasonably return a proxy that can support it, it would be even better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you access f_locals on a frame you got while inside the comprehension, it should include the hidden fast-local (i.e. the comprehension iteration variable), and that variable should still be present (in the f_locals of the frame you got while inside the comprehension) even after the comprehension is done.

I don't think this is correct. f_locals is a proxy, so as the frame changes, so will the proxy. Inside the comprehension, the comprehension variables will be visible. After the comprehension has executed, the comprehension variables no longer exist.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the key here is "as if it's a one-shot function". That's infeasible with the current compiler because it's not a function anymore. For a function, we have a dedicated frame object that keeps the local variables, which can last longer than the intepreter frame as long as there are references on it. In that way, we can always access the variables on that frame even when the actual interpreter frame is gone.

However, with inline comprehension, it fakes the "function call" and clear the hidden variable - there's no mechanism currently to prevent the variable from being cleared. If we want this, we'll probably need to change the compiler and the interpreter which will definitely postpone this PEP to 3.14 (and would probably make the compiler code more complicated).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For class and module level comprehensions, can we return a proxy if f_locals is accessed inside the comprehension and a dict if it's accessed outside which does not have the hidden variable? I feel like that's more consistent.

Copy link
Member

@carljm carljm Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is correct. f_locals is a proxy, so as the frame changes, so will the proxy. Inside the comprehension, the comprehension variables will be visible. After the comprehension has executed, the comprehension variables no longer exist.

It depends whether we are aiming for "correctness" for the actual situation (that the comprehension doesn't have its own frame), or for the backwards-compatible fiction (which PEP 709 only partially maintained, but did jump through hoops to maintain in terms of visibility of class-scoped variables) that comprehensions still behave as if they have their own frame. But as @gaogaotiantian points out, it may be very hard or impossible to maintain that fiction in this case, while still having f_locals be a proxy.

TBH in the long term I think it would be better to move away from that fiction anyway, but it will have backwards-compatibility implications. Today in main, sys._getframe.f_locals() inside a module-level comprehension returns a dictionary that includes the comprehension iteration variable, and that variable is still visible in that dictionary after the comprehension is done.

I don't know why someone relying on that couldn't just use locals() instead, though, and locals() should be fine for backwards-compatibility; it remains a dict. So I don't know how significant that backwards-incompatibility will be in practice.

For class and module level comprehensions, can we return a proxy if f_locals is accessed inside the comprehension and a dict if it's accessed outside which does not have the hidden variable? I feel like that's more consistent.

Certainly the dict returned when f_locals is accessed outside the comprehension should not have the hidden local in it.

I think returning a proxy inside the module- or class-scoped comprehension is more internally consistent for PEP 667; returning a dict (that includes the comprehension iteration variable) would be a bit more backward-compatible.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow the details, but I agree that people who access f_locals are better off being shown implementation details that may vary by Python version than a costly fiction.

"""
import sys
self._check_in_scopes(code, {"val": False}, ns={"sys": sys})

code = """
val = [sys._getframe().f_locals["a"] for a in [0]][0]
"""
self._check_in_scopes(code, {"val": 0}, ns={"sys": sys})

def _recursive_replace(self, maybe_code):
Expand Down
17 changes: 0 additions & 17 deletions Lib/test/test_peepholer.py
Original file line number Diff line number Diff line change
Expand Up @@ -933,23 +933,6 @@ def f():
self.assertNotInBytecode(f, "LOAD_FAST_CHECK")
return f

def test_deleting_local_warns_and_assigns_none(self):
f = self.make_function_with_no_checks()
co_code = f.__code__.co_code
def trace(frame, event, arg):
if event == 'line' and frame.f_lineno == 4:
del frame.f_locals["x"]
sys.settrace(None)
return None
return trace
e = r"assigning None to unbound local 'x'"
with self.assertWarnsRegex(RuntimeWarning, e):
sys.settrace(trace)
f()
self.assertInBytecode(f, "LOAD_FAST")
self.assertNotInBytecode(f, "LOAD_FAST_CHECK")
self.assertEqual(f.__code__.co_code, co_code)

def test_modifying_local_does_not_add_check(self):
f = self.make_function_with_no_checks()
def trace(frame, event, arg):
Expand Down
3 changes: 3 additions & 0 deletions Lib/test/test_stable_abi_ctypes.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Lib/test/test_sys.py
Original file line number Diff line number Diff line change
Expand Up @@ -1555,7 +1555,7 @@ class C(object): pass
def func():
return sys._getframe()
x = func()
check(x, size('3Pi3c7P2ic??2P'))
check(x, size('3Pi2cP7P2ic??2P'))
# function
def func(): pass
check(func, size('15Pi'))
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Implement PEP 667 - converted ``frame.f_locals`` to a write through proxy
6 changes: 6 additions & 0 deletions Misc/stable_abi.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2501,3 +2501,9 @@
added = '3.13'
[function.PyType_GetModuleByDef]
added = '3.13'
[function.PyEval_GetFrameBuiltins]
added = '3.13'
[function.PyEval_GetFrameGlobals]
added = '3.13'
[function.PyEval_GetFrameLocals]
added = '3.13'
Loading
Loading