gh-74929: Implement PEP 667

[gaogaotiantian]

This is a prototype implementation of PEP 667.
All the examples given in PEP 667 passed.

The code only serves as a prototype, the proxy object is not even tracked by GC. However, the most essential features are implemented.

frame.f_locals is now a real-time proxy for the actual local variables in the frame (there's no mapping yet but that's an implementation detail).

locals() behaves basically like before.

All tests passed except for 3 - all expected

• test_sys for frame size - yes it changed
• test_frame - The pop method is not implemented yet
• test_listcomps - the behavior of frame.f_locals changed so the corresponding test is not valid anymore

• Issue: Local variable assignment is broken when combined with threads + tracing + closures #74929

[markshannon]

Thanks so much for doing this.

I've a few comments, but nothing major.

[carljm]

Thanks for doing this! I was hoping this would get done for 3.13 but hadn't found time to do it yet.

I don't agree that test_frame_locals in test_listcomps should be expected to fail under PEP 667. f_locals is supposed to be a real-time proxy for the live variables in the frame. Within a comprehension, comprehension-scoped variables are live and should be present in f_locals. The fact that this test is failing suggests that the PR does not yet correctly handle "fast-hidden" locals.

EDIT: on second thought, I realized the problem is likely that f_locals is now a live view instead of a snapshot, so by the time we look for the key, the comprehension is no longer running, and a is no longer a live name. So you are right that the test as written should be expected to fail; it should be updated to take a copy of f_locals within the comprehension.

[gaogaotiantian]

EDIT: on second thought, I realized the problem is likely that f_locals is now a live view instead of a snapshot, so by the time we look for the key, the comprehension is no longer running, and a is no longer a live name. So you are right that the test as written should be expected to fail; it should be updated to take a copy of f_locals within the comprehension.

Yes, that's why the test fails. The test is val = [sys._getframe().f_locals for a in [0]][0]["a"] so when it tries to access ["a"], it's already out of the comprehension scope. This actually represents one of the changes we made to f_locals.

[gaogaotiantian]

I cleaned up the code a bit to address the review and added the GC part in. This is still in draft state, which only aims to demostrate the possibility of PEP 667. There's plenty of work to be done before this can be merged in. The current goal is to make it enough for the PSF to decide whether to accept PEP 667. Let me know if there's still missing pieces (like do we want to actually remove all the FAST/LOCAL conversion calls to prove the point).

[markshannon]

Removing the fast locals <--> slow locals conversion is key to fixing #74929. I think that is worth doing before submitting PEP 667 to the SC.

@gaogaotiantian since you are doing most of the work on this, would you like to be added as co-author of PEP 667?

[gaogaotiantian]

Sure, I'm happy to be listed as the co-author. I'll work on the fast vs local thing and check if #74929 is fixed.

[gvanrossum]

FWIW the discussion has been started at https://discuss.python.org/t/pep-667-consistent-views-of-namespaces/46631.

We now need at least the following updates to the PEP:

• Add Discussion-To header pointing to that Discourse thread
• Add Tian Gao to list of Authors
• Link to implementation prototype PR from the Implementation section

[gaogaotiantian]

I've already added myself to the list of authors with Mark's guidance. Let me know if you want me to make the PR to update the PEP.

[gvanrossum]

Go right ahead.

[gaogaotiantian]

@markshannon okay I made all fast/local related functions no-op. It did not break anything new and #74929 (comment) is not an issue anymore. The original test case in #74929 has been accidentally fixed in 3.11, is there any repro case that I can test against?

[carljm]

Is there a reason you're leaving the failing tests as failing? I think it is easier for reviewers to evaluate the compatibility impact on tests by seeing the required changes to tests in the PR, rather than by having to go look at the CI test results.

[gaogaotiantian]

The current implementation is just a prototype and far from being done. I can make some modifications to the test so they can pass with the change if that's preferred.

[gaogaotiantian]

We have two more months until feature freeze. If we want to land this on 3.13, we probably should push it forward soon.

[gvanrossum]

How is the Discourse discussion on this topic going? Is it substantial enough to send the PEP to the SC? Do we need to make any PEP updates first?

[gaogaotiantian]

I made a PR to the PEP and it's approved but not merged (I think Mark has to review and approve it?), if you meant the link changes mentioned before. I'm not sure if the discussion thread attracts enough attention. What point should we get for the Disclosure discussion?

[markshannon]

We need to decide which mapping methods we want FrameLocalsProxy to support.
Listing all the methods on UserDict:

Dunder methods

• Comparison operators, __le__, etc.
• __or__, __ior__, __ror__
• __contains__
• __copy__
• __getitem__
• __setitem__
• __delitem__ (see comment below under normal methods)
• __format__
• __len__
• __iter__
• __getitem__
• __reversed__
• __sizeof__
• __repr__, __str__
• Pickling support, __reduce__, etc

I think we need to support all the above, except pickling, and that __copy__ should return a normal __dict__, not a shallow copy.

Normal methods

• clear,
• copy,
• get,
• items,
• keys,
• pop,
• popitem,
• setdefault,
• update,
• values

Methods that remove items are tricky as we cannot properly delete fast locals; we just set them to None and emit a warning.
So, I think it best not to implement clear, pop, or popitem.

[markshannon]

We don't need to implement them all before submitting the PEP though, just decide on which methods to support.

[gaogaotiantian]

Should we follow some standard for the methods we should implement? The original PEP mentioned that we should do a MutableMapping, which probably lists all the methods that need to be implement.

For the remove related methods - should the debugger be able to decref the local variable? Just like del val. I think making that equivalent to f.locals.pop("val") is reasonable. Should we even emit a warning for that? Did we do that now? I think we emit a warning when we have to assign None to unassigned local variables.

[gaogaotiantian]

Okay I resolved most of the comments, and I believe there's one thing left here - comprehensions.

I'm just going to jump to my proposal and see if that makes sense. If not, we can discuss it more.

import sys
def f():
    x = 1
    [print(sys._getframe().f_locals) for a in [0]]
    print([sys._getframe().f_locals for a in [0]][0])
    print(sys._getframe().f_locals)
f()

will print

# Inside comprehension, show hidden variable a and locals in the outside frame
{'x': 1, 'a': 0}
# f_locals from inside but read outside, no hidden variable, only locals.
# This does not match the fiction because the variable should "exist" if it's a frame, but in reality it's gone
{'x': 1}
# f_locals outside, just local variables
{'x': 1}

Notice all the values above are proxies.

import sys
class C:
    x = 1
    [print(sys._getframe().f_locals) for a in [0]]
    print([sys._getframe().f_locals for a in [0]][0])
    print(sys._getframe().f_locals)

will print (ignore __module__ and __qualname__)

# This is a proxy without class-level variable
{'a': 0}
# Empty proxy
{}
# A dict with class level variable
{'x': 1}

An alternative would be:

# A dict, but you can't write to variable a to change the local value
{'a': 0, 'x': 1}
# A dict
{'x': 1}
# Still a dict
{'x': 1}

The latter seems more consistent with the function level result, and it makes locals() backwards compatible, but it creates a horrible exemption where the f_locals is not "write through".

Maybe we could stick to a simple rule here - f_locals will always return something that's write through. locals() always return dict(proxy) or the dict itself (in class/module level, outside of the comprehension).

However, this will break the backwards compatibility for locals() in comprehension in class/module level.

[carljm]

I think that proposal looks reasonable.

I think your first proposal for the class behavior is better. Comprehensions in class scope cannot see class-level variables (as if the comprehension were still a function, since functions defined in a class scope can't see variables in the enclosing class scope either), so those class variables should not show up when f_locals is accessed inside the comprehension.

[gaogaotiantian]

I think your first proposal for the class behavior is better. Comprehensions in class scope cannot see class-level variables (as if the comprehension were still a function, since functions defined in a class scope can't see variables in the enclosing class scope either), so those class variables should not show up when f_locals is accessed inside the comprehension.

But we will probably treat module-level the same as class-level. Do you think it's reasonable to have a module-level comprehension to return a proxy without the global variables?

edit: I guess that makes sense as well. Even in a function in a module, f_locals should not return any global variables.

[gaogaotiantian]

Okay I finished the last piece and cleaned up the code a bit.

Now we have some very consistent behavior for f_locals and locals().

locals() will be either

• dict(f_locals) for function scope frames and all comprehensions
• or f_locals for class and module frames

f_locals will be

• a proxy for function scope frames and all comprehensions (comprehension variables are not accessible outside of the comprehension)
• or a dict for class and module frames

f_locals will always be write through, no exceptions.

This breaks the behavior introduced by pep 709 that [locals() for a in [0]] will include outside variables if it's under a module or class level scope, but I think this specific behavior is not desired nor intentional, it's a compromise to begin with. If we consider the comprehension as a "function", the locals() should never include the outside variables.

locals() in a function scope will include the cell variables as expected.

Let me know if there are missing pieces that I don't know of.

[markshannon]

This looks good. Thanks for your perseverance.

last chance for comments.
I'm going to merge this in 12 hours or so, if no one objects.

[henryiii]

FYI, docs are missing here: https://docs.python.org/3.13/c-api/reflection.html

The back-compat functions were causing a few destructors to not be called properly in three tests in pybind11; switching to the new functions fixed it, though. Not sure if that's expected. I believe it’s because the back compat function for eval frame locals is basically the same as the new function and returns a new reference.