gh-104374: Remove access to class scopes for inlined comprehensions

[JelleZijlstra]

• Issue: List comprehensions now have access to the enclosing class scope #104374

[carljm]

This doesn't look bad at all! Thanks for pursuing it. I guess we can present both this and #104519 as options.

[JelleZijlstra]

Found the following crash with a variation of Carl's fuzzer:

>>> class a: [a for a in [b] for b[[b for _ in [] if a if b]] in [2]]
... 
SystemError: compiler_lookup_arg(name='a') with reftype=1 failed in <module>; freevars of code a: ('a',)

This doesn't reproduce on main, so it's new from this PR.

[JelleZijlstra]

Found a different issue trying to simplify it:

>>> class C: [a for a in [] if [a for _ in []]]
... 
SystemError: _PyST_GetScope(name='a') failed: unknown scope in unit <module> (5033699440); symbols: {'C': 2050}; locals: {}; globals: {}

[JelleZijlstra]

The problem occurs if it's a cell var in the outer comprehension and a free var in the inner one. In that case we should just do LOAD_FAST, but right now we're probably emitting _DEREF opcodes that explode because those cells don't exist in the class namespace.

[carljm]

I pushed a fix for the above issues; it's just a matter of correcting how we decide if a comprehension is "in a class block." Looking only at c->u->u_ste->ste_type gives the wrong answer for nested comprehensions; they are inside their outer comprehension, so never in a class block. We need to use c->u->u_ste->ste_type == ClassBlock && !c->u->u_in_inlined_comp.

[JelleZijlstra]

Would be nice if we could test here that the name resolution in the inner listcomp here is right. I'll try to come up with some more tests around nested listcomps.

[carljm]

Are you still planning to add more tests here, or improve this one?

[JelleZijlstra]

Yes, will push some more tests today or tomorrow. If you'd prefer to merge this now we can also do that; I can just write more tests in another PR.

[carljm]

Yes, I think if we get known fixes merged sooner it's easier to do more effective fuzz testing; if we find more issues we can fix them in separate PRs, and we can add more tests that we think of in separate PRs.

[JelleZijlstra]

Just added two more tests and turned on automerge.

[JelleZijlstra]

Now this segfaults:

class a:
    def a():
        class a:
            [(lambda : (a := a[(a := 2)])[b]) for b in (lambda b, a: 7)[a]]
            [][2] = b
            (1)[lambda a: a] = 4
            (2)[2] = b = a
        (4)[lambda b, a: b] = a = lambda : 1

Haven't tried to minimize it, but it's a compiler crash, something related to the cell/freevars:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
  * frame #0: 0x000000010008b33c python.exe`Py_TYPE(ob=0x0000000000000000) at object.h:204:16 [opt]
    frame #1: 0x000000010008ca8c python.exe`Py_IS_TYPE(ob=<unavailable>, type=0x000000010042d148) at object.h:235:12 [opt]
    frame #2: 0x000000010008db38 python.exe`_PyCode_ConstantKey(op=0x0000000000000000) at codeobject.c:2163:11 [opt]
    frame #3: 0x000000010008dd00 python.exe`_PyCode_ConstantKey(op=0x0000000103557a70) at codeobject.c:2225:24 [opt]
    frame #4: 0x0000000100173334 python.exe`_PyCompile_ConstCacheMergeOne(const_cache=0x0000000103023890, obj=0x000000016fdfedd8) at compile.c:7380:21 [opt]
    frame #5: 0x000000010014d88c python.exe`makecode(umd=0x0000000103407bb0, a=0x000000016fdfee50, const_cache=0x0000000103023890, constslist=0x000000010354b750, maxdepth=5, nlocalsplus=2, code_flags=0, filename=0x0000000103077e20) at assemble.c:569:9 [opt]
    frame #6: 0x000000010014d5b8 python.exe`_PyAssemble_MakeCodeObject(umd=0x0000000103407bb0, const_cache=0x0000000103023890, consts=0x000000010354b750, maxdepth=5, instrs=<unavailable>, nlocalsplus=2, code_flags=0, filename=0x0000000103077e20) at assemble.c:598:14 [opt]

[JelleZijlstra]

Reduced:

def a():
    class a:
        [(lambda : b) for b in [a]]
        print(b)

[JelleZijlstra]

This also reproduces on main (without this PR) and with two functions, so it's not just related to class scopes:

def a():
    def a():
        [(lambda : b) for b in [a]]
        print(b)

Crashes on current main with the same backtrace as above.

[carljm]

Thanks, I'll open a separate issue and PR for this.

[carljm]

Are you still planning to add more tests here, or improve this one?