mypyc: Cls.__dict__ is mappingproxy, not dict

[kai3341]

Bug Report

It looks mypy's type inference works fine and it's mypyc (maybe) bug
Maybe it's connected to builtins vars signature -- vars by the signature should return dict
But it's false, vars may return mappingproxy:

>>> class Foo:
... 
>>> Foo.__dict__
mappingproxy({'__module__': '__main__', '__dict__': <attribute '__dict__' of 'Foo' objects>, '__weakref__': <attribute '__weakref__' of 'Foo' objects>, '__doc__': None})
>>> vars(Foo)
mappingproxy({'__module__': '__main__', '__dict__': <attribute '__dict__' of 'Foo' objects>, '__weakref__': <attribute '__weakref__' of 'Foo' objects>, '__doc__': None})
>>>

To Reproduce

from typing import Type, TypeVar

T = TypeVar("T")


def decorate(cls: Type[T]) -> Type[T]:
    clsdict = cls.__dict__
    # clsdict = vars(cls)
    print(clsdict)
    return cls


@decorate
class Foo:
    pass

Then build it via mypy:

(.venv) kai@asus-ux360c:~$ mypyc mypyc_weird.py 
running build_ext
copying build/lib.linux-x86_64-3.9/mypyc_weird.cpython-39-x86_64-linux-gnu.so -> 

And run it

Expected Behavior

(.venv) kai@asus-ux360c:~$ python -m mypyc_weird
{'__module__': '__main__', '__dict__': <attribute '__dict__' of 'Foo' objects>, '__weakref__': <attribute '__weakref__' of 'Foo' objects>, '__doc__': None}

Actual Behavior

(.venv) kai@asus-ux360c:~$ python -m mypyc_weird.cpython-39-x86_64-linux-gnu.so 
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "mypyc_weird.py", line 14, in <module>
    class Foo:
  File "mypyc_weird.py", line 7, in decorate
    clsdict = cls.__dict__
TypeError: dict object expected; got mappingproxy

Your Environment

• Mypy version used: mypy-1.0.0+dev.dbcbb3f5c3ef791c98088da0bd1dfa6cbf51f301 (latest@git)
• Mypy command-line flags: empty
• Mypy configuration options from mypy.ini (and other config files): empty
• Python version used: 3.9
• Description: Debian GNU/Linux 11 (bullseye)

[AlexWaygood]

Currently the stub for vars() in typeshed is this:

def vars(__object: Any = ...) -> dict[str, Any]: ...

Maybe it should be this:

def vars(__object: Any = ...) -> Mapping[str, Any]: ...

Or this:

@overload
def vars(__object: type) ->  types.MappingProxyType[str, Any]: ...
@overload
def vars(__object: Any = ...) -> dict[str, Any]: ...

However, that doesn't explain why mypyc seems to think that cls.__dict__ should be of type dict. Typeshed correctly already has type.__dict__ as being of type types.MappingProxyType: https://github.com/python/typeshed/blob/7b3fff714a48f0413141ba10344f9473e3ec4a18/stdlib/builtins.pyi#L149

[AlexWaygood]

python/typeshed#9146 has just been merged, which improves the signature for vars().

[kai3341]

@AlexWaygood I didn't find docs how to install latest typeshed and my try to install it directly into venv from git failed. Is there any way to update typeshed to test new behavior?

[AlexWaygood]

In general you can get mypy to use the latest typeshed using the --custom-typeshed-dir command-line argument: clone typeshed, then use that option to point to your local clone of typeshed. I've never done any hacking on mypyc so I'm not sure if the option works the same way with mypyc, but it certainly doesn't reject the argument if you pass it on the command line.

Unfortunately if I do mypyc mypyc_weird.py --custom-typeshed-dir <path-to-my-typeshed-clone>, the compiled module still raises the same exception when I import it. Which is sort of expected -- my PR to typeshed improved the stub for vars(), but didn't alter the stub for type.__dict__ in any way (since that's already correct over at typeshed). EDIT: However, it does look like my typeshed PR fixed the version where you use vars() instead of accessing __dict__ directly! So that's nice.

It looks like the issue with cls.__dict__ is ultimately a mypy/mypyc bug rather than a typeshed bug.

[AlexWaygood]

It's a mypy bug rather than a mypyc bug, and it stems from a more general bug where mypy misunderstands how attribute access on class objects should be understood when metaclasses are involved. Here's a minimal repro:

class Meta(type):
    bar: str

class Foo(metaclass=Meta):
    bar: int

reveal_type(Foo().bar)  # Revealed type is int (correct!)
reveal_type(Foo.bar)  # Revealed type is int, but should be str

https://mypy-play.net/?mypy=latest&python=3.11&gist=011c7d73769c1d028b4d4bfa8a30155d

If this bug is fixed, mypy will correctly infer from typeshed's stub for builtins.type that the __dict__ of a class object is of type types.MappingProxyType, which will solve the mypyc issue.

[JelleZijlstra]

Fixing that bug may be quite disruptive; I expect we'll have to get much stricter in typeshed about adding ClassVar annotations.

[AlexWaygood]

Fixing that bug may be quite disruptive; I expect we'll have to get much stricter in typeshed about adding ClassVar annotations.

I'd like to have a go, anyway, and see what mypy_primer says. I have some ideas :)

[JukkaL]

(I created a new label for mypy issues that are known to cause issues for mypyc users.)