Runtime implementation of TypedDict

[ilevkivskyi]

Fixes #28

@gvanrossum I think it is time to consider the runtime implementation of TypedDict (mypy counterpart seems to be ready soon). Here is a simple (but quite flexible) implementation that supports three forms (iterable, keywords, class for 3.6+). In all forms type info for runtime introspection is accessible via __annotations__, multiple inheritance is supported. Some examples:

class Point1D(TypedDict):
    x: float
class Point2D(Point1D):
    y: float

Grid2D = TypedDict('Grid2D', [('x', int), ('y', int)])
Grid3D = TypedDict('Grid3D', {'x': int, 'y': int, 'z': int})
Info = TypedDict('Info', color=str, size=int)

class CoolPoint2D(Point2D, Info):
    pass

assert Point2D(x=1, y=2) == dict(x=1, y=2)
assert CoolPoint2D.__bases__ == (dict,)
assert CoolPoint2D.__annotations__ == {'x': float, 'y': float, 'size': int, 'color': str}

Everything is implemented to be pickleable and fast. I have measured and Point2D(x=1, y=2) is slower that dict(x=1, y=2) by only 3% to 11% depending on Python version.

Please, don't be scared by the size of diff. more than half of it is tests and implementations for Python 2 and Python 3 are identical.

[ilevkivskyi]

@gvanrossum Important comment: the iterable syntax supports arbitrary iterables, including both list of tuples and a dictionary. I edited the description to clarify this.

[gvanrossum]

FYI I think we should not put this in typing.py yet. The plan for TypedDict is to use the new mypy_extensions machanism first.

[ilevkivskyi]

@gvanrossum

The plan for TypedDict is to use the new mypy_extensions machanism first.

I see. So should I close it here and instead make a PR to mypy? As I understand, the runtime implementation will be needed anyway. Or is it supposed to be a typecheck only extension? (Sorry I was not following this idea closely.)

[markshannon]

OOI, why isn't the runtime implementation of TypedDict just dict?

[gvanrossum]

For design questions it's probably better to address these to the issue in mypy: python/mypy#985 or perhaps the corresponding issue here in typing: #28

I think Mark's POV has value: after X = TypedDict(...) we could just leave X as an alias for dict. This would be more analogous to NewType than to NamedTuple (the former is designed to disappear without a trace at runtime; the latter cannot just be an alias for tuple because it supports new attributes and keyword args). I don't recall if we reached agreement on this (I believe the author of the mypy PRs, @davidfstr, surprised us all).

[JukkaL]

If a typed dictionary type object would be an alias for dict, then isinstance({'x': 1}, MyTDict) would be true, even if the first argument actually isn't valid as a particular typed dictionary. It would be less confusing if typed dictionary types can't be used with isinstance.

[ilevkivskyi]

Indeed, X = TypedDict(...) could lead to X is dict. This PR actually does almost this. A bit more complex implementation proposed here has three advantages:

• Runtime introspection by accessing __annotations__ (this might be helpful for documentation tools).
• Runtime check: TypedDict('X', {'x': int, 'y': 1}) will raise a TypeError (same as e.g. List[1]).
  Also a recently added note by Jukka could be easily implemented (again, in accordance with current behaviour of generics).
• If someone does:

class Point1D(TypedDict):
    x: float
class Point2D(Point1D):
    y: float
class Point3D(Point2D):
    z: float

then Point3D will be probably slower at runtime if Point1D is dict (this PR provides a direct short-cut to dict for all subclasses).

[davidfstr]

Indeed I was thinking X = TypedDict(...) would be identical to dict at runtime. And yes that would mean that queries like isinstance(foo, X) would not be reliable. It probably also implies X is dict, assuming total erasure.

[davidfstr]

Hi @ilevkivskyi, thanks for taking a sizeable stab at the runtime syntax implementation! Were you planning on migrating this PR to mypy_extensions?

I'm personally most interested in incremental support for keyword-args syntax (i.e. Point = TypedDict('Point', x=int, y=int)). If you wanted to migrate just that part, I'd be highly motivated to review it quickly and shepherd it for checkin.

I'm holding off on support for the class-based syntax in mypy for a while until issues in the initial implementation are shaken out. This doesn't preclude you adding runtime support for the class-based syntax. It just means it won't be recognized in mypy until I or someone else gets around to implementing type-checker support for it, which might be a while. :-)

[ilevkivskyi]

@davidfstr Yes, I was going to move this to mypy_extensions, also I could implement the support for class-based syntax in mypy after everything else is settled (same way as I did this for NamedTuple some time ago). I don't have time to do this right now, but I will definitely will get back to this beginning of next week.

[ilevkivskyi]

I am closing this in favour of python/mypy#2552