Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-89263: Add typing.get_overloads #31716

Merged
merged 37 commits into from
Apr 16, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
2ee377d
initial
JelleZijlstra Mar 6, 2022
831b565
Implementation, tests, and docs
JelleZijlstra Mar 7, 2022
f03f8a9
fix versionadded
JelleZijlstra Mar 7, 2022
404668a
Merge branch 'main' into funcregistry
JelleZijlstra Mar 8, 2022
7a5b0d1
make get_key_for_callable private
JelleZijlstra Mar 8, 2022
6998255
doc updates; remove unnecessary try-except
JelleZijlstra Mar 9, 2022
26bb908
Merge remote-tracking branch 'upstream/main' into funcregistry
JelleZijlstra Mar 27, 2022
f52b757
rename method
JelleZijlstra Mar 27, 2022
fc6a925
Don't store singledispatch in the registry
JelleZijlstra Mar 27, 2022
b524244
more tests
JelleZijlstra Mar 27, 2022
e95558e
and another
JelleZijlstra Mar 27, 2022
31fd72d
fix line length in new tests
JelleZijlstra Mar 27, 2022
7041ad3
Update Doc/library/functools.rst
JelleZijlstra Mar 27, 2022
e26b0db
Update Doc/library/typing.rst
JelleZijlstra Mar 27, 2022
1bf89fb
only for overload
JelleZijlstra Apr 2, 2022
83ac432
Merge remote-tracking branch 'upstream/main' into funcregistry
JelleZijlstra Apr 2, 2022
dfdbdc7
fix tests
JelleZijlstra Apr 2, 2022
e16c8d0
undo stray changes, fix NEWS entry
JelleZijlstra Apr 2, 2022
b3d2227
remove extra import
JelleZijlstra Apr 2, 2022
9727eee
Apply suggestions from code review
JelleZijlstra Apr 2, 2022
2e374b8
Apply suggestions from code review
JelleZijlstra Apr 3, 2022
ff03b12
Guido's feedback
JelleZijlstra Apr 3, 2022
17f0710
Optimizations suggested by Guido and Alex
JelleZijlstra Apr 3, 2022
2346970
inline _get_firstlineno, store outer objects for classmethod/staticme…
JelleZijlstra Apr 3, 2022
f2053a0
use defaultdict
JelleZijlstra Apr 3, 2022
b6131ad
another optimization
JelleZijlstra Apr 4, 2022
506bd66
Update Lib/typing.py
JelleZijlstra Apr 7, 2022
e9a2100
Merge remote-tracking branch 'upstream/main' into funcregistry
JelleZijlstra Apr 8, 2022
2b1a5cc
Merge remote-tracking branch 'upstream/main' into funcregistry
JelleZijlstra Apr 9, 2022
103bfd4
Simpler implementation (thanks Guido)
JelleZijlstra Apr 9, 2022
d453f7f
More comments and tests
JelleZijlstra Apr 9, 2022
450afeb
Merge remote-tracking branch 'upstream/main' into funcregistry
JelleZijlstra Apr 14, 2022
ea62287
simplify clear_overloads
JelleZijlstra Apr 14, 2022
905253c
use partial
JelleZijlstra Apr 14, 2022
debbf8a
add test
JelleZijlstra Apr 14, 2022
754c134
docs changes (thanks Alex)
JelleZijlstra Apr 14, 2022
1ad8224
Merge remote-tracking branch 'upstream/main' into funcregistry
JelleZijlstra Apr 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions Doc/library/typing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2407,6 +2407,35 @@ Functions and decorators

See :pep:`484` for details and comparison with other typing semantics.

.. versionchanged:: 3.11
Overloaded functions can now be introspected at runtime using
:func:`get_overloads`.


.. function:: get_overloads(func)

Return a sequence of :func:`@overload <overload>`-decorated definitions for
JelleZijlstra marked this conversation as resolved.
Show resolved Hide resolved
*func*. *func* is the function object for the implementation of the
overloaded function. For example, given the definition of ``process`` in
Comment on lines +2418 to +2419
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence sounds quite strange to me:

*func* is the function object for the implementation of the overloaded function.

But I'm not sure I have a better suggestion off the top of my head ://

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, unfortunately the overload docs don't have a clear term for the implementation function either.

the documentation for :func:`@overload <overload>`,
``get_overloads(process)`` will return a sequence of three function objects
for the three defined overloads. If called on a function with no overloads,
``get_overloads`` returns an empty sequence.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
``get_overloads`` returns an empty sequence.
return an empty sequence.

Tiny nit -- this paragraph starts off using the imperative mood

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the previous sentence is in the indicative, so I think this is fine?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it's not really a big deal either way :)


``get_overloads`` can be used for introspecting an overloaded function at
runtime.
JelleZijlstra marked this conversation as resolved.
Show resolved Hide resolved

.. versionadded:: 3.11


.. function:: clear_overloads()

Clear all registered overloads in the internal registry. This can be used
to reclaim the memory used by the registry.

.. versionadded:: 3.11


.. decorator:: final

A decorator to indicate to type checkers that the decorated method
Expand Down
72 changes: 68 additions & 4 deletions Lib/test/test_typing.py
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
import contextlib
import collections
from collections import defaultdict
from functools import lru_cache
import inspect
import pickle
import re
import sys
import warnings
from unittest import TestCase, main, skipUnless, skip
from unittest.mock import patch
from copy import copy, deepcopy

from typing import Any, NoReturn, Never, assert_never
from typing import overload, get_overloads, clear_overloads
from typing import TypeVar, TypeVarTuple, Unpack, AnyStr
from typing import T, KT, VT # Not in __all__.
from typing import Union, Optional, Literal
Expand Down Expand Up @@ -3890,11 +3893,22 @@ def test_or(self):
self.assertEqual("x" | X, Union["x", X])


@lru_cache()
def cached_func(x, y):
return 3 * x + y


class MethodHolder:
@classmethod
def clsmethod(cls): ...
@staticmethod
def stmethod(): ...
def method(self): ...


class OverloadTests(BaseTestCase):

def test_overload_fails(self):
from typing import overload

with self.assertRaises(RuntimeError):

@overload
Expand All @@ -3904,8 +3918,6 @@ def blah():
blah()

def test_overload_succeeds(self):
from typing import overload

@overload
def blah():
pass
Expand All @@ -3915,6 +3927,58 @@ def blah():

blah()

def set_up_overloads(self):
def blah():
pass

overload1 = blah
overload(blah)

def blah():
pass

overload2 = blah
overload(blah)

def blah():
pass

return blah, [overload1, overload2]

# Make sure we don't clear the global overload registry
@patch("typing._overload_registry",
defaultdict(lambda: defaultdict(dict)))
def test_overload_registry(self):
JelleZijlstra marked this conversation as resolved.
Show resolved Hide resolved
# The registry starts out empty
self.assertEqual(typing._overload_registry, {})

impl, overloads = self.set_up_overloads()
self.assertNotEqual(typing._overload_registry, {})
self.assertEqual(list(get_overloads(impl)), overloads)

def some_other_func(): pass
overload(some_other_func)
other_overload = some_other_func
def some_other_func(): pass
self.assertEqual(list(get_overloads(some_other_func)), [other_overload])

# Make sure that after we clear all overloads, the registry is
# completely empty.
clear_overloads()
self.assertEqual(typing._overload_registry, {})
self.assertEqual(get_overloads(impl), [])

# Querying a function with no overloads shouldn't change the registry.
def the_only_one(): pass
self.assertEqual(get_overloads(the_only_one), [])
self.assertEqual(typing._overload_registry, {})

def test_overload_registry_repeated(self):
for _ in range(2):
impl, overloads = self.set_up_overloads()

self.assertEqual(list(get_overloads(impl)), overloads)


# Definitions needed for features introduced in Python 3.6

Expand Down
34 changes: 34 additions & 0 deletions Lib/typing.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

from abc import abstractmethod, ABCMeta
import collections
from collections import defaultdict
import collections.abc
import contextlib
import functools
Expand Down Expand Up @@ -121,9 +122,11 @@ def _idfunc(_, x):
'assert_type',
'assert_never',
'cast',
'clear_overloads',
'final',
'get_args',
'get_origin',
'get_overloads',
'get_type_hints',
'is_typeddict',
'LiteralString',
Expand Down Expand Up @@ -2436,6 +2439,10 @@ def _overload_dummy(*args, **kwds):
"by an implementation that is not @overload-ed.")


# {module: {qualname: {firstlineno: func}}}
_overload_registry = defaultdict(functools.partial(defaultdict, dict))


def overload(func):
"""Decorator for overloaded functions/methods.

Expand All @@ -2461,10 +2468,37 @@ def utf8(value: bytes) -> bytes: ...
def utf8(value: str) -> bytes: ...
def utf8(value):
# implementation goes here

The overloads for a function can be retrieved at runtime using the
get_overloads() function.
"""
# classmethod and staticmethod
f = getattr(func, "__func__", func)
try:
_overload_registry[f.__module__][f.__qualname__][f.__code__.co_firstlineno] = func
except AttributeError:
# Not a normal function; ignore.
pass
return _overload_dummy


def get_overloads(func):
"""Return all defined overloads for *func* as a sequence."""
# classmethod and staticmethod
f = getattr(func, "__func__", func)
if f.__module__ not in _overload_registry:
return []
mod_dict = _overload_registry[f.__module__]
if f.__qualname__ not in mod_dict:
return []
return list(mod_dict[f.__qualname__].values())


def clear_overloads():
"""Clear all overloads in the registry."""
_overload_registry.clear()
JelleZijlstra marked this conversation as resolved.
Show resolved Hide resolved


def final(f):
"""A decorator to indicate final methods and final classes.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Add :func:`typing.get_overloads` and :func:`typing.clear_overloads`.
Patch by Jelle Zijlstra.