functools' singledispatch does not support GenericAlias

[kumaraditya303]

BPO	46032
Nosy	@rhettinger, @ambv, @serhiy-storchaka, @miss-islington, @uriyyo, @kumaraditya303, @AlexWaygood
PRs	bpo-46032: Check types in singledispatch's register() at declaration time #30050 [3.10] bpo-46032: Check types in singledispatch's register() at declaration time (GH-30050) #30254 [3.9] bpo-46032: Check types in singledispatch's register() at declaration time (GH-30050) (GH-30254) #30255

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2021-12-26.12:23:57.239>
created_at = <Date 2021-12-10.09:04:56.137>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'docs']
title = "functools' singledispatch does not support GenericAlias"
updated_at = <Date 2021-12-26.12:23:57.239>
user = 'https://github.com/kumaraditya303'

bugs.python.org fields:

activity = <Date 2021-12-26.12:23:57.239>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2021-12-26.12:23:57.239>
closer = 'serhiy.storchaka'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2021-12-10.09:04:56.137>
creator = 'kumaraditya'
dependencies = []
files = []
hgrepos = []
issue_num = 46032
keywords = ['patch']
message_count = 9.0
messages = ['408179', '408181', '408204', '408205', '408306', '408348', '409169', '409170', '409197']
nosy_count = 7.0
nosy_names = ['rhettinger', 'lukasz.langa', 'serhiy.storchaka', 'miss-islington', 'uriyyo', 'kumaraditya', 'AlexWaygood']
pr_nums = ['30050', '30254', '30255']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46032'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-90190: Add doc for using singledispatch with precise collection type hints #116544
• [3.13] gh-90190: Add doc for using singledispatch with precise collection type hints (GH-116544) #124710
• [3.12] gh-90190: Add doc for using singledispatch with precise collection type hints (GH-116544) #124711

[kumaraditya303]

functools' singledispatch does not support GenericAlias

from functools import singledispatch

@singledispatch
def func(x):
    print("any")

@func.register
def _(x: list[str]):
    print("list[str]")


func(["a", "b"])

[AlexWaygood]

My opinion is that supporting GenericAlias here would be a bad idea. Associating an implementation of the function with the argument type list[str] is ambiguous. Would this implementation be called if any argument of type list was supplied, or would it only be called if all elements in the list were of type str?

The first option would be efficient, simple, and similar to the way singledispatch treats most other argument-types. However, it would be unintuitive.

The second option would be more intuitive, but could be extremely inefficient if a very long list was passed in. It would also make the code more complicated.

[AlexWaygood]

It would be well worth it to improve the error message, however:

>>> from functools import singledispatch
>>> @singledispatch
... def func(arg):
...     raise NotImplementedError
... 
>>> @func.register
... def _(arg: list[str]):
...     print('Got a list of strings')
... 
>>> func(1)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/functools.py", line 830, in dispatch
    impl = dispatch_cache[cls]
  File "/usr/local/lib/python3.9/weakref.py", line 405, in __getitem__
    return self.data[ref(key)]
KeyError: <weakref at 0x7f2a0d9141d0; to 'type' at 0x7f2a0e08b200 (int)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/functools.py", line 833, in dispatch
    impl = registry[cls]
KeyError: <class 'int'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/functools.py", line 877, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
    return dispatch(args[0].__class__)(*args, **kw)
  File "/usr/local/lib/python3.9/functools.py", line 835, in dispatch
    impl = _find_impl(cls, registry)
  File "/usr/local/lib/python3.9/functools.py", line 782, in _find_impl
    mro = _compose_mro(cls, registry.keys())
  File "/usr/local/lib/python3.9/functools.py", line 743, in _compose_mro
    types = [n for n in types if is_related(n)]
  File "/usr/local/lib/python3.9/functools.py", line 743, in <listcomp>
    types = [n for n in types if is_related(n)]
  File "/usr/local/lib/python3.9/functools.py", line 742, in is_related
    and issubclass(cls, typ))
TypeError: issubclass() argument 2 cannot be a parameterized generic

[AlexWaygood]

The above traceback is because the isinstance(list[str], type) check at Lib/functools.py:848 evaluates to True. Related: bpo-45665.

[serhiy-storchaka]

Yes, it is related to bpo-45665. It is a complicated case due to coincidence of several circumstances.

1. isinstance(list[int], type) is True, while isinstance(typing.List[int], type) is False. list[int] is considered a type in this check.

2. list[int].__mro__ == list.__mro__, while typing.List[int] does not have the __mro__ attribute. list[int] is considered a type in this check.

3. issubclass(cls, list[int]) raises a TypeError (the same for typing.List[int]). list[int] cannot be used as a type here.

4. 2-argument registry() does not check the type of its first argument. f.registry(42, ...) is silently passed.

In 2-argument registry() typing.List[int] is passed due to (4) and ignored in dispatch() due to (2). list[int] is passed due to (4), but caused error due to (3).

In other uses of registry() (1-argument decorator factory and decorator with annotations) typing.List[int] is not passed due to 1. list[int] is passed due to (1) and caused error due to (3).

The proposed PR makes list[int] be treated the same way as typing.List[int]. It also makes 2-argument registry() rejecting invalid first argument, so all three forms of registry() accept and reject now the same types.

[AlexWaygood]

The PR looks good to me. I think it's also important that we document that these types aren't supported, as it's not mentioned anywhere at the moment. Related: bpo-34498.

[serhiy-storchaka]

New changeset 078abb6 by Serhiy Storchaka in branch 'main':
bpo-46032: Check types in singledispatch's register() at declaration time (GH-30050)
078abb6

[serhiy-storchaka]

New changeset 03c7449 by Serhiy Storchaka in branch '3.10':
[3.10] bpo-46032: Check types in singledispatch's register() at declaration time (GH-30050) (GH-30254)
03c7449

[serhiy-storchaka]

New changeset 25a12aa by Miss Islington (bot) in branch '3.9':
[3.9] bpo-46032: Check types in singledispatch's register() at declaration time (GH-30050) (GH-30254) (GH-30255)
25a12aa

[Delengowski]

It would be well worth it to improve the error message, however:

>>> from functools import singledispatch
>>> @singledispatch
... def func(arg):
...     raise NotImplementedError
... 
>>> @func.register
... def _(arg: list[str]):
...     print('Got a list of strings')
... 
>>> func(1)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/functools.py", line 830, in dispatch
    impl = dispatch_cache[cls]
  File "/usr/local/lib/python3.9/weakref.py", line 405, in __getitem__
    return self.data[ref(key)]
KeyError: <weakref at 0x7f2a0d9141d0; to 'type' at 0x7f2a0e08b200 (int)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/functools.py", line 833, in dispatch
    impl = registry[cls]
KeyError: <class 'int'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/functools.py", line 877, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
    return dispatch(args[0].__class__)(*args, **kw)
  File "/usr/local/lib/python3.9/functools.py", line 835, in dispatch
    impl = _find_impl(cls, registry)
  File "/usr/local/lib/python3.9/functools.py", line 782, in _find_impl
    mro = _compose_mro(cls, registry.keys())
  File "/usr/local/lib/python3.9/functools.py", line 743, in _compose_mro
    types = [n for n in types if is_related(n)]
  File "/usr/local/lib/python3.9/functools.py", line 743, in <listcomp>
    types = [n for n in types if is_related(n)]
  File "/usr/local/lib/python3.9/functools.py", line 742, in is_related
    and issubclass(cls, typ))
TypeError: issubclass() argument 2 cannot be a parameterized generic

I came across this today, and I was quite surprised I could not do this but then it made sense since we cannot do isinstance on GenericAlias. It certainly makes the idea of using functools.singledispatch less appealing. If I had 3.10 available to me I would just use match case.

I see there is still no documentation stating this is not supported. https://docs.python.org/3/library/functools.html#functools.singledispatch

Proposal, I'll create a PR adding an example with additional example of how to properly type hint it correctly? I'm not sure if there even is a way to properly type hint that a function takes a list[str] while still dispatching using that type hint and not just list

i.e.

@singledispatch
def func(x):
    print("any")

@func.register
def _(x: list[str]):
    print("list[str]")

raises error TypeError: Invalid annotation for 'a'. list[str] is not a class. but I'm not sure of a way to actually type hint that func should take a list[str] and not just a list.

[AlexWaygood]

If you want to use type hints with generics in a singledispatch function, I'd recommend doing something like this:

@singledispatch
def func(x):
    print("any")

@func.register(list)
def _(x: list[str]):
    print("list[str]")

(But note that that will dispatch on any call that passes a list, not just calls that pass lists where all items in the list are strs.)

A PR to improve the docs would be welcome. If you ping me on the PR, I can review it.