How do I type a generic factory function?

[njsmith]

[Not sure if this is the right place to file issues like this, feel free to tell me to go somewhere else :-)]

I have two generic types:

class SendChannel(Generic[T]):
    ...

class ReceiveChannel(Generic[T]):
    ...

I have a factory function that creates a linked pair of these objects:

def open_channel_pair(buffer_size: int) -> Tuple[SendChannel[T], ReceiveChannel[T]]:
    # ... construct some objects ...
    return (s, r)

I would like to allow people to call it like:

s, r = open_channel_pair[str](0)
# Now 's' is a SendChannel[str], and 'r' is a ReceiveChannel[str]

This works for the factory callables built into the language, e.g. calling SendChannel[str]() returns an object of type SendChannel[str]. This is basically the same thing, but since I have to construct two objects in a single operation, the regular thing doesn't work.

I did come up with a sneaky hack:

# Complete cut-and-pasteable example
from typing import Generic, TypeVar, Tuple

T = TypeVar("T")

class SendChannel(Generic[T]):
    pass

class ReceiveChannel(Generic[T]):
    pass

class open_channel_pair(Tuple[SendChannel[T], ReceiveChannel[T]]):
    def __new__(self, buffer_size: int):
        return (SendChannel[T](), ReceiveChannel[T]())

    # Note: this function is unreachable, but we have to define it, and keep
    # its signature in sync with __new__'s signature, to make mypy happy.
    def __init__(self, buffer_size: int) -> None:
        assert False

s, r = open_channel_pair[str](0)
# Confirms that it works:
reveal_type(s)
reveal_type(r)

This is gross, though. Is there any way to define a regular callable that has the same type as a generic class constructor, but is less gross than this?

What even is the type of a generic class constructor? reveal_type(SendChannel) prints def [T] () -> SendChannel[T`1], which is a very strange looking type.

[ilevkivskyi]

This is not really mypy's problem. This just doesn't work at runtime. From mypy perspective it is totally fine. For example, as you can see in your initial version of the code:

s, r = open_channel_pair[str](0) # Type application is only supported for generic classes
reveal_type(s)  # Revealed type is 'SendChannel[builtins.str*]'
reveal_type(r)  # Revealed type is 'ReceiveChannel[builtins.str*]'

the first error is bogus, because it is actually supported in mypy, just not at runtime. I think we should just remove this error, so that one can at least use cast('open_channel_pair[str]', open_channel_pair)(0).

Later we can support a safer version of cast for such cases (we already discussed this), or you can try discussion whether this can be allowed at runtime on CPython forums (a trivial __getitem__ returning self on types.FunctionType).

What even is the type of a generic class constructor? reveal_type(SendChannel) prints def [T] () -> SendChannel[T`1], which is a very strange looking type

What is wrong with this type? It looks totally fine to me.

[njsmith]

I think we should just remove this error, so that one can at least use cast('open_channel_pair[str]', open_channel_pair)(0).

It seems like this might need an update to PEP 484? And ideally some more details on exactly what the semantics are :-).

or you can try discussion whether this can be allowed at runtime on CPython forums (a trivial __getitem__ returning self on types.FunctionType).

It's also easy to hack a __getitem__ in without changing the language:

class generic_function:
    def __init__(self, fn):
        self._fn = fn

    def __call__(self, *args, **kwargs):
        return self._fn(*args, **kwargs)

    def __getitem__(self, _):
        return self

@generic_function
def open_channel_pair(...):
    ...

...except that then I can't figure out how to type generic_function without breaking mypy further.

I tried the blunt hammer of generic_function = cast("Callable[[T], T]", _generic_function) (i.e. "this decorator just returns whatever it gets, please ignore it entirely"), but mypy doesn't ignore it – I get:

error: Value of type "Callable[[int], Tuple[SendChannel[T], ReceiveChannel[T]]]" is not indexable

and both revealed types are Any. So apparently passing the magic generic function type through the identity function strips off the magic.

What is wrong with this type? It looks totally fine to me.

It's not... type syntax? Like I can definitely not write my_factory: def [T] () -> SendChannel[T`1] = ..., right? I guess I could stick it in a string, but I don't remember seeing anything like that in PEP 484...

I guess in PEP 484 it is normal that you can't point to a function and say "what's the type of this?" (e.g., anything with keyword arguments also has an inexpressible type). It's just not what what other type systems have trained me to expect.

[njsmith]

I found another way to write this that works:

class open_channel_pair_type:
    def __call__(self, buffer_size: int) -> Tuple[SendChannel[Any], ReceiveChannel[Any]]:
        # ... insert real implementation here ...
        return (SendChannel(), ReceiveChannel())

    def __getitem__(self, _: Type[T]) -> Callable[[int], Tuple[SendChannel[T], ReceiveChannel[T]]]:
        return self

open_channel_pair = open_channel_pair_type()

I guess this is less gross than abusing __new__, though it still has a lot of repetition, and is going to get even wordier if we ever make buffer_size optional or add kwonly arguments.

[ilevkivskyi]

I guess in PEP 484 it is normal that you can't point to a function and say "what's the type of this?" (e.g., anything with keyword arguments also has an inexpressible type). It's just not what what other type systems have trained me to expect.

The syntax of reveal_type() is currently not standardized in any way (and unfortunately not documented). There is a proposal python/typing#277, but it didn't get any traction.

It's also easy to hack a __getitem__ in without changing the language:

Requiring a decorator on every generic function is a big change. We can of course clearly document that it is optional, and is only required if one wants to subscript the function. In any case, I think this requires a wider discussion.

[njsmith]

Requiring a decorator on every generic function is a big change. We can of course clearly document that it is optional, and is only required if one wants to subscript the function. In any case, I think this requires a wider discussion.

Oh yeah, I just meant to use it on that subset of generic functions where you explicitly specify the type when calling. Not sure if there's a name for that.

Is there a better place to have this discussion for people to see it, or...?

[gvanrossum]

Typing-sig?

On Tue, Dec 18, 2018 at 6:26 PM Nathaniel J. Smith ***@***.***> wrote: Requiring a decorator on every generic function is a big change. We can of course clearly document that it is optional, and is only required if one wants to subscript the function. In any case, I think this requires a wider discussion. Oh yeah, I just meant to use it on that subset of generic functions where you explicitly specify the type when calling. Not sure if there's a name for that. Is there a better place to have this discussion for people to see it, or...? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6073 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMuV-BKNUs4r2bWxJl_4xfT52BtY7ks5u6aPdgaJpZM4ZUiwa> .

-- --Guido (mobile)

[oremanj]

@njsmith 's most recent suggestion doesn't work for "special forms", because they can't be inferred as the T in Type[T].. Not sure if this is a bug in mypy or intended behavior.

from typing import Generic, TypeVar, List, Tuple, Callable, Any, Type, Awaitable

T = TypeVar("T")

class SendChannel(Generic[T]): ...
class ReceiveChannel(Generic[T]): ...

class open_channel_pair_type:
    def __call__(self, buffer_size: int) -> Tuple[SendChannel[Any], ReceiveChannel[Any]]:
        # ... insert real implementation here ...
        return (SendChannel(), ReceiveChannel())

    def __getitem__(self, _: Type[T]) -> Callable[[int], Tuple[SendChannel[T], ReceiveChannel[T]]]:
        return self

open_channel_pair = open_channel_pair_type()

reveal_type(open_channel_pair[int](5))
reveal_type(open_channel_pair[List[int]](10))
reveal_type(open_channel_pair[Tuple[int, str]](10))
reveal_type(open_channel_pair[Callable[[], Awaitable[None]]](10))

shows that int and List[int] work, but the tuple and callable forms don't.

t.py:18: error: Revealed type is 'Tuple[t.SendChannel[builtins.int*], t.ReceiveChannel[builtins.int*]]'
t.py:19: error: Revealed type is 'Tuple[t.SendChannel[builtins.list*[builtins.int*]], t.ReceiveChannel[builtins.list*[builtins.int*]]]'
t.py:20: error: Revealed type is 'Tuple[t.SendChannel[Any], t.ReceiveChannel[Any]]'
t.py:21: error: Revealed type is 'Tuple[t.SendChannel[Any], t.ReceiveChannel[Any]]'

[njsmith]

@oremanj Oh, that is a subtle flaw indeed, nice catch.

I just tried it, and it looks like my original version (the class open_channel_pair(Tuple[...]) one) does work in all your tricky cases, so that's still an option.

@ilevkivskyi's trick of simply writing def open_channel_pair() -> Tuple[...] also works with these tricky cases, if we ignore the "Type application is only supported for generic classes" error.

Probably someone should start a discussion about this on typing-sig, basically suggesting making the def version legal (probably with some decorator to allow the __getitem__ syntax). Unfortunately I've been feeling exhausted at the idea of signing up for yet another mailing list just to have one conversation :-/. It'd be easier if it were a category on discourse, though I don't suppose it will change just for me! I'll get around to it eventually...