TypeVar to represent a Callable's arguments

[JelleZijlstra]

This is an idea for a new type system feature that can help more accurately type some decorators and wrapper functions, like @async() from https://github.com/quora/asynq.

I'd like to see something like this work:

from mypy_extensions import ArgumentsVar
from typing import TypeVar, Awaitable

_ArgsT = ArgumentsVar('_ArgsT')
_RetT = TypeVar('_RetT')

def make_awaitable(fn: Callable[_ArgsT, _RetT]) -> Callable[_ArgsT, Awaitable[_RetT]]:
    async def wrapper(**args: _ArgsT, **kwargs: _ArgsT) -> Awaitable[_RetT]:
        result = fn(*args, **kwargs)
        return result
    return wrapper

@make_awaitable
def f(x: int, y: str) -> int:
    return 3

reveal_type(f(1, 'x'))  # Awaitable[int]
f(1, 1)  # error, second argument must be str
f(1, z=3)  # error, z is not a valid kwarg

Having a function's args and kwargs annotated with an ArgumentsVar would mean that it takes the arguments that the ArgumentsVar resolves to. As an extension, we could also make something like def f(x: int, *args: _ArgsT, **kwargs: _ArgsT) -> _T: ... work to indicate that a function takes an argument called x plus the arguments specified in _ArgsT.

This could also improve some types in the standard library. For example, the annotations for functools.singledispatch currently don't check function arguments. With ArgumentsVar, it could be typed as follows:

_T = TypeVar('_T')
_ArgsT = ArgumentsVar('_ArgsT')

class _SingleDispatchCallable(Generic[_ArgsT, _T]):
    registry = ...  # type: Mapping[Any, Callable[_ArgsT, _T]]
    def dispatch(self, cls: Any) -> Callable[_ArgsT, _T]: ...
    @overload
    def register(self, cls: Any) -> Callable[[Callable[_ArgsT, _T]], Callable[_ArgsT, _T]]: ...
    @overload
    def register(self, cls: Any, func: Callable[_ArgsT, _T]) -> Callable[_ArgsT, _T]: ...
    def _clear_cache(self) -> None: ...
    def __call__(self, *args: _ArgsT, **kwargs: _ArgsT) -> _T: ...

def singledispatch(func: Callable[_ArgsT, _T]) -> _SingleDispatchCallable[_ArgsT, _T]: ...

This kind of thing has come up before, but I'm not sure a concrete solution has been proposed. I'll try to implement this to see how well it works.

[JelleZijlstra]

@gnprice helped me find a similar (but more general) past proposal from @sixolet: python/typing#239 (comment).

[sixolet]

Let me know if you want to coordinate regarding implementation. My next mypy related task is to get variadic type variables pull-request-worthy-then I was going to tackle arg vars.

…

On Mar 19, 2017, at 12:58 PM, Jelle Zijlstra ***@***.***> wrote: @gnprice helped me find a similar (but more general) past proposal from @sixolet: python/typing#239 (comment). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[gvanrossum]

Also relevant: #3157 (comment) (esp. "Argspec type variables")

[srittau]

One thing that came up in #3157 is that a hypothetical ArgumentsVar should not just replace the arguments in a callable, but you should be able to add to or consume arguments:

def my_decorator(f: Callable[[str, *_ArgsT], str]) -> Callable[[*_ArgsT, int], str]): ...

Maybe it would make sense to only allow ArgumentsVar inside Callable parameter lists and then drop the asterisk?

[Ran4]

Is this still an open problem?

[JelleZijlstra]

PEP 612 (https://www.python.org/dev/peps/pep-0612/) ended up providing a similar feature. I think we can close this issue now.

[sebastian-philipp]

For future reference. The related issue for PEP 612 is: #8645

[BojieSheng]

Anyone know who to annotate below function "examle_a"?

def example_a(cls):
    class A(cls):
        def __init__(self, x, y):
            self.x=x
            self.y=y
    return A

cls can be any class.

[Dreamsorcerer]

Seems like a bit of an anti-pattern, I think the closest you'll get is:

T = TypeVar("T")
def example_a(cls: type[T]) -> type[T]:

Maybe with intersections (if/when supported), you might be able to something like this in future:

class Sub(Protocol):
    x: Any
    y: Any

def example_a(cls: type[T]) -> Intersection[type[T], type[Sub]]

But, I wouldn't count on it working even after intersections are introduced.

[BojieSheng]

Seems like a bit of an anti-pattern, I think the closest you'll get is:

T = TypeVar("T")
def example_a(cls: type[T]) -> type[T]:

Maybe with intersections (if/when supported), you might be able to something like this in future:

class Sub(Protocol):
    x: Any
    y: Any

def example_a(cls: type[T]) -> Intersection[type[T], type[Sub]]

But, I wouldn't count on it working even after intersections are introduced.

Dreamsorcerer, many thanks for your answer! However the return is a subclass of type[T], do you think the return type should be changed?

T = TypeVar("T")
def example_a(cls: type[T]) -> ???:

I do not quite understand your second suggestion, can. you please explain a little bit further?

[Dreamsorcerer]

Dreamsorcerer, many thanks for your answer! However the return is a subclass of type[T], do you think the return type should be changed?

The subtype doesn't exist at static time, so there isn't a more precise type you can use. A subclass is still of type T, so it is a correct annotation, but you won't be able to use any of the subclass things without a type error. As I said, this seems like an antipattern; without knowing the type, mypy can't tell you if the subclass is even correct (are you trying to overload attributes/methods that are incompatible with the superclass, for example).

I do not quite understand your second suggestion, can. you please explain a little bit further?

I'm just saying that it might be possible (some day) to define an intersection between the superclass and a Protocol (i.e. something that is defined as supporting the behaviour of both). But, intersections are not supported yet, and I have no idea if they will actually work in this case.

[BojieSheng]

Really appreciate for your support, it works!