Define functools.partial as overloaded function instead of its own class

[berdario]

So, I opened this issue 3 years ago, I see that some use cases are now handled, but there're still issues with properly checking simple snippets like:

from functools import partial

def inc(a:int) -> int:
    return a + 1

partial(inc, 1)(1)

(thanks for the all the work in Mypy and Typeshed since then, btw! I haven't touched much Python in the meanwhile, and it's good to see the improvements)

I thought a bit more about what can be done. One possibility could be to define different overloads for __init__ and __call__, like:

_T = TypeVar('_T')
_T2 = TypeVar('_T2')
_OT = TypeVar('_OT')


class partial(Generic[_T, _T2, _OT]):
    @overload
    def __init__(self, func: Callable[[_T], _OT], a: _T) -> None: ... 
    @overload
    def __init__(self, func: Callable[[_T, _T2], _OT], a: _T) -> None: ...
    @overload
    def __init__(self, func: Callable[[_T, _T2], _OT], a: _T, b: _T2) -> None: ...

    @overload
    def __call__(self) -> _OT: ...
    @overload
    def __call__(self, a: _T2) -> _OT: ...

But this would have the same problem with the snippet above, since there'd be no relationship between how the partial had been constructed (which __init__) had been invoked, and its __call__ (the wrong arity for the Callable could be invoked). Also, every partial would have to be parametrized on the number of types of the max arity supported by the stub (even for the types that aren't used)

Ideally, we'd want to "overload the whole class", something like:

@overload
class partial(Generic[_OT]:
    def __init__(self, func: Callable[[_T], _OT], a: _T) -> None: ...
    
    def __call__(self) -> _OT: ...


@overload
class partial(Generic[_T2, _OT]):
    @overload
    def __init__(self, func: Callable[[_T, _T2], _OT], a: _T) -> None: ...
    @overload
    def __init__(self, func: Callable[[_T, _T2], _OT], a: _T, b: _T2) -> None: ...

    @overload
    def __call__(self) -> _OT: ...
    @overload
    def __call__(self, a: _T2) -> _OT: ...

but this obviously cannot work.

This is when I thought of defining partial's type as a simple function.

Ideally we'd want to return a different class (with different type parameters) in each case. I don't think we'd be able to properly type that class, since the only 2 type constructors that I've seen with an arbitrary number of type parameters are Tuple and Callable (besides other stuff like Generic), and neither can be subclassed (though that's a detail that might change). It'd probably have to be defined as a SpecialForm

So, that's why I settled on it simply returning a Callable.

This is obviously not ideal, since this way we'd lose the ability to type-check inspections of func, args, keywords. But when people use functools.partial, that's to obtain some callables and call them, so I think the most common use case can actually be covered with the overloading, at the expense of these 3 other attributes.

I picked a maximum arity of 5, as that's also the arity used to type map (above it'll just rely on Any). This also won't help in the case of partial application with keyword arguments.

I don't particularly like this approach, and I'm sure that it had already been thought of when first implementing the current type stub for functools.partial, but I thought worthwhile to open this PR since I haven't seen anyone else proposing this.

On the other hand, I think this approach can also be justified by the other precedents we have, like itertools.cycle and itertools.tee, which are implemented as returning a custom type and yet are defined in Typeshed as simply being a function that returns an abstract type like Iterator (even if the actual implementation provides a couple more implementation details, namely {'__setstate__'} and {'__setstate__', '__copy__'} respectively)

Another advantage for doing things this way, is that it wouldn't require any special type checker implementation, so the stub could improve the current checking in both mypy and pytype.

Again: I'm not particularly fond of this change, so I understand if the current approach will continue to be preferred.

[JelleZijlstra]

Thanks for your PR, and sorry for the long wait!

I think this approach makes sense, but just have one comment.

[JukkaL]

Note that this PR generates a small number of false positives in our internal repositories since keyword arguments can no longer be used with the result of partial(). This is probably still worth it, as this seems to work in the vast majority of cases.

[jekbradbury]

This change also causes false positives in situations where functools.partial is used as a class (e.g. isinstance or subclassing). Not sure about the right way to handle that.

[JelleZijlstra]

With the new mypy release this also caused a couple of new errors in our codebase: two from isinstance() checks and a dozen or so more from keyword arguments. Perhaps it's better to undo this change, unfortunately. To get better types for partial we'll have to add a mypy plugin.

[wkschwartz]

This change caused errors in my code base where I had used partial's instance attributes func and keywords.