Type variables of generic returned callables not resolved/leaking

[alexd2580]

While trying to implement a typesafe @curry decorator which would enable partial application without writing any additional function-specific wrapper-code or other syntactic noise, I came across the problem of representing function-transformers using the python type system.

E.g.
Given two typed functions:

id(x) -> x
transform(f) -> f

I expect transform(id) to be typed and typesafe as well.

The problem arises with Generics and can be reproduced by typechecking the following snippet:

from typing import Callable, TypeVar

X = TypeVar("X")


def id_int(x: int) -> int:
    return x


def id_generic(x: X) -> X:
    return x


def id_f(x: Callable[[X], X]) -> Callable[[X], X]:
    return x


a: Callable[[int], int] = id_int
b: int = id_int(1)

# c: Callable[[X], X] = id_generic
d: int = id_generic(2)

# e: Callable[[Callable[[X], X]], Callable[[X], X]] = id_f

bound_id_int = id_f(id_int)
f: Callable[[int], int] = bound_id_int
g: int = bound_id_int(3)

bound_id_generic = id_f(id_generic)
# h: Callable[[X], X] = bound_id_generic
i: int = bound_id_generic(4) # ERROR HERE

Same as mypy, pyright cannot infer the type of X@id_generic to be int or Literal[4].

>>> pyright --version
pyright 1.1.126
>>> pyright callable_return_type.py
No configuration file found.
stubPath /[REDACTED] is not a valid directory.
Assuming Python platform Linux
Searching for source files
Found 1 source file
/[REDACTED]/callable_return_type.py
  /[REDACTED]/callable_return_type.py:32:27 - error: Argument of type "Literal[4]" cannot be assigned to parameter of type "X@id_generic"
    Type "Literal[4]" cannot be assigned to type "X@id_generic" (reportGeneralTypeIssues)
  /[REDACTED]/callable_return_type.py:32:10 - error: Expression of type "X@id_generic" cannot be assigned to declared type "int"
    "object" is incompatible with "int" (reportGeneralTypeIssues)
2 errors, 0 warnings, 0 infos 
Completed in 0.523sec
>>> mypy --version
mypy 0.812
>>> mypy callable_return_type.py
callable_return_type.py:32: error: Incompatible types in assignment (expression has type "X", variable has type "int")
callable_return_type.py:32: error: Argument 1 has incompatible type "int"; expected "X"
Found 2 errors in 1 file (checked 1 source file)

This issue might be related to #1639, with the exception that here we have a transformer that produces a generic function, like python/mypy#1317.

[erictraut]

You're encountering a limitation of the Python type system. Each type variable has a defined scope, as defined by the PEP 484 scoping rules. The type variable needs to be resolved within the context of its own scope. If the type variables are not resolved (i.e. replaced with an actual type) within a function call, they cannot be resolved later because they are now "out of scope". PEP 484 doesn't indicate what should happen with these variables. Pyright and mypy tend to "leak" them — that is, retain their identity — so you can tell where they came from and fix the problem. One could argue that they should be forcibly resolved to "Any", but that would mask the problem, so that has downsides.

One workaround is to take advantage of the fact that generic type aliases (in addition to generic functions and generic classes) can provide the scope for a type variable. Pyright has special logic to handle type generic type aliases that include Callable in their definition. This provides a potential workaround that appears to work fine in Pyright, but it still produces an error in mypy.

MyCallable = Callable[[X], X]

def id_f(x: MyCallable[X]) -> MyCallable[X]:
    return x

I'm going to give this more thought. Perhaps the special-case logic applied to Callable within a generic type alias should also be applied to Callable used within a generic function return type.

[erictraut]

I think it makes sense to include the same special-case logic to Callable return types that Pyright already applies when a generic type alias includes a Callable. With this change, your code example works as you expect it to.

This change will be included in the next release.

[erictraut]

This is included in pyright 1.1.127, which I just published. It will also be included in the next release of pylance.

[alexd2580]

Awesome!