Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support self-types containing ParamSpec #15903

Merged
merged 2 commits into from
Aug 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 14 additions & 20 deletions mypy/typeops.py
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ class B(A): pass
return cast(F, func)
self_param_type = get_proper_type(func.arg_types[0])

variables: Sequence[TypeVarLikeType] = []
variables: Sequence[TypeVarLikeType]
if func.variables and supported_self_type(self_param_type):
from mypy.infer import infer_type_arguments

Expand All @@ -312,46 +312,40 @@ class B(A): pass
original_type = erase_to_bound(self_param_type)
original_type = get_proper_type(original_type)

all_ids = func.type_var_ids()
# Find which of method type variables appear in the type of "self".
self_ids = {tv.id for tv in get_all_type_vars(self_param_type)}
self_vars = [tv for tv in func.variables if tv.id in self_ids]

# Solve for these type arguments using the actual class or instance type.
typeargs = infer_type_arguments(
func.variables, self_param_type, original_type, is_supertype=True
self_vars, self_param_type, original_type, is_supertype=True
)
if (
is_classmethod
# TODO: why do we need the extra guards here?
and any(isinstance(get_proper_type(t), UninhabitedType) for t in typeargs)
and isinstance(original_type, (Instance, TypeVarType, TupleType))
):
# In case we call a classmethod through an instance x, fallback to type(x)
# In case we call a classmethod through an instance x, fallback to type(x).
typeargs = infer_type_arguments(
func.variables, self_param_type, TypeType(original_type), is_supertype=True
self_vars, self_param_type, TypeType(original_type), is_supertype=True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to check my intuition here: I presume the speedup is decreasing the amount of typevars inferred here? (And a couple lines above)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, decreasing number of type vars to solve for will speed things up slightly, but the main speed-up is from not using a nested function in relatively hot code. (Which especially affects mypyc, that anecdotally emits inefficient code for nested functions).

)

ids = [tid for tid in all_ids if any(tid == t.id for t in get_type_vars(self_param_type))]

# Technically, some constrains might be unsolvable, make them <nothing>.
# Update the method signature with the solutions found.
# Technically, some constraints might be unsolvable, make them <nothing>.
to_apply = [t if t is not None else UninhabitedType() for t in typeargs]

def expand(target: Type) -> Type:
return expand_type(target, {id: to_apply[all_ids.index(id)] for id in ids})

arg_types = [expand(x) for x in func.arg_types[1:]]
ret_type = expand(func.ret_type)
variables = [v for v in func.variables if v.id not in ids]
func = expand_type(func, {tv.id: arg for tv, arg in zip(self_vars, to_apply)})
variables = [v for v in func.variables if v not in self_vars]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing this whole "expand a type with certain type vars then remove those certain type vars" dance feels weird. Is there a reason expand_type doesn't remove the replaced type variables?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it is time for type theory lesson. There are two kinds of things that people usually call "generic types": type constructors, and polymorphic types. This terminology is still vague (and I am not sure it is standard), so it is better to explain on examples. For example, Callable is a type constructor, by itself it is not a valid type, it expects some type arguments, so e.g. Callable[[int], int] is a valid type. Now when you write:

def foo(x: T) -> T: ...

assuming T is not bound in current scope, this defines a polymorphic type for foo. Such type is valid by itself and does not expect any type arguments. You can imagine it as an infinite intersection or an infinite overload forall T . Callable[[T], T].

(An implementation note: In an ideal world variables should not be a property of CallableType, we would have ForAllType(variables, template), and there would be no fallback in callable types, in an ideal world type of foo would be ForAllType([T], IntersectionType(Callable[[T], T], Instance("types.FunctionType"))), but we are not living in an ideal world.)

If we are dealing with type constructor (in particular with generic type aliases), we never need to "remove certain type vars" (what would it even mean). The only thing we should do is to formally substitute type parameters expected by constructor with type arguments we got. For example, with A[T] = Callable[[T], T] (it is sad that it is not a valid syntax, having explicit vars for type aliases would save a lot of ambiguity/confusion), and with A[int] we just do expand_type(Callable[[T], T], {T -> int}) = Callable[[int], int], end of story.

Now when we talk about type inference this usually involves several steps:

  1. Given an infinite overload (ranging over some type variables), figure out how types of arguments passed in CallExpr narrow down range of each type variable (this is called inferring constraints, done in constraints.py). For foo(42) this would be T :> int.
  2. Find the matching overload by solving those constraints (done in solve.py). In the above example T = int is best solution.
  3. Construct the type of matching overload. This is usually done in applytype.py and has three sub-steps:
    a. Validate that the solution found actually satisfies type variable bounds, values, etc.
    b. Call expand_type() on the r.h.s. of forall expression with solutions found.
    c. Update free variables on the l.h.s of forall expression (usually this means just remove some, but not always).

So what you are seeing here is some ad-hoc version of this logic. Or course it all can be refactored to reduce code duplication but FWIW I don't think it is a big deal in this case, since the amount of code duplication is really small here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And one more clarifying comments on why we don't have the validation step (3a) in the case of type constructor: this is because validation can be complete during semantic analysis, see semanal_typeargs.py.

else:
arg_types = func.arg_types[1:]
ret_type = func.ret_type
variables = func.variables

original_type = get_proper_type(original_type)
if isinstance(original_type, CallableType) and original_type.is_type_obj():
original_type = TypeType.make_normalized(original_type.ret_type)
res = func.copy_modified(
arg_types=arg_types,
arg_types=func.arg_types[1:],
arg_kinds=func.arg_kinds[1:],
arg_names=func.arg_names[1:],
variables=variables,
ret_type=ret_type,
bound_args=[original_type],
)
return cast(F, res)
Expand Down
42 changes: 42 additions & 0 deletions test-data/unit/check-selftype.test
Original file line number Diff line number Diff line change
Expand Up @@ -1973,3 +1973,45 @@ class B(A):
reveal_type(self.x.extra) # N: Revealed type is "builtins.int"
reveal_type(self.xs[0].extra) # N: Revealed type is "builtins.int"
[builtins fixtures/list.pyi]

[case testSelfTypesWithParamSpecExtract]
from typing import Any, Callable, Generic, TypeVar
from typing_extensions import ParamSpec

P = ParamSpec("P")
F = TypeVar("F", bound=Callable[..., Any])
class Example(Generic[F]):
def __init__(self, fn: F) -> None:
...
def __call__(self: Example[Callable[P, Any]], *args: P.args, **kwargs: P.kwargs) -> None:
...

def test_fn(a: int, b: str) -> None:
...

example = Example(test_fn)
example() # E: Missing positional arguments "a", "b" in call to "__call__" of "Example"
example(1, "b") # OK
[builtins fixtures/list.pyi]

[case testSelfTypesWithParamSpecInfer]
from typing import TypeVar, Protocol, Type, Callable
from typing_extensions import ParamSpec

R = TypeVar("R", covariant=True)
P = ParamSpec("P")
class AsyncP(Protocol[P]):
def meth(self, *args: P.args, **kwargs: P.kwargs) -> None:
...

class Async:
@classmethod
def async_func(cls: Type[AsyncP[P]]) -> Callable[P, int]:
...

class Add(Async):
def meth(self, x: int, y: int) -> None: ...

reveal_type(Add.async_func()) # N: Revealed type is "def (x: builtins.int, y: builtins.int) -> builtins.int"
reveal_type(Add().async_func()) # N: Revealed type is "def (x: builtins.int, y: builtins.int) -> builtins.int"
[builtins fixtures/classmethod.pyi]