Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong types and crash with polymorphic Callable #10244

Closed
andriusstank opened this issue Mar 25, 2021 · 1 comment · Fixed by #14093
Closed

Wrong types and crash with polymorphic Callable #10244

andriusstank opened this issue Mar 25, 2021 · 1 comment · Fixed by #14093

Comments

@andriusstank
Copy link

Crash Report

First of all, the code that makess mypy crash:

from dataclasses import dataclass
from typing import Generic, TypeVar, Callable

T = TypeVar('T')
R = TypeVar('R')
X = TypeVar('X')

@dataclass
class Box(Generic[T]):
    inner: T

@dataclass
class Cont(Generic[R]):
    run: Box[Callable[[X], R]]

def f(c: Cont[R]) -> R:
    return c.run.inner(3)

def const_two(x: T) -> str:
    return "two"

c = Cont(Box(const_two))

Adding type annotation at the last line c: Cont[str] = Cont(Box(const_two)) prevents the crash.

There's another problem - mypy gets types in function f wrong. Here c.run.inner returns type R, while mypy insists it must by int.

17: error: Incompatible return value type (got "int", expected "R")

Traceback

Traceback (most recent call last):
  File "/home/as/miniconda3/envs/debug_mypy/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/main.py", line 98, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/build.py", line 179, in build
    result = _build(
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/build.py", line 253, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/build.py", line 2688, in dispatch
    process_graph(graph, manager)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/build.py", line 3012, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/build.py", line 3110, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/build.py", line 2164, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checker.py", line 294, in check_first_pass
    self.accept(d)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/nodes.py", line 1073, in accept
    return visitor.visit_assignment_stmt(self)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checker.py", line 2010, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checker.py", line 2146, in check_assignment
    rvalue_type = self.expr_checker.accept(rvalue)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 3902, in accept
    typ = node.accept(self)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/nodes.py", line 1558, in accept
    return visitor.visit_call_expr(self)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 270, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 352, in visit_call_expr_inner
    ret_type = self.check_call_expr_with_callee_type(callee_type, e, fullname,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 857, in check_call_expr_with_callee_type
    return self.check_call(callee_type, e.args, e.arg_kinds, e,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 916, in check_call
    return self.check_callable_call(callee, args, arg_kinds, context, arg_names,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 1004, in check_callable_call
    callee = self.infer_function_type_arguments(
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 1207, in infer_function_type_arguments
    arg_types = self.infer_arg_types_in_context(
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 1106, in infer_arg_types_in_context
    res[ai] = self.accept(args[ai], callee.arg_types[i])
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 3902, in accept
    typ = node.accept(self)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/nodes.py", line 1558, in accept
    return visitor.visit_call_expr(self)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 270, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 352, in visit_call_expr_inner
    ret_type = self.check_call_expr_with_callee_type(callee_type, e, fullname,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 857, in check_call_expr_with_callee_type
    return self.check_call(callee_type, e.args, e.arg_kinds, e,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 916, in check_call
    return self.check_callable_call(callee, args, arg_kinds, context, arg_names,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 1002, in check_callable_call
    callee = self.infer_function_type_arguments_using_context(
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/checkexpr.py", line 1176, in infer_function_type_arguments_using_context
    args = infer_type_arguments(callable.type_var_ids(), ret_type, erased_ctx)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/infer.py", line 46, in infer_type_arguments
    return solve_constraints(type_var_ids, constraints)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/solve.py", line 71, in solve_constraints
    elif is_subtype(bottom, top):
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/subtypes.py", line 93, in is_subtype
    return _is_subtype(left, right,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/subtypes.py", line 135, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/types.py", line 1153, in accept
    return visitor.visit_callable_type(self)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/subtypes.py", line 299, in visit_callable_type
    return is_callable_compatible(
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/subtypes.py", line 839, in is_callable_compatible
    unified = unify_generic_callable(left, right, ignore_return=ignore_return)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/subtypes.py", line 1082, in unify_generic_callable
    applied = mypy.applytype.apply_generic_arguments(type, non_none_inferred_vars, report,
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/applytype.py", line 99, in apply_generic_arguments
    ret_type=expand_type(callable.ret_type, id_to_type),
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/expandtype.py", line 16, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/types.py", line 743, in accept
    return visitor.visit_erased_type(self)
  File "/home/as/miniconda3/envs/debug_mypy/lib/python3.9/site-packages/mypy/expandtype.py", line 82, in visit_erased_type
    raise RuntimeError()
RuntimeError: 

To Reproduce

Full source code is in the description.

Your Environment

  • Mypy version used: mypy 0.820+dev.641919f9496c8c62b9f45e7ec8402162667b7a11
  • Python version used: 3.9.2
  • Operating system and version: Fedora 32
@andriusstank
Copy link
Author

It no longer crashes with mypy 0.971, but type checking is still wrong. Interestingly, it can be worked around replacing Callable member by abstract method:

class Cont(Generic[R]):
    @abstractmethod
    def run(self, x: X) -> R:
        ...

def f(c: Cont[R]) -> R:
    reveal_type(c.run)
    return c.run(3)
16: note: Revealed type is "def [X] (x: X`1) -> R`-1"
Success: no issues found in 1 source file

I have also noticed that revealed type is different this time. Let's add reveal_type to the original snippet:

@dataclass
class Box(Generic[T]):
    inner: T

@dataclass
class Cont(Generic[R]):
    run: Box[Callable[[X], R]]

def f(c: Cont[R]) -> R:
    reveal_type(c.run.inner)
    return c.run.inner(3)
17: note: Revealed type is "def [X] (X`-1) -> R`-1"
18: error: Incompatible return value type (got "int", expected "R")
Found 1 error in 1 file (checked 1 source file)

Parameter type is X`1 in one case and X`-1 in another.

ilevkivskyi added a commit that referenced this issue Nov 16, 2022
Fixes #10244
Fixes #13515 

This fixes only the crash part, I am going to fix also the embarrassing
type variable clash in a separate PR, since it is completely unrelated
issue.

The crash happens because solver can call `is_suptype()` on the
constraint bounds, and those can contain `<Erased>`. Then if it is a
generic callable type (e.g. `def [S] (S) -> T` when used as a context is
erased to `def [S] (S) -> <Erased>`), `is_subtype()` will try unifying
them, causing the crash when applying unified arguments.

My fix is to simply allow subtyping between callable types that contain
`<Erased>`, we anyway allow checking subtpying between all other types
with `<Erased>` components. And this technically can be useful, e.g. `[T
<: DerivedGen1[<Erased>], T <: DerivedGen2[<Erased>]]` will be solved as
`T <: NonGenBase`.

Btw this crash technically has nothing to do with dataclasses, but it
looks like there is no other way in mypy to define a callable with
generic callable as argument type, if I try:
```python
def foo(x: Callable[[S], T]) -> T: ...
```
to repro the crash, mypy instead interprets `foo` as `def [S, T] (x:
Callable[[S], T]) -> T`, i.e. the argument type is not generic. I also
tried callback protocols, but they also don't repro the crash (at least
I can't find a repro), because protocols use variance for subtyping,
before actually checking member types.
ilevkivskyi added a commit that referenced this issue Nov 16, 2022
Addresses the non-crash part of #10244 (and similar situations).

The `freshen_function_type_vars()` use in `checkmember.py` was
inconsistent:
* It needs to be applied to attributes too, not just methods
* It needs to be a visitor, since generic callable can appear in a
nested position

The downsides are ~2% performance regression, and people will see more
large ids in `reveal_type()` (since refreshing functions uses a global
unique counter). But since this is a correctness issue that can cause
really bizarre error messages, I think it is totally worth it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants