Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal Error due to overloaded generic method? #13515

Closed
dbramel opened this issue Aug 25, 2022 · 3 comments · Fixed by #14093
Closed

Internal Error due to overloaded generic method? #13515

dbramel opened this issue Aug 25, 2022 · 3 comments · Fixed by #14093

Comments

@dbramel
Copy link

dbramel commented Aug 25, 2022

Bug Report

I ran into an internal error that seems to be related to type inference with an overloaded method. Apologies for the bland description but I don't know mypy internals well enough to debug further.

To Reproduce
Create a py file with the following code and run it through mypy with no arguments:

from typing import Any, Iterator, Optional, Protocol, TypeVar, Union, overload

KEY = TypeVar("KEY")
VAL = TypeVar("VAL", covariant=True)

_T = TypeVar("_T", contravariant=True)


class MappingProtocol(Protocol[KEY, VAL]):
    def __iter__(self) -> Iterator[KEY]:
        pass

    @overload
    def get(self, key: KEY) -> Optional[VAL]:
        ...

    @overload
    def get(self, key: KEY, default: Union[VAL, _T]) -> Union[VAL, _T]:
        ...

    def get(self, key: KEY, default: _T = None) -> Union[VAL, _T]:
        ...


def do_thing(stuff: MappingProtocol[KEY, VAL]) -> None:
    print(stuff)


def this_does_not_cause_mypy_to_crash() -> None:
    my_var: MappingProtocol[str, Any] = {"foo": lambda x: None}
    do_thing(my_var)


def this_causes_mypy_to_crash() -> None:
    do_thing({"foo": lambda x: None})

Running mypy on the file will crash at line 35 (in the this_causes_mypy_to_crash method) even on the current dev build:

dbramel:/usr/scratch/dbramel/local/mypy_bug$ mypy mypy_bug_example.py --show-traceback
mypy_bug_example.py:35: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.980+dev.80c09d56c080cad93847eeee50ddddf5e3abab06
Traceback (most recent call last):
  File "/usr/scratch/dbramel/local/ops3venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/main.py", line 95, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/main.py", line 174, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 186, in build
    result = _build(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 269, in _build
    graph = dispatch(sources, manager, stdout)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 2873, in dispatch
    process_graph(graph, manager)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 3257, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 3358, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 2300, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 461, in check_first_pass
    self.accept(d)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 569, in accept
    stmt.accept(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/nodes.py", line 810, in accept
    return visitor.visit_func_def(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 934, in visit_func_def
    self._visit_func_def(defn)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 938, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 1000, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 1186, in check_func_def
    self.accept(item.body)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 569, in accept
    stmt.accept(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/nodes.py", line 1206, in accept
    return visitor.visit_block(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 2415, in visit_block
    self.accept(s)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 569, in accept
    stmt.accept(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/nodes.py", line 1224, in accept
    return visitor.visit_expression_stmt(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 3906, in visit_expression_stmt
    expr_type = self.expr_checker.accept(s.expr, allow_none_return=True, always_allow_any=True)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 4622, in accept
    typ = self.visit_call_expr(node, allow_none_return=True)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 386, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 506, in visit_call_expr_inner
    ret_type = self.check_call_expr_with_callee_type(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1156, in check_call_expr_with_callee_type
    ret_type, callee_type = self.check_call(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1239, in check_call
    return self.check_callable_call(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1379, in check_callable_call
    callee = self.infer_function_type_arguments(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1678, in infer_function_type_arguments
    arg_types = self.infer_arg_types_in_context(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1573, in infer_arg_types_in_context
    res[ai] = self.accept(args[ai], callee.arg_types[i])
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 4630, in accept
    typ = node.accept(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/nodes.py", line 2121, in accept
    return visitor.visit_dict_expr(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 4105, in visit_dict_expr
    rv = self.check_call(constructor, args, [nodes.ARG_POS] * len(args), e)[0]
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1239, in check_call
    return self.check_callable_call(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1378, in check_callable_call
    callee = self.infer_function_type_arguments_using_context(callee, context)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1644, in infer_function_type_arguments_using_context
    args = infer_type_arguments(callable.type_var_ids(), ret_type, erased_ctx)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/infer.py", line 68, in infer_type_arguments
    constraints = infer_constraints(template, actual, SUPERTYPE_OF if is_supertype else SUBTYPE_OF)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/constraints.py", line 163, in infer_constraints
    return _infer_constraints(template, actual, direction)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/constraints.py", line 243, in _infer_constraints
    return template.accept(ConstraintBuilderVisitor(actual, direction))
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/types.py", line 1257, in accept
    return visitor.visit_instance(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/constraints.py", line 734, in visit_instance
    and mypy.subtypes.is_protocol_implementation(erased, instance)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 1005, in is_protocol_implementation
    is_compat = is_subtype(subtype, supertype, ignore_pos_arg_names=ignore_names)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 178, in is_subtype
    return _is_subtype(left, right, subtype_context, proper_subtype=False)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 328, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/types.py", line 2056, in accept
    return visitor.visit_overloaded(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 829, in visit_overloaded
    ) or is_callable_compatible(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 1327, in is_callable_compatible
    unified = unify_generic_callable(left, right, ignore_return=ignore_return)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 1626, in unify_generic_callable
    applied = mypy.applytype.apply_generic_arguments(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/applytype.py", line 113, in apply_generic_arguments
    arg_types = [expand_type(at, id_to_type) for at in callable.arg_types]
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/applytype.py", line 113, in <listcomp>
    arg_types = [expand_type(at, id_to_type) for at in callable.arg_types]
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/types.py", line 1125, in accept
    return visitor.visit_erased_type(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/expandtype.py", line 134, in visit_erased_type
    raise RuntimeError()
RuntimeError: 
mypy_bug_example.py:35: : note: use --pdb to drop into pdb

Expected Behavior
Successfully run and tell me if I have any lint errors.

Actual Behavior
Crashes with Internal Error

Environment

  • mypy version used: 0.980+dev.80c09d56c080cad93847eeee50ddddf5e3abab06
  • mypy command-line flags: none
  • mypy config options from mypy.ini: none
  • Python version: 3.8.7
  • Operating system: GNU/Linux
@AlexWaygood
Copy link
Member

AlexWaygood commented Aug 25, 2022

Slightly more minimal repro:

from __future__ import annotations
from typing import Protocol, TypeVar, overload

_KT_contra = TypeVar("_KT_contra", contravariant=True)
_VT_co = TypeVar("_VT_co", covariant=True)
_T = TypeVar("_T")

class MappingProtocol(Protocol[_KT_contra, _VT_co]):
    @overload
    def get(self, key: _KT_contra) -> _VT_co | None: ...
    @overload
    def get(self, key: _KT_contra, default: _VT_co | _T) -> _VT_co | _T: ...

def do_thing(stuff: MappingProtocol[_KT_contra, _VT_co]) -> None: ...
do_thing({"foo": None})

https://mypy-play.net/?mypy=latest&python=3.10&flags=strict&gist=cebf8bc0e851ea6fa3b52d6d3639096d

@sobolevn sobolevn self-assigned this Aug 29, 2022
@sobolevn
Copy link
Member

Thanks! I will take a look today.

@sobolevn
Copy link
Member

So, I was finally able to dig into this issue. What happens?

  1. erased_ctx = replace_meta_vars(ctx, ErasedType()) make all type vars ErasedType
  2. applied = mypy.applytype.apply_generic_arguments( later calls expand_type with ErasedType inside
  3. It fails

But, the question is: why does it happen? Right now I still have no idea.

ilevkivskyi added a commit that referenced this issue Nov 16, 2022
Fixes #10244
Fixes #13515 

This fixes only the crash part, I am going to fix also the embarrassing
type variable clash in a separate PR, since it is completely unrelated
issue.

The crash happens because solver can call `is_suptype()` on the
constraint bounds, and those can contain `<Erased>`. Then if it is a
generic callable type (e.g. `def [S] (S) -> T` when used as a context is
erased to `def [S] (S) -> <Erased>`), `is_subtype()` will try unifying
them, causing the crash when applying unified arguments.

My fix is to simply allow subtyping between callable types that contain
`<Erased>`, we anyway allow checking subtpying between all other types
with `<Erased>` components. And this technically can be useful, e.g. `[T
<: DerivedGen1[<Erased>], T <: DerivedGen2[<Erased>]]` will be solved as
`T <: NonGenBase`.

Btw this crash technically has nothing to do with dataclasses, but it
looks like there is no other way in mypy to define a callable with
generic callable as argument type, if I try:
```python
def foo(x: Callable[[S], T]) -> T: ...
```
to repro the crash, mypy instead interprets `foo` as `def [S, T] (x:
Callable[[S], T]) -> T`, i.e. the argument type is not generic. I also
tried callback protocols, but they also don't repro the crash (at least
I can't find a repro), because protocols use variance for subtyping,
before actually checking member types.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants