Internal Error due to overloaded generic method?

[dbramel]

Bug Report

I ran into an internal error that seems to be related to type inference with an overloaded method. Apologies for the bland description but I don't know mypy internals well enough to debug further.

To Reproduce
Create a py file with the following code and run it through mypy with no arguments:

from typing import Any, Iterator, Optional, Protocol, TypeVar, Union, overload

KEY = TypeVar("KEY")
VAL = TypeVar("VAL", covariant=True)

_T = TypeVar("_T", contravariant=True)


class MappingProtocol(Protocol[KEY, VAL]):
    def __iter__(self) -> Iterator[KEY]:
        pass

    @overload
    def get(self, key: KEY) -> Optional[VAL]:
        ...

    @overload
    def get(self, key: KEY, default: Union[VAL, _T]) -> Union[VAL, _T]:
        ...

    def get(self, key: KEY, default: _T = None) -> Union[VAL, _T]:
        ...


def do_thing(stuff: MappingProtocol[KEY, VAL]) -> None:
    print(stuff)


def this_does_not_cause_mypy_to_crash() -> None:
    my_var: MappingProtocol[str, Any] = {"foo": lambda x: None}
    do_thing(my_var)


def this_causes_mypy_to_crash() -> None:
    do_thing({"foo": lambda x: None})

Running mypy on the file will crash at line 35 (in the this_causes_mypy_to_crash method) even on the current dev build:

dbramel:/usr/scratch/dbramel/local/mypy_bug$ mypy mypy_bug_example.py --show-traceback
mypy_bug_example.py:35: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.980+dev.80c09d56c080cad93847eeee50ddddf5e3abab06
Traceback (most recent call last):
  File "/usr/scratch/dbramel/local/ops3venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/main.py", line 95, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/main.py", line 174, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 186, in build
    result = _build(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 269, in _build
    graph = dispatch(sources, manager, stdout)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 2873, in dispatch
    process_graph(graph, manager)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 3257, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 3358, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/build.py", line 2300, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 461, in check_first_pass
    self.accept(d)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 569, in accept
    stmt.accept(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/nodes.py", line 810, in accept
    return visitor.visit_func_def(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 934, in visit_func_def
    self._visit_func_def(defn)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 938, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 1000, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 1186, in check_func_def
    self.accept(item.body)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 569, in accept
    stmt.accept(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/nodes.py", line 1206, in accept
    return visitor.visit_block(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 2415, in visit_block
    self.accept(s)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 569, in accept
    stmt.accept(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/nodes.py", line 1224, in accept
    return visitor.visit_expression_stmt(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checker.py", line 3906, in visit_expression_stmt
    expr_type = self.expr_checker.accept(s.expr, allow_none_return=True, always_allow_any=True)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 4622, in accept
    typ = self.visit_call_expr(node, allow_none_return=True)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 386, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 506, in visit_call_expr_inner
    ret_type = self.check_call_expr_with_callee_type(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1156, in check_call_expr_with_callee_type
    ret_type, callee_type = self.check_call(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1239, in check_call
    return self.check_callable_call(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1379, in check_callable_call
    callee = self.infer_function_type_arguments(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1678, in infer_function_type_arguments
    arg_types = self.infer_arg_types_in_context(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1573, in infer_arg_types_in_context
    res[ai] = self.accept(args[ai], callee.arg_types[i])
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 4630, in accept
    typ = node.accept(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/nodes.py", line 2121, in accept
    return visitor.visit_dict_expr(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 4105, in visit_dict_expr
    rv = self.check_call(constructor, args, [nodes.ARG_POS] * len(args), e)[0]
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1239, in check_call
    return self.check_callable_call(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1378, in check_callable_call
    callee = self.infer_function_type_arguments_using_context(callee, context)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 1644, in infer_function_type_arguments_using_context
    args = infer_type_arguments(callable.type_var_ids(), ret_type, erased_ctx)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/infer.py", line 68, in infer_type_arguments
    constraints = infer_constraints(template, actual, SUPERTYPE_OF if is_supertype else SUBTYPE_OF)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/constraints.py", line 163, in infer_constraints
    return _infer_constraints(template, actual, direction)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/constraints.py", line 243, in _infer_constraints
    return template.accept(ConstraintBuilderVisitor(actual, direction))
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/types.py", line 1257, in accept
    return visitor.visit_instance(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/constraints.py", line 734, in visit_instance
    and mypy.subtypes.is_protocol_implementation(erased, instance)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 1005, in is_protocol_implementation
    is_compat = is_subtype(subtype, supertype, ignore_pos_arg_names=ignore_names)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 178, in is_subtype
    return _is_subtype(left, right, subtype_context, proper_subtype=False)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 328, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/types.py", line 2056, in accept
    return visitor.visit_overloaded(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 829, in visit_overloaded
    ) or is_callable_compatible(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 1327, in is_callable_compatible
    unified = unify_generic_callable(left, right, ignore_return=ignore_return)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/subtypes.py", line 1626, in unify_generic_callable
    applied = mypy.applytype.apply_generic_arguments(
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/applytype.py", line 113, in apply_generic_arguments
    arg_types = [expand_type(at, id_to_type) for at in callable.arg_types]
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/applytype.py", line 113, in <listcomp>
    arg_types = [expand_type(at, id_to_type) for at in callable.arg_types]
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/types.py", line 1125, in accept
    return visitor.visit_erased_type(self)
  File "/usr/scratch/dbramel/local/ops3venv/lib/python3.8/site-packages/mypy/expandtype.py", line 134, in visit_erased_type
    raise RuntimeError()
RuntimeError: 
mypy_bug_example.py:35: : note: use --pdb to drop into pdb

Expected Behavior
Successfully run and tell me if I have any lint errors.

Actual Behavior
Crashes with Internal Error

Environment

• mypy version used: 0.980+dev.80c09d56c080cad93847eeee50ddddf5e3abab06
• mypy command-line flags: none
• mypy config options from mypy.ini: none
• Python version: 3.8.7
• Operating system: GNU/Linux

[AlexWaygood]

Slightly more minimal repro:

from __future__ import annotations
from typing import Protocol, TypeVar, overload

_KT_contra = TypeVar("_KT_contra", contravariant=True)
_VT_co = TypeVar("_VT_co", covariant=True)
_T = TypeVar("_T")

class MappingProtocol(Protocol[_KT_contra, _VT_co]):
    @overload
    def get(self, key: _KT_contra) -> _VT_co | None: ...
    @overload
    def get(self, key: _KT_contra, default: _VT_co | _T) -> _VT_co | _T: ...

def do_thing(stuff: MappingProtocol[_KT_contra, _VT_co]) -> None: ...
do_thing({"foo": None})

https://mypy-play.net/?mypy=latest&python=3.10&flags=strict&gist=cebf8bc0e851ea6fa3b52d6d3639096d

[sobolevn]

Thanks! I will take a look today.

[sobolevn]

So, I was finally able to dig into this issue. What happens?

1. erased_ctx = replace_meta_vars(ctx, ErasedType()) make all type vars ErasedType
2. applied = mypy.applytype.apply_generic_arguments( later calls expand_type with ErasedType inside
3. It fails

But, the question is: why does it happen? Right now I still have no idea.