Fine grained crash when class is turned into a generic class

[JukkaL]

a.py:

import b
def f() -> b.C: pass

Original b.py:

import a
class C: pass

Updated b.py:

from typing import TypeVar, Generic
import a

T = TypeVar('T')
class C(Generic[T]): pass

reveal_type(a.f)
c: C[int]
d = a.f()
d = c
c = d

Now if I run mypy --quick a.py first with the original files, then with the updated b.py, I get a crash:

$ mypy --quick --show-traceback a.py
b.py:7: error: Revealed type is 'def () -> b.C'
/Users/jukka/src/mypy/b.py:11: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.510-dev-c856a53fe06c0ad9193d86ab35fb86693940f273-dirty
Traceback (most recent call last):
  File "/Users/jukka/src/mypy/scripts/mypy", line 6, in <module>
    main(__file__)
  File "/Users/jukka/src/mypy/mypy/main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/jukka/src/mypy/mypy/main.py", line 93, in type_check_only
    options=options)
  File "/Users/jukka/src/mypy/mypy/build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "/Users/jukka/src/mypy/mypy/build.py", line 1570, in dispatch
    process_graph(graph, manager)
  File "/Users/jukka/src/mypy/mypy/build.py", line 1813, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/jukka/src/mypy/mypy/build.py", line 1912, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/jukka/src/mypy/mypy/build.py", line 1485, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/Users/jukka/src/mypy/mypy/checker.py", line 177, in check_first_pass
    self.accept(d)
  File "/Users/jukka/src/mypy/mypy/checker.py", line 262, in accept
    stmt.accept(self)
  File "/Users/jukka/src/mypy/mypy/nodes.py", line 853, in accept
    return visitor.visit_assignment_stmt(self)
  File "/Users/jukka/src/mypy/mypy/checker.py", line 1177, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
  File "/Users/jukka/src/mypy/mypy/checker.py", line 1239, in check_assignment
    rvalue_type = self.check_simple_assignment(lvalue_type, rvalue, lvalue)
  File "/Users/jukka/src/mypy/mypy/checker.py", line 1698, in check_simple_assignment
    rvalue_type = self.expr_checker.accept(rvalue, lvalue_type)
  File "/Users/jukka/src/mypy/mypy/checkexpr.py", line 2056, in accept
    typ = node.accept(self)
  File "/Users/jukka/src/mypy/mypy/nodes.py", line 1213, in accept
    return visitor.visit_name_expr(self)
  File "/Users/jukka/src/mypy/mypy/checkexpr.py", line 123, in visit_name_expr
    return self.narrow_type_from_binder(e, result)
  File "/Users/jukka/src/mypy/mypy/checkexpr.py", line 2271, in narrow_type_from_binder
    ans = narrow_declared_type(known_type, restriction)
  File "/Users/jukka/src/mypy/mypy/meet.py", line 46, in narrow_declared_type
    return meet_types(declared, narrowed)
  File "/Users/jukka/src/mypy/mypy/meet.py", line 25, in meet_types
    return t.accept(TypeMeetVisitor(s))
  File "/Users/jukka/src/mypy/mypy/types.py", line 381, in accept
    return visitor.visit_instance(self)
  File "/Users/jukka/src/mypy/mypy/meet.py", line 198, in visit_instance
    args.append(self.meet(t.args[i], si.args[i]))
IndexError: list index out of range
/Users/jukka/src/mypy/b.py:11: note: use --pdb to drop into pdb

[JukkaL]

I can also cause a crash if I switch a generic class into a non-generic one. However, I need to switch the order of assignments in the second version:

...
reveal_type(a.f)
c: C[int]
d = a.f()
c = d   # this comes first now
d = c

[ilevkivskyi]

I think this could be also a symptom of another problem: there are many places where e.g. named_type('builtins.tuple') is used instead of named_generic_type('builtins.tuple', [AnyType()]). So that many types with wrong type argument count are created. This leads to crashes like #3117. I think that ideally we could "fix" all such code, this process is started in #3129 (also in protocols PR I do the same for typing.Awaitable).

[JukkaL]

I think this could be also a symptom of another problem: there are many places where e.g. named_type('builtins.tuple') is used instead of named_generic_type('builtins.tuple', [AnyType()]).

These are all wrong. We should only create Instance types with the correct number of type arguments, and crash hard if we try to create an bad type.

[ilevkivskyi]

These are all wrong. We should only create Instance types with the correct number of type arguments, and crash hard if we try to create an bad type.

OK, then a possible "robust" solution would be to remove named_type and only use named_generic_type that will check argument count on Instance creation. (We cannot make this check in Instance.__init__, since type argument counts are not known until third pass.)

[ilevkivskyi]

The quick mode was deprecated and is now removed, but now the same crash happens in the fine-grained mode. (Same files and the same traceback.)

[ilevkivskyi]

(Removed the incremental label, since the crash doesn't happen in the normal incremental mode.)

[ilevkivskyi]

Here is the full test case:

[case testFormerQuick]
import a
[file a.py]
import b
def f() -> b.C: pass
[file b.py]
import a
class C: pass
[file b.py.2]
from typing import TypeVar, Generic
import a

T = TypeVar('T')
class C(Generic[T]): pass

reveal_type(a.f)
c: C[int]
d = a.f()
d = c
c = d
[out]
==
tmp/b.py:7: error: Revealed type is 'def () -> b.C[Any]'