mypy crashes with KeyError: 'django_stubs_ext' when using django-stubs 1.9.0

[pandabuilder]

Bug report

mypy crashes when reaching code with Django queries that annotate fields, which did not happen which django-stubs version 1.8.0.

What's wrong

viewer/models.py:951: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.910
viewer/models.py:951: : note: use --pdb to drop into pdb
Traceback (most recent call last):
  File "mypy/checkexpr.py", line 3911, in accept
  File "mypy/nodes.py", line 1558, in accept
  File "mypy/checkexpr.py", line 271, in visit_call_expr
  File "mypy/checkexpr.py", line 353, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 858, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 917, in check_call
  File "mypy/checkexpr.py", line 1029, in check_callable_call
  File "mypy/checkexpr.py", line 738, in apply_function_plugin
  File "/home/gitlab-runner/builds/b5691a72/0/acctest/proj/ENV/lib/python3.9/site-packages/mypy_django_plugin/transformers/querysets.py", line 231, in extract_proper_type_queryset_annotate
    annotated_type = get_or_create_annotated_type(api, model_type, fields_dict=fields_dict)
  File "/home/gitlab-runner/builds/b5691a72/0/acctest/proj/ENV/lib/python3.9/site-packages/mypy_django_plugin/transformers/models.py", line 447, in get_or_create_annotated_type
    model_module_file = api.modules[model_module_name]  # type: ignore
KeyError: 'django_stubs_ext'

How is that should be

I Just upgraded from 1.8.0 to 1.9.0, and it was working without errors before upgrading.

System information

• OS: Ubuntu 18.04.5 LTS
• python version: 3.9.1
• django version: 3.2.7
• mypy version: 0.910
• django-stubs version: 1.9.0

[sobolevn]

Please, check that django-stubs-ext is installed. It is a direct dependecy now: https://github.com/typeddjango/django-stubs/blob/master/setup.py#L26

[pandabuilder]

Forgot to mention that, django-stubs-ext was installed before, and it is installed now, using the latest version.

[sobolevn]

Just to double check: 0.3.0?

[sobolevn]

I would also highly appreciate a simple reproduction. In this case I will be able to fix this today / during weekend.

[pandabuilder]

Yes it is running with 0.3.0. I'll try to replicate it with a barebones project and post the steps here.

[pandabuilder]

OK, apparently, (might be because of newer mypy version, 0.910), my follow_imports = skip directive in mypy.ini was causing the fatal error. After removing it, mypy runs without throwing the error. Didn't happen before upgrading.

This too happened with a barebones Django project.

Closing this issue.

[sobolevn]

Ok, thanks a lot! If this happens again - I will add this to the docs.

[Melevir]

Same here. Downgrade to 1.8.0 helped. Will try to investigate what's wrong.

[Melevir]

Previous error raises for python 3.9. For 3.10 it's different:

Traceback (most recent call last):
  File "/Users/melevir/.virtualenvs/test3.10/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/main.py", line 87, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/main.py", line 165, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/build.py", line 179, in build
    result = _build(
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/build.py", line 254, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/build.py", line 2697, in dispatch
    process_graph(graph, manager)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/build.py", line 3014, in process_graph
    process_fresh_modules(graph, prev_scc, manager)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/build.py", line 3092, in process_fresh_modules
    graph[id].fix_cross_refs()
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/build.py", line 1991, in fix_cross_refs
    fixup_module(self.tree, self.manager.modules,
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/fixup.py", line 26, in fixup_module
    node_fixer.visit_symbol_table(tree.names, tree.fullname)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/fixup.py", line 92, in visit_symbol_table
    value.node.accept(self)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/nodes.py", line 687, in accept
    return visitor.visit_func_def(self)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/fixup.py", line 100, in visit_func_def
    func.type.accept(self.type_fixer)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/types.py", line 1156, in accept
    return visitor.visit_callable_type(self)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/fixup.py", line 185, in visit_callable_type
    ct.ret_type.accept(self)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/types.py", line 846, in accept
    return visitor.visit_instance(self)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/fixup.py", line 154, in visit_instance
    inst.type = lookup_qualified_typeinfo(self.modules, type_ref, self.allow_missing)
  File "/Users/melevir/.virtualenvs/test3.10/lib/python3.10/site-packages/mypy/fixup.py", line 274, in lookup_qualified_typeinfo
    assert allow_missing, "Should never get here in normal mode," \
AssertionError: Should never get here in normal mode, got TypeAlias:django.db.models.query.QuerySet instead of TypeInfo

Feels like something's wrong with types of annotated queryset: without .annotate everything works fine even with follow_imports = skip.

[sobolevn]

Looks like a Partial type issue to me 😢

[sobolevn]

I would love to fix it with a proper reproduction.

[sidmitra]

@sobolevn I've tried to create a minimal django app to reproduce.

See https://github.com/sidmitra/django-stubs-issue

Also see the github failed github action with the mypy traceback:
https://github.com/sidmitra/django-stubs-issue/runs/4239774289?check_suite_focus=true

./stubissue/fake_issue.py:9: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.910
./stubissue/fake_issue.py:9: : note: use --pdb to drop into pdb
Traceback (most recent call last):
  File "mypy/checkexpr.py", line 3911, in accept
  File "mypy/nodes.py", line 1558, in accept
  File "mypy/checkexpr.py", line 271, in visit_call_expr
  File "mypy/checkexpr.py", line 353, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 858, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 917, in check_call
  File "mypy/checkexpr.py", line 1029, in check_callable_call
  File "mypy/checkexpr.py", line 738, in apply_function_plugin
  File "/tmp/venv/lib/python3.9/site-packages/mypy_django_plugin/transformers/querysets.py", line 231, in extract_proper_type_queryset_annotate
    annotated_type = get_or_create_annotated_type(api, model_type, fields_dict=fields_dict)
  File "/tmp/venv/lib/python3.9/site-packages/mypy_django_plugin/transformers/models.py", line 447, in get_or_create_annotated_type
    model_module_file = api.modules[model_module_name]  # type: ignore
KeyError: 'django_stubs_ext'

Note that this only happens with 1.9.0v of django-stubs, but not 1.8.0, so it's holding up an upgrade. I'll try to dig deeper this week, but not very familiar with the new django-stubs-ext stuff.

Even if type annotation with queryset annotate() + Count() is not possible, it might still be workable if i can just #type: ignore any false positives. Currently it just crashes on any file with Count() on a larger project.

[sobolevn]

AWESOME! Thanks a lot.

[sidmitra]

This issue might be somewhat related(or the same root cause): #745

Just adding a reference to link both.

[sidmitra]

I updated my project to reproduce this with the latest mypy and git master branch for django-stubs. This error still seems to be present.

https://github.com/sidmitra/django-stubs-issue/runs/6951208566?check_suite_focus=true

./stubissue/fake_issue.py:9: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.961
./stubissue/fake_issue.py:9: : note: use --pdb to drop into pdb
Traceback (most recent call last):
  File "mypy/checkexpr.py", line 3991, in accept
  File "mypy/nodes.py", line 1760, in accept
  File "mypy/checkexpr.py", line 305, in visit_call_expr
  File "mypy/checkexpr.py", line 383, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 892, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 957, in check_call
  File "mypy/checkexpr.py", line [10](https://github.com/sidmitra/django-stubs-issue/runs/6951208566?check_suite_focus=true#step:6:11)85, in check_callable_call
  File "mypy/checkexpr.py", line 772, in apply_function_plugin
  File "/tmp/venv/lib/python3.9/site-packages/mypy_django_plugin/transformers/querysets.py", line [23](https://github.com/sidmitra/django-stubs-issue/runs/6951208566?check_suite_focus=true#step:6:24)1, in extract_proper_type_queryset_annotate
    annotated_type = get_or_create_annotated_type(api, model_type, fields_dict=fields_dict)
  File "/tmp/venv/lib/python3.9/site-packages/mypy_django_plugin/transformers/models.py", line 576, in get_or_create_annotated_type
    model_module_file = api.modules[model_module_name]  # type: ignore
KeyError: 'django_stubs_ext'
Error: Process completed with exit code 2.

[uttamo]

This worked for me

• Create an empty queryset class for the model

from django.db.models import QuerySet

class FooQuerySet(QuerySet):
    pass

• Add a model manager using this new class

from django.db.models import Model, Manager

class Foo(Model):
    ...
    objects = Manager.from_queryset(FooQuerySet)()