Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

default_return_type is Any and hits assertion error for Model.objects.values_list #843

Closed
markberger opened this issue Feb 5, 2022 · 5 comments · Fixed by #1727
Closed

default_return_type is Any and hits assertion error for Model.objects.values_list #843

markberger opened this issue Feb 5, 2022 · 5 comments · Fixed by #1727
Labels
bug Something isn't working crash "Internal error" crashes from mypy mypy-plugin Issues specific to mypy_django_plugin

Comments

@markberger
Copy link

Hello, I'm trying to debug an issue when evaluating values_list and I'm unable to find the root cause or create a minimal example. What is the best way to determine what might be settings default_return_value=Any?

My code is roughly this:

class ModelManager(models.Manager):
    pass

class MyModel(models.Model):
    field_name = models.CharField(max_length=64)

...

def view_func(
    self, request: Request, pk: t.Optional[uuid.UUID] = None
) -> Response:
    query_set = MyModel.objects.values_list('field_name', flat=True).distinct()
    return Response(query_set.all())

During typechecking I hit the following assertion:

views.py:101: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.931
Dropping into pdb
> /Users/mark.berger/.pyenv/versions/3.8.10/envs/my-project/lib/python3.8/site-packages/mypy_django_plugin/transformers/querysets.py(149)extract_proper_type_queryset_values_list()
-> assert isinstance(default_return_type, Instance)
(Pdb) print(default_return_type)
Any
(Pdb) print(ctx)
MethodContext(type=path.to.ModelManager[path.to.MyModel], arg_types=[[Literal['field_name']?], [Literal[True]?], []], arg_kinds=[[<ArgKind.ARG_POS: 0>], [<ArgKind.ARG_NAMED: 3>], []], callee_arg_names=['fields', 'flat', 'named'], arg_names=[[None], ['flat'], []], default_return_type=Any, args=[[<mypy.nodes.StrExpr object at 0x114c3a270>], [<mypy.nodes.NameExpr object at 0x114c397c0>], []], context=<mypy.nodes.CallExpr object at 0x114c384a0>, api=<mypy.checker.TypeChecker object at 0x114dd5040>)
(Pdb) up
> /Users/mark.berger/robinhood/rh/device-risk/mypy/checkexpr.py(758)apply_function_plugin()
(Pdb) list
[EOF]

I'm using version 1.9.0 of the stubs with mypy 0.931. I've also tried downgrading to 0.910 without any luck. I've also tried using master for the stubs but this also doesn't address the issue.

Here is the minimal example I tried to create but this passes as expected:

-   case: my_test_case
    main: |
        from myapp.models import Transaction
        reveal_type(Transaction.objects.values_list('total', flat=True))  # N: Revealed type is "django.db.models.query._QuerySet[myapp.models.Transaction, builtins.int]"
        reveal_type(Transaction.objects.values_list('status', flat=True))  # N: Revealed type is "django.db.models.query._QuerySet[myapp.models.Transaction, builtins.str]"
    installed_apps:
        - myapp
    files:
        -   path: myapp/__init__.py
        -   path: myapp/models.py
            content: |
                from django.db import models
                class TransactionManager(models.Manager['Transaction']):
                    pass
                class Transaction(models.Model):
                    status = models.CharField()
                    total = models.IntegerField()

                    objects = TransactionManager()

Thanks for the help!
@markberger markberger added the bug Something isn't working label Feb 5, 2022
@sterliakov
Copy link
Contributor

sterliakov commented Feb 5, 2022 •
edited
Loading

I can confirm that happens with mypy 0.940+dev (latest from their github) too. Also both qs.values(...) and qs.values_list(...) are affected.
Adding full traceback:

./app/models/treatment.py:683: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.940+dev.26fdd456b7e665c3c0f161234ee48b9e6144940d
Traceback (most recent call last):
  File "/home/stas/Documents/Work/stomatology/pyenv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/main.py", line 96, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/main.py", line 173, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/build.py", line 180, in build
    result = _build(
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/build.py", line 256, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/build.py", line 2715, in dispatch
    process_graph(graph, manager)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/build.py", line 3046, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/build.py", line 3144, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/build.py", line 2183, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 318, in check_first_pass
    self.accept(d)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 424, in accept
    stmt.accept(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/nodes.py", line 1008, in accept
    return visitor.visit_class_def(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 1802, in visit_class_def
    self.accept(defn.defs)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 424, in accept
    stmt.accept(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/nodes.py", line 1077, in accept
    return visitor.visit_block(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 2105, in visit_block
    self.accept(s)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 424, in accept
    stmt.accept(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/nodes.py", line 730, in accept
    return visitor.visit_func_def(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 774, in visit_func_def
    self._visit_func_def(defn)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 778, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 840, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 1025, in check_func_def
    self.accept(item.body)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 424, in accept
    stmt.accept(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/nodes.py", line 1077, in accept
    return visitor.visit_block(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 2105, in visit_block
    self.accept(s)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 424, in accept
    stmt.accept(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/nodes.py", line 1143, in accept
    return visitor.visit_assignment_stmt(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 2147, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checker.py", line 2333, in check_assignment
    rvalue_type = self.expr_checker.accept(rvalue)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checkexpr.py", line 3975, in accept
    typ = node.accept(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/nodes.py", line 1735, in accept
    return visitor.visit_call_expr(self)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checkexpr.py", line 288, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checkexpr.py", line 371, in visit_call_expr_inner
    ret_type = self.check_call_expr_with_callee_type(callee_type, e, fullname,
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checkexpr.py", line 880, in check_call_expr_with_callee_type
    return self.check_call(callee_type, e.args, e.arg_kinds, e,
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checkexpr.py", line 940, in check_call
    return self.check_callable_call(callee, args, arg_kinds, context, arg_names,
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checkexpr.py", line 1070, in check_callable_call
    new_ret_type = self.apply_function_plugin(
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy/checkexpr.py", line 760, in apply_function_plugin
    return method_callback(
  File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy_django_plugin/transformers/querysets.py", line 149, in extract_proper_type_queryset_values_list
    assert isinstance(default_return_type, Instance)
AssertionError: 
./app/models/treatment.py:683: : note: use --pdb to drop into pdb

For qs.values() only last line differs:

File "/home/stas/Documents/Work/stomatology/pyenv/lib/python3.10/site-packages/mypy_django_plugin/transformers/querysets.py", line 272, in extract_proper_type_queryset_values
    assert isinstance(default_return_type, Instance)

Using custom manager is not related. All cases where it happened (I commented out first error source to find next one) with me were using related fields: reverse foreign key (something_set) or ManyToManyField. MRE:

# myapp/models.py
from django.db import models

class Child(models.Model):
    name = models.CharField(max_length=10)
class Parent(models.Model):
    children = models.ManyToManyField('myapp.Child')

def foo(instance: Parent) -> None:
    instance.children.values_list('name', flat=True)

# mre/settings.py
...
INSTALLED_APPS =[..., 'myapp']
...

# ./pyproject.toml 
[tool.mypy]
plugins = ["mypy_django_plugin.main"]
ignore_missing_imports = true
disallow_untyped_defs = true
check_untyped_defs = true
disallow_incomplete_defs = true
disallow_any_generics = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_unreachable = true
exclude = [
    'migrations/*',
]

[tool.django-stubs]
django_settings_module = "mre.settings"

Run $ mypy . from top project directory (with manage.py and pyproject.toml).

@sterliakov
Copy link
Contributor

It's fixed by removing django-stubs and django_stubs_ext and installing from git instead of PyPi.

@markberger
Copy link
Author

@sterliakov thank you for looking into this! Unfortunately I'm not able to reproduce the error with your example. This is what I did:

> django-admin startproject mre
> cd mre/
> python manage.py startapp myapp
>
# myapp/models.py
from django.db import models

class Child(models.Model):
    name = models.CharField(max_length=10)
class Parent(models.Model):
    children = models.ManyToManyField('myapp.Child')

def foo(instance: Parent) -> None:
    instance.children.values_list('name', flat=True)

# mre/settings.py
...
INSTALLED_APPS =[..., 'myapp']
...

# ./pyproject.toml 
[tool.mypy]
plugins = ["mypy_django_plugin.main"]
ignore_missing_imports = true
disallow_untyped_defs = true
check_untyped_defs = true
disallow_incomplete_defs = true
disallow_any_generics = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_unreachable = true
exclude = [
    'migrations/*',
    'manage.py',  # only change here
]

[tool.django-stubs]
django_settings_module = "mre.settings"


> pip freeze
asgiref==3.5.0
Django==3.2.12
django-stubs==1.9.0
django-stubs-ext==0.3.1
mypy==0.931
mypy-extensions==0.4.3
pytz==2021.3
sqlparse==0.4.2
toml==0.10.2
tomli==2.0.0
types-pytz==2021.3.4
types-PyYAML==6.0.4
typing-extensions==4.0.1


> mypy .
mre/settings.py:28: error: Need type annotation for "ALLOWED_HOSTS" (hint: "ALLOWED_HOSTS: List[<type>] = ...")
Found 1 error in 1 file (checked 11 source files)

Did I miss anything? Thanks!

@sterliakov
Copy link
Contributor

I'm really confused. I can't reproduce with that setup either, however, it is exactly layout of test project I created. Also I cannot reproduce it with exactly that project (haven't deleted it yet). I have played with versions of mypy and django-stubs and finally lost the origin of this problem. Looks like it might be cache error or smth like this, because it happened with me when I was upgrading large set of project dependencies and migrating to Django 4.0 and I used that dev virtualenv to create test project. After a few attempts everything started working and now works with PyPi version too.

@markberger
Copy link
Author

Rolling back to version 1.8.0 fixes the issue I'm seeing

@intgr intgr added mypy-plugin Issues specific to mypy_django_plugin crash "Internal error" crashes from mypy labels Nov 8, 2022
@flaeppe flaeppe mentioned this issue Sep 21, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working crash "Internal error" crashes from mypy mypy-plugin Issues specific to mypy_django_plugin
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Handle mismatching types in queryset method resolving gracefully flaeppe/django-stubs
3 participants
@intgr @markberger @sterliakov