Mypy Deamon crash on second run

[ivan94fi]

Crash Report
I saw this crash when using Mypy extension for VS Code, but it is reproducible in my code base even from command line by directly invoking dmypy as a module. The crash is in two phases:

1. In the first run, the deamon correctly reports errors: the final message is Found 238 errors in 46 files (checked 1125 source files), then the deamon exits (I don't know if exiting in this situation is the expected behaviour).
2. When I try to restart the deamon the following traceback appears.

Each time I run the command, these two phases alternate.

Traceback

Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 227, in serve
  File "mypy/dmypy_server.py", line 270, in run_command
  File "mypy/dmypy_server.py", line 336, in cmd_run
  File "mypy/dmypy_server.py", line 405, in check
  File "mypy/dmypy_server.py", line 589, in fine_grained_increment_follow_imports
  File "mypy/server/update.py", line 277, in update
  File "mypy/server/update.py", line 862, in propagate_changes_using_dependencies
  File "mypy/server/update.py", line 996, in reprocess_nodes
  File "mypy/server/astmerge.py", line 140, in merge_asts
  File "mypy/server/astmerge.py", line 192, in replace_nodes_in_ast
  File "mypy/nodes.py", line 374, in accept
  File "mypy/server/astmerge.py", line 213, in visit_mypy_file
  File "mypy/traverser.py", line 114, in visit_mypy_file
  File "mypy/nodes.py", line 1127, in accept
  File "mypy/server/astmerge.py", line 238, in visit_class_def
  File "mypy/server/astmerge.py", line 393, in process_synthetic_type_info
  File "mypy/nodes.py", line 208, in accept
RuntimeError: Not implemented

To Reproduce

Unfortunately I cannot share code for now, I will try to make a reproducible example later. For now I post this error to see if someone can direct me to further steps to better understand the root of the problem or point out trivial mistakes or misconfigurations.

Your Environment

• Mypy version used: dmypy version 0.982
• Mypy command-line flags: the following is the command executed by the VS Code extension and then executed by me afterwards:

$path_to_python -m mypy.dmypy \
    --status-file /home/.../filename.json \
    run \
    --log-file /home/.../filename.log \
    -- $project_directory \
    --show-traceback --show-column-numbers --python-executable $path_to_python

• Mypy configuration options from mypy.ini (and other config files):

[mypy]
warn_return_any = True
warn_unused_configs = True
ignore_missing_imports = True
explicit_package_bases = True
namespace_packages = True

• Python version used: Python 3.9.6
• Operating system and version: Ubuntu 18.04

[hauntsaninja]

Thanks for reporting! Hard to figure out without a repro. From the stacktrace I suspect it's something to do with namedtuples, in case that helps narrow things down.

You could try installing mypy master and seeing if it repros. Once you've installed mypy master you could also put in a breakpoint and see exactly what class definition it's struggling with.

[ivan94fi]

Hi! Yes I know, without code this is almost nonsense. Thanks for the suggestion, as soon as I can I will install mypy master and try to debug! I will report back if I find something.

[ilevkivskyi]

I have a repro

[case testNamedTupleNestedCrash]
import m
[file m.py]
from typing import NamedTuple

class NT(NamedTuple):
    class C: ...
    x: int
    y: int

[file m.py.2]
from typing import NamedTuple

class NT(NamedTuple):
    class C: ...
    x: int
    y: int
# change
[builtins fixtures/tuple.pyi]

crashes with the same traceback (in cached mode). Although @ivan94fi please still check your code, maybe you have a different scenario. If it is the same, I think I have a simple fix for this crash, similar to what @Michael0x2a did for TypedDicts.

[ivan94fi]

Hi everyone, I tested mypy master and I can reproduce the crash. However I noticed that the crash only happens when using the dmypy daemon, non when running mypy directly (both for the master and the 0.982 version).

I tried to put a breakpoint in the affected code but I always get a bdb.BdbQuit and cannot perform interactive debugging. I suppose because the code is executed by the daemon. I added the --pdb flag at the end of the command but this did not change anything.

I tried to put a breakpoint in the following files, in many places, but I always got the above behaviour:

• dmypy_server.py: here I tried inside Server.serve method, in various lines, both inside the try catches, and outside
• dmypy/__main__.py: here i put a breakpoint before console_entry, and then continued, just to be sure to be inside the debugger

Can someone suggest me where to put a breakpoint? Thank you.

[ilevkivskyi]

You should try putting a breakpoint before this line https://github.com/python/mypy/blob/master/mypy/server/astmerge.py#L393 (also maybe try using pdb, I never tried bdb). If this still will not work, you can simply add a print statement before that line, but then you will need to pass some file to --log-file when starting daemon (it doesn't print to stdout). And check tail of that file when daemon crashes.

[ivan94fi]

Ok I will try as soon as possible, thank you.

By the way, I used pdb too, but that exception resides in bdb, which apparently is the package containing the basic functions of pdb.

[ivan94fi]

Hi I managed to track down the file that was causing the issue and it has the same exact structure of the example reported by @ilevkivskyi 3 comments back, that is, a NamedTuple with an inner class:

from typing import Dict, NamedTuple, List

# Class and attributes names are changed, the structure is identical to the one in the original code
class A(NamedTuple):
    a: List[str]
    b: Dict
    c: str
    d: List[str]

    class AInner:
        e = False

It is interesting to note that executing mypy (both daemon and cli) on the above code, does not make mypy crash but it returns an error:
mypy_crash.py:10: error: Invalid statement in NamedTuple definition; expected "field_name: field_type [= default]"

However removing this code from the original project where it provoked a mypy crash, makes mypy stop crashing.

[ilevkivskyi]

OK, thanks for confirming! I will try to find time for this in coming days (unless someone else does it first).