Semantic analysis deferral crash

[atsuoishimoto]

Crash Report

Mypy crashes with code at https://github.com/atsuoishimoto/mypy_issue.

Traceback

$ mypy .
./test1.py:2: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
If this issue continues with mypy master, please report a bug at https://github.com/python/mypy/issues
version: 0.920+dev.b17ae30d1efe58095c61c1731c29bcfe4dd7065c
./test1.py:2: : note: please use --show-traceback to print a traceback when reporting a bug
(.venv) ~/src/src mypy . --show-traceback
./test1.py:2: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.920+dev.b17ae30d1efe58095c61c1731c29bcfe4dd7065c
Traceback (most recent call last):
  File "/Users/ishimoto/src/src/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/main.py", line 87, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/main.py", line 165, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/build.py", line 179, in build
    result = _build(
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/build.py", line 254, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/build.py", line 2707, in dispatch
    process_graph(graph, manager)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/build.py", line 3031, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/build.py", line 3123, in process_stale_scc
    mypy.semanal_main.semantic_analysis_for_scc(graph, scc, manager.errors)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal_main.py", line 78, in semantic_analysis_for_scc
    process_top_levels(graph, scc, patches)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal_main.py", line 199, in process_top_levels
    deferred, incomplete, progress = semantic_analyze_target(next_id, state,
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal_main.py", line 326, in semantic_analyze_target
    analyzer.refresh_partial(refresh_node,
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal.py", line 402, in refresh_partial
    self.refresh_top_level(node)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal.py", line 413, in refresh_top_level
    self.accept(d)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal.py", line 4928, in accept
    node.accept(self)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/nodes.py", line 402, in accept
    return visitor.visit_import_all(self)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal.py", line 1983, in visit_import_all
    self.add_imported_symbol(name, node, i,
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal.py", line 4576, in add_imported_symbol
    self.add_symbol_table_node(name, symbol, context)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal.py", line 4481, in add_symbol_table_node
    self.defer(context)
  File "/Users/ishimoto/src/src/.venv/lib/python3.9/site-packages/mypy/semanal.py", line 4639, in defer
    assert not self.final_iteration, 'Must not defer during final iteration'
AssertionError: Must not defer during final iteration
./test1.py:2: : note: use --pdb to drop into pdb

To Reproduce

Run mypy . at src directory below.

Test code can be cloned from https://github.com/atsuoishimoto/mypy_issue.

Directory structure

.
└── src
    ├── dir1
    │   ├── __init__.py
    │   └── test2.py
    └── test1.py

test1.py

from dir1 import *

dir1/__init__.py

from .test2 import *

dir1/test2.py

from test1 import aaaa

Your Environment

• Mypy version used: 0.920+dev.b17ae30d1efe58095c61c1731c29bcfe4dd7065c
• Mypy command-line flags: mypy .
• Mypy configuration options from mypy.ini (and other config files): None
• Python version used: Python 3.9
• Operating system and version: macOS Catalina 10.15.7

[hauntsaninja]

I can repro; thank you for taking the time to make a great bug report!

[SwagatSBhuyan]

I guess it is showing an error due to cyclic importing? What is the expected behavior, that mypy should detect such cycles on execution?
Maybe I can work on this, will require a little bit of error handling and stuff I am guessing. Feel free to guide me on this issue @atsuoishimoto @hi-ogawa @hauntsaninja

[jrabbit]

Any remediations would be welcome in the mean time. I haven't been able to exclude my way out of the error or by changing import following.

[SwagatSBhuyan]

Any remediations would be welcome in the mean time. I haven't been able to exclude my way out of the error or by changing import following.

Ok, I'll read up on it. Again, I'm new to contributing and the mypy repository. I'll do what I can!

[SwagatSBhuyan]

On testing this on my local system, I could reproduce the issue. However, on simply compiling the script test1.py, it stated that it could't find 'aaaa'. I simply added a function aaaa that printed "aaaa" in test1.py and now even the mypy . terminal command is working properly:

Swagat@DESKTOP-9F5G26Q MINGW64 ~/Documents/Python Scripts/mypy/testing/mypy_issue (main)
$ mypy .
Success: no issues found in 3 source files

Maybe this wasn't intended? Or it was required that mypy print a different error message on the absence of a function 'aaaa'? Please guide me on this guys @jrabbit @hauntsaninja @hi-ogawa

[hi-ogawa]

Hi everyone, I just got back to my draf PR #11346 after quite some time, and indeed OP's example seems to fixed already on current master. But I found there are some similar cyclic import examples where it leads to crash, so I reopend my PR with some description updated. I would appreciate if someone could review it. Thanks a lot!

[hauntsaninja]

OP's example does not look fixed on current master to me, but does seem fixed with your PR :-)
I added testCyclicUndefinedImportWithStar3 at a couple different commits, but haven't seen it fail yet. In general, mypy's unit tests differ a little bit from real mypy, the most significant difference being that they use a minimal set of fake type stubs, as opposed to all of typeshed — but it's not clear to me what is causing the different behaviour between testCyclicUndefinedImportWithStar3 and actually running mypy.