Bug involving Python 2, incremental mode, partial cache invalidation, and possibly nested folders

[Michael0x2a]

I recently ran into a bug that manifests when running Python 2 with incremental mode. Here is the shortest repro I was able to create so far:

File structure

parent/
    __init__.py
    a.py
    b.py
    subfolder/
        __init__.py
        child_a/
            __init__.py
            c.py
        child_b/
            __init__.py
            d.py
e.py

parent/a.py

import e 
a_const = 0

parent/b.py

b_const = 0

parent/subfolder/child_a/__init__.py

from .c import c_const

parent/subfolder/child_a/c.py

from parent.b import b_const
c_const = 0

parent/subfolder/child_b/d.py

from parent.a import a_const
from ..child_a import c

e.py

from types import FunctionType

All of the other __init__.py files are empty.

When you try running mypy in incremental mode with Python 2, you get a spurious error in the second run (note that I update the timestamp of parent/b.py to force mypy to recheck that file):

$ rm -rf .mypy_cache
$ python3 -m mypy --py2 --incremental parent
$ touch parent/b.py
$ python3 -m mypy --py2 --incremental parent
experiments/parent/subfolder/child_b/d.py:2: error: Module has no attribute 'c'

However, mypy works as expected when running in Python 3 mode. I'm not entirely sure why that is.

I'm planning on continuing to try and make a shorter/more enlightening repro, but I figured I should file an issue now to get this on the record.

[gvanrossum]

That's amazing. The one difference between py2 and py3 that might matter here is that the cycle of modules dependent on "builtins" differs between versions. I guess running `mypy -v` might give some insight, if you haven't tried that already. Also, I'd be interested to see what's in the metadata for various files after the first run, and whether there's a difference in the metadata between py2 and py runs. Finally, one more difference would be absolute vs. relative import -- py2 defaults to relative import. But all your imports would seem to be unambiguous.

[gnprice]

Apparently this triggers frequently when using --incremental on some of our larger internal codebases at Dropbox.

[gvanrossum]

I cannot repro this any more. According to git bisect it was fixed on August 18 by revision 95a0186, "Make incremental not propagate staleness if the public interface of a module is unchanged (v2) (#2014)". That's good enough for me!