Make it easier to work with circular module dependencies

[JukkaL]

It's sometimes painful to work around issues related to circular module dependencies. Some examples:

• Mypy complains about not being able to infer a type because of a circular dependency.
• Mypy type checks okay but at runtime something is undefined because of a circular dependency.
• A type annotation introduces a circular module dependency where there was none before.

Not sure how these should be resolved. Some ideas:

• Add a flag that dumps information about any circular module dependencies in the target program. This should help with debugging.
• Warn about 'from imports' that are part of a circular module dependency, as these could fail at runtime.

[JukkaL]

Another idea is improving type inference when there are circular module dependencies. A multi-phase approach might work: if a function fails to type check because it depends on something that has no inferred type yet, don't report the errors but put the function in a work list. After a circular module group has been type checked, type check any functions in the work list again. This can be repeated until the work list is empty, up to a certain maximum number of iterations.

[JukkaL]

I tentatively added this to the 0.4.0 milestone, as this could be a big issue as users start annotating code with big dependency cycles as we have at Dropbox.

Some additional things we could do to help this:

• Have a stable ordering when type checking modules within import cycles, as different orders may cause mypy to complain about different things (such as not being able to infer a type for something).
• Prioritize certain dependencies higher when determining type checking order (discussed at The mother of all pull requests: add --incremental #1292). For example, from m import y could be higher precedence than import m, and if a class in a module subclasses a class defined in another module, make that dependency a high priority one.
• Infer types for (most) decorated functions before the final type checking pass. Either do this during semantic analysis (we already do some, but it's pretty limited) or introduce another type checking pass. Decorated functions often cause trouble as mypy sometimes only infers their types during the final pass, when it can be too late as modules must be type checked in some specific, linear order.
• Make incremental type checking do something clever when there are cycles to avoid type checking the entire cycle if there is a local change and the cycle depends on a changed file (or the change is within the cycle).
• If mypy hasn't inferred the type of something during type checking, give a more useful message such as pointing to the definition of the thing without an inferred type and asking the user to add an annotation. This doesn't help with decorated functions as they can't be annotated, but see below.
• Make it possible to annotate the type of a decorated function (the external type). Currently only the type of the innermost function can be annotated, but the decorators can modify the type.

[JukkaL]

A specific issue about decorators: #1303

[JukkaL]

Here is more detailed idea for import prioritization:

• We'd have two levels of module import priorities, high and low.
• from m import x (when x is not a module) at module top level (not within a function) has high priority. Also import m or from mm import m has high priority if there is an expression that refers to m at the module top level, or if there is a type reference that contains m that is not in a comment and that is not string literal quoted.
• All other imports have low priority. This includes imports within functions and import m where m is only used within a function.
• If we detect an import cycle, use only the high priority imports to topo sort modules within the cycle. This approximates runtime import initialization order.

The intuition here is that runtime import initialization order is the "natural" order to type check modules, and it will ensure that base classes are checked before derived classes when they are defined in different modules. This should make many annotations to self.x = x assignments that I've seen in the code of our internal customers at Dropbox unnecessary.

[ddfisher]

Some other thoughts:

• It's probably worth taking a look at some common import cycle ordering errors to see if we can break down mypy's passes a bit more to fix them.
• It could be nice for debugging/understanding problems to have a mode where mypy prints out a list of all import cycles in the code.
• I'm not sure how hard this is, but we could try to process imports in the same order as Python itself, which might help significantly.