New semantic analyzer: Namespace package with __init__.py breaks analysis

[fireundubh]

I'm getting a lot of errors reported that are not errors.

• Classes are reported as invalid types
• Calls to methods are reported as modules not having those attributes
• Instances of classes are reported as modules not callable
• Types passed to, e.g., isinstance are reported as incompatible type Module (suggests that all custom types are treated as type Module)
• Base classes are reported as invalid base classes
• Calls to instances of classes are reported as not having the specified attributes
• Package imports in __init__.py are reported as already defined

I am not sure whether these are issues are caused by the new semantic analyzer or by the Mypy plugin for PyCharm which has not been updated yet.

In comparison, with the new semantic analyzer disabled, there are no errors. The application works correctly.

[ilevkivskyi]

Can you give some examples of the code that triggers this?

[fireundubh]

Invalid type "Presenters.BasePresenter" (35:20)
Module has no attribute "get_presenter" (35:36)

presenter: BasePresenter = PresenterHelper.get_presenter(options=presenter_args)

BasePresenter is imported from a Presenters package. get_presenter is a static method.

Argument 2 to "isinstance" has incompatible type "Tuple[Module, Module, Module]"; expected "Union[type, Tuple[Union[type, Tuple[Any, ...]], ...]]" (96:12)
BasePresenter? has no attribute "save_to_file" (97:13)

if isinstance(presenter, (ConsolePresenter, JsonPresenter, MarkdownPresenter)):
    presenter.save_to_file(records, args.documents, args.output)

ConsolePresenter, JsonPresenter, and MarkdownPresenter are imported from a package.

Invalid type "Presenters.ThemeManager" (36:24)
Module not callable (36:39)

theme_manager: ThemeManager = ThemeManager(args.output)

ThemeManager is also imported from a package.

Argument "formatter_class" to "ArgumentParser" has incompatible type Module; expected "Type[HelpFormatter]" (8:15)

from Extensions import HelpFormatterEx

if __name__ == '__main__':
    _parser = argparse.ArgumentParser(formatter_class=HelpFormatterEx)

HelpFormatterEx inherits from argparse.HelpFormatter and is imported from a package.

The overarching pattern seems to be related to package imports.

[ilevkivskyi]

Can you show the place in Extensions where HelpFormatterEx is defined? Also do you have a module with the same name?

[fireundubh]

Tree:

Extensions
  __init__.py
  HelpFormatterEx.py

__init__.py

from Extensions.HelpFormatterEx import HelpFormatterEx

HelpFormatterEx.py

import argparse

class HelpFormatterEx(argparse.HelpFormatter):
    def _format_action_invocation(self, action: argparse.Action) -> str:
# snip

[ilevkivskyi]

There must be something else, this test passes cleanly:

[case testNewAnalyzerArgParseSubmoduleAsClass_newsemanal]
import argparse
from Extensions import HelpFormatterEx

if __name__ == '__main__':
    _parser = argparse.ArgumentParser(formatter_class=HelpFormatterEx)

[file Extensions/__init__.py]
from Extensions.HelpFormatterEx import HelpFormatterEx

[file Extensions/HelpFormatterEx.py]
import argparse

class HelpFormatterEx(argparse.HelpFormatter):
    def _format_action_invocation(self, action: argparse.Action) -> str:
        ...

Do you have any tricky import cycles and/or forward references there?

[fireundubh]

Extensions is a package within a package, so it's really more like MyApplication.Extensions.HelpFormatterEx.HelpFormatterEx.

I just tried to repro this specifically in a new project, but I got the same result as you. It's possible the error is somewhere else and tripping up the other checks.

I'll try to narrow it down.

[ilevkivskyi]

I'll try to narrow it down.

OK, thanks!

[fireundubh]

Okay, I found the error. The issue was I had an empty and errant __init__.py in my namespace package.

Using the docs example tree as a reference:

mynamespace/
    # No __init__.py here. <---- had an __init__.py there
    subpackage_a/
        # Sub-packages have __init__.py.
        __init__.py
        module.py

If that file exists, mypy reports a ton of false-positive errors. Deleting that file clears up the report.

[ilevkivskyi]

Hm, interesting. Could you please post a self contained repro that you have?

[ilevkivskyi]

(I also added the usability label because the errors emitted make it hard to understand what went wrong, although it is kind of related to #4030).

[fireundubh]

Here you go: mypytest.zip.

[ilevkivskyi]

I tried checking the zipped files with mypy master and mypy generates only two errors:

mypytest/Presenters/PresenterHelper.py:13: error: Module not callable
mypytest/Presenters/PresenterHelper.py:15: error: Module not callable

both with new and old analyzers. Also the errors look reasonable.

[ilevkivskyi]

To clarify, I don't see any bug here.

[JukkaL]

Removed priority label for now since we don't have a repro.

[fireundubh]

@ilevkivskyi These are my results.

Mypy found 18 errors in 8 files

Application.py : 3 errors 
  Invalid type "mypytest.Presenters.BasePresenter" (15:20)
  Module has no attribute "get_presenter" (15:36)
  Argument 2 to "isinstance" has incompatible type Module; expected "Union[type, Tuple[Union[type, Tuple[Any, ...]], ...]]" (17:12)

ConsolePresenter.py : 2 errors 
  Invalid type "mypytest.Presenters.BasePresenter" (4:24)
  Invalid base class (4:24)

JsonPresenter.py : 2 errors 
  Invalid type "mypytest.Presenters.BasePresenter" (4:21)
  Invalid base class (4:21)

MarkdownPresenter.py : 2 errors 
  Invalid type "mypytest.Presenters.BasePresenter" (4:25)
  Invalid base class (4:25)

PresenterHelper.py : 4 errors 
  Invalid type "mypytest.Presenters.BasePresenter" (9:1)
  Module not callable (11:20)
  Module not callable (13:20)
  Module not callable (15:20)

__init__.py : 1 error 
  Name 'HelpFormatterEx' already defined (by an import) (1:1)

__init__.py : 3 errors 
  Name 'BasePresenter' already defined (by an import) (1:1)
  Name 'ConsolePresenter' already defined (by an import) (3:1)
  Name 'PresenterHelper' already defined (by an import) (5:1)

__main__.py : 1 error 
  Argument "formatter_class" to "ArgumentParser" has incompatible type Module; expected "Type[HelpFormatter]" (8:15)

Are you using the mypy.ini included in the archive?

Did you try removing the empty __init__.py file from the mypytest namespace package?

If you remove the empty __init__.py file from the mypytest namespace package, there will be no errors.

Python can run the application with or without the empty __init__.py file in the mypytest namespace package whereas Mypy blows up.

[ilevkivskyi]

Yes, I use the mypy.ini file. If I remove the __init__.py file, the two errors I see do indeed disappear, so there might be a bug. However, this behaviour is 1:1 with what I see with the old analyser.

(Btw note I use a recent master, not the released version.)

[fireundubh]

The Mypy plugin for PyCharm has the ability to run Mypy on all files in the project.

If I restore the empty __init__.py file and disable the new semantic analyzer, Mypy reports only two errors in 1 file across the project:

Mypy found 2 errors  in 1 file

PresenterHelper.py : 2 errors 
  Module not callable (13:20)
  Module not callable (15:20)

That's compared to the 18 errors in 8 files across the project with the new semantic analyzer enabled.

Regardless of whether the new semantic analyzer enabled, Mypy's behaviour - with respect to namespace packages and __init__.py - is not 1:1 with Python.

I would also say that "module not callable" is not a reasonable error because it is false.

[fireundubh]

Not fixed in 0.720.

[Cito]

This looks somewhat similar to #7203.

And I am also seeing false "... already defined (by an import)" errors with the new semantic analyzer.