Suppress 'duplicate module named xx' errors?

[OJFord]

Am I right in thinking it's not presently possible to ignore errors for having multiple (as the error should really say, rather than 'duplicate') modules with the same name?

I'm running mypy on files in a repo that contains multiple packages with some commonality among them; for example they each have a src/wsgi.py, which renders:

error: Duplicate module named 'src.wsgi'

1. I think this should be reworded Multiple modules named xx
2. Would you be open to an --allow-same-name-modules/allow_same_name_modules = True or similar?

[gvanrossum]

Those probably aren't packages, they're folders (or there are __init__.py files that shouldn't be there). You should really run mypy separately over each package in that case, since otherwise it won't know what to do when one of these is imported from another place being checked.

For this reason, implementing such a flag would not be easy, and probably not very useful.

[leinardi]

Hi @gvanrossum, I am writing a PyCharm/IntelliJ IDEA plugin to run mypy scanning and I am encountering the same issue.

PyCharm/IntelliJ IDEA code inspection API sends me a list of open files in the IDE but, if the file list contains both a __init__.py and a python file from the same package, I get the "Duplicate module named" error:

$ mypy --show-column-numbers  /home/leinardi/PycharmProjects/gsi/gsi/__init__.py /home/leinardi/PycharmProjects/gsi/gsi.py
gsi.py: error: Duplicate module named 'gsi'

Do you have any suggestion to scan both this files without running mypy separately for each of them?

[ilevkivskyi]

I am writing a PyCharm/IntelliJ IDEA plugin to run mypy scanning

Would you like to contribute to existing one instead? https://github.com/dropbox/mypy-PyCharm-plugin

[leinardi]

Hi @ilevkivskyi, normally that would be the best thing to do, but I have already done adapting the code of my Pylint plugin to work with mypy.

This plugin is based on Checkstyle-IDEA plugin and offers also real-time scanning, checkin validations and various on-demand scan actions.

This is my latest build, is still not production ready but should give you an idea of what feature are there: Mypy-0.7.0.zip

[gvanrossum]

$ mypy --show-column-numbers  /home/leinardi/PycharmProjects/gsi/gsi/__init__.py /home/leinardi/PycharmProjects/gsi/gsi.py
gsi.py: error: Duplicate module named 'gsi'

This is because apparently you seem to have both a folder named gsi and a file name gsi.py. This is a problem for Python too (IIRC the folder/package wins) so you can't really blame mypy for complaining about this.

[leinardi]

Oh ok, thanks for the clarification 👍.

[leinardi]

Hi @gvanrossum, I am now getting the same error when I try to check these files from mypy project:

$ mypy  /home/leinardi/PycharmProjects/mypy/test-data/packages/typedpkg-stubs/setup.py /home/leinardi/PycharmProjects/mypy/extensions/setup.py /home/leinardi/PycharmProjects/mypy/setup.py
extensions/setup.py: error: Duplicate module named 'setup'

or just simply these 2 files:

$ mypy /home/leinardi/PycharmProjects/mypy/extensions/setup.py /home/leinardi/PycharmProjects/mypy/setup.py
setup.py: error: Duplicate module named 'setup'

Is it also in this case a project problem? And why it does not happen when I scan the entire project using mypy /home/leinardi/PycharmProjects/mypy/?

[gvanrossum]

Presumably the folders containing the setup.py files don't have __init__.py files, so mypy treats them both as top-level modules. You can fix this using --scripts-are-modules: https://mypy.readthedocs.io/en/latest/command_line.html#miscellaneous

[leinardi]

Hey @gvanrossum, thank you for your answer. You are correct, the folders don't have __init__.py files.

But unfortunately the option --scripts-are-modules is not fixing the issue:

$ mypy --scripts-are-modules extensions/setup.py setup.py
setup.py: error: Duplicate module named 'setup'

[gvanrossum]

I suggest asking for help in a more interactive forum, e.g. https://gitter.im/python/typing

[leinardi]

Done, but still no answer:

But isn't this a mypy bug? why I cannot scan these 2 files together but it is fine if I scan the entire directory with mypy .?

[petergaultney]

I have this problem frequently where I'm scanning separate packages with pre-commit hooks. It would be very useful to be able to disable this check completely, or else have an obvious workaround.

Scripts-are-modules does not help.

[gvanrossum]

You'll have to write a script that scans the packages and produces a full list of files to pass to mypy except for the files you don't want.

[OJFord]

That's exactly when this problem occurs - when that full list of files contains 'duplicate modules named x'.

(An example of the script that does that is pre-commit --all-files.)

[gvanrossum]

Obviously that's a bug in your script, not in mypy. :-)

[OJFord]

I hit this again, but found a solution (for specifically the pre-commit case) that seems to be workable:

- id: mypy
   name: mypy-one
    files: ^one/
    entry: mypy one/
    pass_filenames: false
- id: mypy
   name: mypy-two
   files: ^two/
   entry: mypy two/
   pass_filenames: false
...

and of course whatever args or other options you were using on your single entry, but that's the barebones.

Bit of a pain to have to maintain this - especially because if you add a three/ it's not linted unless you remember to add it to the whitelist - but at least it seems to work.

(NB: Note that the id is the same on each (just not the name) and it's that that's the optional argument to pre-commit run, so you can still run them all with pre-commit run mypy.)

[br3ndonland]

Edit by hauntsaninja: I strongly recommend against disabling error code misc, putting in a dropdown so people don't copy it

The "duplicate module" error is in the misc category. You can see the error code by adding show_error_codes = True to mypy.ini. If you're not concerned about omitting the other miscellaneous checks, try adding disable_error_code = misc to mypy.ini.

[mypy]
disable_error_code = misc
show_error_codes = True
files = **/*.py

For the pre-commit case mentioned above (#4008 (comment), also see https://github.com/pre-commit/mirrors-mypy/issues/5), try using pre-commit/mirrors-mypy with language: system and pass_filenames: false.

language: system tells pre-commit to "rely on whatever state the user's machine is in" (https://github.com/pre-commit/mirrors-mypy/issues/3#issuecomment-547999959), which may include mypy already installed in the environment and a mypy.ini configuration file. This may be an acceptable tradeoff if running both pre-commit and mypy from the same environment, such as a virtual environment or CI environment. When using language: system for pre-commit in CI such as GitHub Actions, it is necessary to configure the Python environment before running pre-commit (set up Python and install dependencies).

pass_filenames: false tells pre-commit not to override the list of files to check. This can be helpful simply to avoid maintaining duplicate lists of files. mypy.ini uses glob pattern matching to specify files to check, but .pre-commit-config.yaml uses Python regex for its files key.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.790
    hooks:
      - id: mypy
        language: system
        pass_filenames: false

[hauntsaninja]

You should not use @br3ndonland's disable_error_code = misc hack... more details at #9727

[tuukkamustonen]

Unless I'm mistaken, none of the workarounds allow to scan all the files in a monorepo, which has something like:

component1/
    setup.py
    ...
component2/
    setup.py
   ...
tasks.py
some-other-file.py

1. I don't want to ignore all "misc" errors, only this one issue in particular.
2. Setting pass_filenames: false will cause scan of all files, not just those that were modified (for pre-commit hooks).
3. I don't want --exclude any files - want everything scanned.
4. Cannot use --scripts-are-modules (it's intended for non .py files)

Scanning above-like monorepo from main level simply does not work now, as each component holds their "duplicate" setup.py (tests/conftest.py, etc) files. In order to scan this sort of a structure, I would need to scan in parts:

• Scan each directory separately (e.g. component1, component2)
• Then scan the main level, excluding all components/sub-directories (something like .*/.*, I guess)

(And I'm not sure if I can even do this with pre-commit framework, need to study.)

Besides, what's the harm in having similarly named files in different directories? They are standalone .py files and those directories are not declared packages (there's no __init__.py anywhere). Even if they were considered packages in above example, there wouldn't be no clash (component1.setup vs component2.setup).

Now that 0.800+ supports recursive file scanning, could this issue be re-considered?

[tuukkamustonen]

Opened #10428 to perhaps better describe (or ask about) the problem.

[jaklan]

@tuukkamustonen described the issue perfectly in terms of monorepo usage - I also don't find any of these workarounds to resolve the issue. pass_filenames: false is an overkill, creating separate mypy hook for each of the projects in monorepo - as well. For now we are just excluding the problematic files, which is not really a solution... Is there any chance to have that issue revisited? Monorepos would stay there and the problem would be just more and more significant.

[billcrook]

To add another case where this breaks down, when authoring data pipelines in airflow it is common practice to name each pipeline dag.py. I cannot use mypy because of this.

[adam-grant-hendry]

For the pre-commit case mentioned above (#4008 (comment), also see pre-commit/mirrors-mypy#5), try using pre-commit/mirrors-mypy with language: system and pass_filenames: false.

language: system tells pre-commit to "rely on whatever state the user's machine is in" (pre-commit/mirrors-mypy#3 (comment)), which may include mypy already installed in the environment and a mypy.ini configuration file. This may be an acceptable tradeoff if running both pre-commit and mypy from the same environment, such as a virtual environment or CI environment. When using language: system for pre-commit in CI such as GitHub Actions, it is necessary to configure the Python environment before running pre-commit (set up Python and install dependencies).

pass_filenames: false tells pre-commit not to override the list of files to check. This can be helpful simply to avoid maintaining duplicate lists of files. mypy.ini uses glob pattern matching to specify files to check, but .pre-commit-config.yaml uses Python regex for its files key.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.790
    hooks:
      - id: mypy
        language: system
        pass_filenames: false

Can you use a github repo if specifying language: system, or do you have to use repo: local instead?

[adam-grant-hendry]

pass_filenames: false tells pre-commit not to override the list of files to check. This can be helpful simply to avoid maintaining duplicate lists of files. mypy.ini uses glob pattern matching to specify files to check, but .pre-commit-config.yaml uses Python regex for its files key.

I agree with @tuukkamustonen and @jaklan

This has already been said, but to reiterate: By default, pre-commit only runs against changed files (i.e. files that have been modified and added to your git staging area with git add). Unstaged files will not be checked. Specifying pass_filenames: false tells pre-commit to not pass staged files as input to mypy. As a result, mypy will run against all files in your repo (except those excluded in a glob in your .ini/toml config settings) every time pre-commit is run.

Hence everytime pre-commit is run, if pass_filenames: false is set, it will run against all files (not just changed ones) in your project (as if you simply entered mypy . --config-file=pyproject.toml), which the pre-commit author has stated will make your mypy hook run slower.

Sometimes specifying pass_filenames: false can be useful if you want to always run a hook against the same set of files (not just changed files in your staging area), for example if you have to use pytest as a hook. However, I don't think using it if you're only wanting to avoid "maintaining a duplicate list of files" is a good idea. I think using the files: and exclude: options are better suited for this.

The real issue has to do with mypy (not pre-commit) because it can't work well with namespace packages and monorepos due to the Duplicate module error.

[macieyng]

To those that are still looking for some solution for monorepo:

  - repo: local
    hooks:
      - id: mypy
        name: mypy
        entry: bash -c 'SUBREPO=$(echo "$@" | sed "s/\/.*//"); ABS_PATH="$PWD/$@"; cd $SUBREPO; mkdir -p .mypy_cache; mypy "$ABS_PATH" --install-types --non-interactive'
        language: python
        language_version: python3
        types: [python]

1. First of all, specify pyproject.toml/mypy.ini for each of your sub-repo. You're likely to have different configs for each project.
2. For mypy specifically - make sure you're using mypy installed in your local environment. By default pre-commit runs mypy in isolated environment that doesn't have all the info about other packages (like pydantic, sqlalchemy, etc.) and therefore will make mistakes. This shouldn't be an issue if you work in monorepo environment.
3. This solution assumes that each sub-repo in monorepo has a sub-root directory under root (/project1, /project2, etc). SUBREPO is exactly this - the name of your project.
4. ABS_PATH - absolute path to the file that will be checked, to avoid duplicate module named.
5. We cd to the sub-repo directory and run mypy here on absolute path. Here is where your mypy config should be, if you share configuration among other sub-repos use --config-file ../.mypy.ini or similar. Use other args and flags as you wish.

Now mypy should have correct idea of what is going on. I did the same trick with bandit. It works correctly even if you want to commit edits on few projects in your monorepo. This solution my not be "clean", but works. It's the best I could come up with and pre-commit maintainer have said his word about supporting monorepos, so here we are.

I hope that some of you will find it useful.

[Soneji]

My solution was to add this option in the mypy configuration:

[tool.mypy]
packages = ["package1", "package2"]