Skip to content

Exclude patterns don't work for the nested modules #10820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
arseniiarsenii opened this issue Jul 13, 2021 · 10 comments
Closed

Exclude patterns don't work for the nested modules #10820

arseniiarsenii opened this issue Jul 13, 2021 · 10 comments
Labels
bug mypy got something wrong topic-configuration Configuration files and flags

Comments

@arseniiarsenii
Copy link

My project directory structure is something like:

.
├── feecc_spoke
│   ├── __init__.py
...
│   └── waveshare_epd
...
│       ├── epd2in13d.py
│       ├── epdconfig.py
│       └── __init__.py
...
├── main.py
└── pyproject.toml

"waveshare_epd" is a third party module which is not published on PyPi, so I had to include it's files directly into the project folder. However, it generates a lot of errors which ruin my CI so I want to exclude those files from checking.

For that I use the "exclude" option in my configuration file (pyproject.toml).

The issue is that none of the regular expressions I've tried putting into the exclude option actually work.

I have tried: "waveshare_epd", ".*/waveshare_epd/", "feecc_spoke/waveshare_epd/.*", ".*epd.*", "[epd2in13d\.py|epdconfig\.py]" and many more patterns.

I have tried putting in the patterns via: command line arguments, mypy.ini file, pyproject.toml file. None of the options had any effect - the files still get included into the report.

My current configuration (a section in the pyproject.toml file) looks like this:

[tool.mypy]
strict = true
exclude = ".*epd.*"
ignore_missing_imports  = true
allow_subclassing_any = true
allow_untyped_calls = true

At this point I have spent hours at this issue over the course of multiple days and still can't find any fix for the problem.

I am using MyPy v.0.910 inside of a Poetry generated venv with base interpreter being Python 3.9.6 on a Linux system.

@arseniiarsenii arseniiarsenii added the bug mypy got something wrong label Jul 13, 2021
@hauntsaninja
Copy link
Collaborator

Exclude currently only affects mypy's recursive directory, not its import following. Similar issue to #10377. In addition to an exclude pattern, try adding:

[mypy-feecc_spoke.waveshare_epd]
follow_imports = skip

to your config file.

@arseniiarsenii
Copy link
Author

Exclude currently only affects mypy's recursive directory, not its import following. Similar issue to #10377. In addition to an exclude pattern, try adding:

[mypy-feecc_spoke.waveshare_epd]
follow_imports = skip

to your config file.

Hi, I've treid adding the suggested section into my pyproject.toml but it didn't help. I also tried these variants and they did not work either:

[mypy-feecc_spoke.waveshare_epd]
follow_imports = "skip"
[tool.mypy-feecc_spoke.waveshare_epd]
follow_imports = "skip"
[[tool.mypy.overrides]]
module = "feecc_spoke.waveshare_epd"
follow_imports = "skip"

BTW setting follow_imports value equal to skip with no quotes results in an error:
pyproject.toml: invalid literal for int() with base 0: 'skip' (line 36 column 1 char 737)

Also, I think it is worth mentioning in the docs that exclude doesn't affect follow_imports. This is some very confusing behavior.

@hauntsaninja
Copy link
Collaborator

Sorry, not too familiar with the TOML config file, maybe try the following:

[[tool.mypy.overrides]]
module = "feecc_spoke.waveshare_epd.*"
follow_imports = "skip"

Yeah, agreed on the point about documentation as a minimum thing to do here.

@arseniiarsenii
Copy link
Author

Sorry, not too familiar with the TOML config file, maybe try the following:

[[tool.mypy.overrides]]
module = "feecc_spoke.waveshare_epd.*"
follow_imports = "skip"

Yeah, agreed on the point about documentation as a minimum thing to do here.

This has worked, thank you

hauntsaninja pushed a commit to hauntsaninja/mypy that referenced this issue Jul 21, 2021
Helps with python#10842, python#10820 and others

There have been a number of issues recently where having this spelt out
a little more explicitly would help users. The introduction of
`--exclude` also (pretty understandably) confuses users who don't realise
mypy's recursive file discovery is a little separate from its dogged
import following.

I think it could be reasonable to change mypy's behaviour so that
exclude also implies follow_imports=skip (or maybe silent), but it might
be a little finnicky (one is a regex on filenames, the other is patterns
on fully qualified module names). I'm also just wary of attempting to
change this - import following configuration is probably one of the
more complicated and poorly understood parts of mypy's UX - so passing
on that for now.
hauntsaninja added a commit that referenced this issue Aug 1, 2021
Helps with #10842, #10820 and others

There have been a number of issues recently where having this spelt out
a little more explicitly would help users. The introduction of
--exclude also (pretty understandably) confuses users who don't realise
mypy's recursive file discovery is a little separate from its dogged
import following.
@JelleZijlstra JelleZijlstra added the topic-configuration Configuration files and flags label Mar 19, 2022
@adam-grant-hendry
Copy link

Unfortunately, though

[[tool.mypy.overrides]]
module = "mypkg.*"
follow_imports = "skip"
ignore_errors = true

will skip following imports and ignore errors, mypy still checks all the modules in the package. When the package is large, this slows down checking considerably (which is especially annoying for pre-commit and/or CI).

@VannTen
Copy link

VannTen commented Sep 8, 2022

There is several similar issues, and I just encountered the same. I think reporting another concrete case could help, and I'm willing to contribute to the documentation if I can find the solution. If I should instead another issue, please let me know.


I got the same problem/question, but not for a third party package, for a part of my codebase which is hard to refactor for now, while I still want to catch type errors elsewhere with the CI.

Simplified tree of the project:

.
├── pyproject.toml
├── thoth
│   └── storages
│       ├── buildlogs.py
│       ├── ceph_cache.py
│       ├── ceph.py
│       ├── cli.py
│       ├── data
│       │   ├── alembic
│       │   │   ├── env.py
│       │   │   ├── script.py.mako
│       │   │   └── versions
│       │   │       ├── 0165f03250a6_add_cve_timestamp_table.py
│       │   │       ├── 079e1d92f9d4_add_is_external_to_packageextractrun.py
│       │   └── alembic.ini
│       ├── exceptions.py
│       ├── graph
│       │   ├── enums.py
│       │   ├── __init__.py
│       │   ├── models_base.py
│       │   ├── models_performance.py
│       │   ├── models.py
│       │   ├── postgres.py
│       │   ├── query_result_base.py
│       │   └── sql_base.py
│       ├── graph_backup.py
│       ├── __init__.py
│       ├── py.typed

pyproject.toml (only relevant parts)

[tool.mypy]
exclude = [
    '^(docs|tasks|tests)|setup\.py'
]

[[tool.mypy.overrides]]
module = "thoth.*" # Trying to test the ignore_errors fonctionnality on the whole package before being more specific
follow_imports = "skip"
ignore_errors = true

Unfortunately it does not seem to have any effect (= I get mypy errors for every code under thoth/). I tried to browse through similar issues and the documentation but I can't make it work either.

The mentionned code is availabe at https://github.com/thoth-station/storages for context.

@hauntsaninja
Copy link
Collaborator

Are you sure your module name is thoth.*? Run mypy -v to see what module name mypy has, from that tree there's a good chance it has storage.*.

@VannTen
Copy link

VannTen commented Sep 14, 2022

Indeed, you're right. I'm apparently still confused by the package concept in Python. Sorry for the noise then, and thanks for the tip 👍

@erictraut
Copy link

It looks like there is no bug or feature request here. If so, this issue can be closed.

@hauntsaninja
Copy link
Collaborator

The documentation was improved a while back. #10377 can be used to track the possibility of making exclude imply a per module follow imports skip. For anything else, open a new issue.

@hauntsaninja hauntsaninja closed this as not planned Won't fix, can't repro, duplicate, stale Aug 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug mypy got something wrong topic-configuration Configuration files and flags
Projects
None yet
Development

No branches or pull requests

6 participants