find_sources: find build sources recursively #9614

hauntsaninja · 2020-10-19T05:18:22Z

This is important to fix because it's a really common source of
surprising false negatives.

I also lay some groundwork for supporting passing in namespace packages
on the command line. The approach towards namespace packages that this
anticipates is that if you pass in files to mypy and you want mypy to
understand your namespace packages, you'll need to specify explicit
package roots.

We also make many fewer calls to crawl functions, since we just pass
around what we need.

Fixes python#8548 This is important to fix because it's a really common source of surprising false negatives. I also lay some groundwork for supporting passing in namespace packages on the command line. The approach towards namespace packages that this anticipates is that if you pass in files to mypy and you want mypy to understand your namespace packages, you'll need to specify explicit package roots. We also make many fewer calls to crawl functions, since we just pass around what we need.

Fixes part of python#5759 The other part is passing files arguments, which we make steps towards in python#9614

JukkaL

Thanks for fixing this bug! This was pretty bad. Looks good -- my only comments are about additional test cases.

Additional thanks for cleaning up some of the code, even if it wasn't needed for this PR.

JukkaL · 2020-10-23T14:52:28Z

test-data/unit/cmdline.test

@@ -68,6 +81,7 @@ undef
 undef
 [out]
 pkg/a.py:1: error: Name 'undef' is not defined
+pkg/subdir/a.py:1: error: Name 'undef' is not defined


Test importing from modules found via recursive directory traversal. Test cases where the directory contains __init__.py and where it doesn't exist.

Maybe test this also with --namespace-packages?

testCmdlinePackageSlash tests when the directory contains init.py and testCmdlinePackageContainingSubdir tests when it doesn't.
I added importing of the modules.

There currently isn't an interaction with --namespace-packages and cmdline tests are slow, so I won't do that unless told.
That said, I have another diff coming soon where I flesh out the package root code and make --namespace-packages the default. All these tests pass with that.

I'll also update the docs once all these changes (essentially important bug fixes + first three points of #8584) are shipped :-)

Fixes part of #5759 The other part is passing files arguments, which we make steps towards in #9614 Co-authored-by: hauntsaninja <>

This should cover the current state on master, as implemented across python#9742, python#9683, python#9632, python#9616, python#9614, etc. This will need to be changed if we can make `--namespace-packages` the default (python#9636).

This should cover the current state on master, as previously discussed / implemented across python#9742, python#9683, python#9632, python#9616, python#9614, etc. This will need to be changed if we can make `--namespace-packages` the default (python#9636). I haven't documented some of the finer points of the changes, since it felt like an inappropriate level of detail (e.g. using absolute paths when crawling, how directories with invalid package names affect crawling, etc)

This should cover the current state on master, as previously discussed / implemented across #9742, #9683, #9632, #9616, #9614, etc. This will need to be changed if we can make `--namespace-packages` the default (#9636). I haven't documented some of the finer points of the changes, since it felt like an inappropriate level of detail (e.g. using absolute paths when crawling, how directories with invalid package names affect crawling, etc) Co-authored-by: hauntsaninja <>

With python/mypy#9614, mypy now tries to collect all files in tests/, but it fails due to: tests/end2end/conftest.py: error: Duplicate module named 'conftest' (also at 'tests/end2end/features/conftest.py') tests/end2end/conftest.py: error: Are you missing an __init__.py? [misc] We should probably add __init__.py files to tests/ at some point... See #6059 and #5249

intgr · 2021-01-27T14:52:30Z

Ugh, turns out many users were relying on the old behavior and this causes issues because mypy now finds unexpected Python files from node_modules, virtualenv paths (venv), temporary directories etc. And there isn't a good way to exclude those paths (feature request #4675).

For example, creating an empty virtualenv with Python 3.9 breaks mypy:

% mkdir test
% cd test
% python3 -m venv ./venv
% echo 'a: str = 10' > test.py
% mypy .
venv/lib/python3.9/site-packages/setuptools/_distutils/command/bdist_msi.py:355: error: expected an indented block
Found 1 error in 1 file (errors prevented further checking)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I think this is a big enough regression that this change should be reverted in a hotfix release, until a longer-term solution can be figured out.

hauntsaninja force-pushed the find branch from f1832fe to 7ca6efa Compare October 19, 2020 06:29

hauntsaninja pushed a commit to hauntsaninja/mypy that referenced this pull request Oct 19, 2020

modulefinder: make -p handle namespace packages correctly

5a0c2fb

Fixes part of python#5759 The other part is passing files arguments, which we make steps towards in python#9614

hauntsaninja mentioned this pull request Oct 19, 2020

modulefinder: make -p handle namespace packages correctly #9616

Merged

JukkaL approved these changes Oct 23, 2020

View reviewed changes

test_cmdline: import modules found

333d79e

hauntsaninja merged commit 0fb671c into python:master Oct 23, 2020

hauntsaninja deleted the find branch October 23, 2020 21:56

hauntsaninja mentioned this pull request Oct 23, 2020

mypy: allow explicit specification of package roots #9632

Closed

hauntsaninja added a commit that referenced this pull request Oct 24, 2020

modulefinder: make -p handle namespace packages correctly (#9616)

a0542c1

Fixes part of #5759 The other part is passing files arguments, which we make steps towards in #9614 Co-authored-by: hauntsaninja <>

vbarbaresi mentioned this pull request Oct 25, 2020

Module '<namespace package>' has no attribute '<module/pkg in namespace>' #9619

Closed

hauntsaninja mentioned this pull request Oct 29, 2020

Redesign import handling #8584

Closed

hauntsaninja mentioned this pull request Nov 16, 2020

[Feature Request] Option to Recurse into All Subdirectories with Namespace Packages #6385

Closed

hauntsaninja mentioned this pull request Jan 18, 2021

Document new source finding behaviour #9923

Merged

michael-k mentioned this pull request Jan 26, 2021

Add ability to ignore a file or directory without modifying it #4675

Closed

intgr mentioned this pull request Jan 27, 2021

Please revert new source finding behaviour or add some way to exclude paths #9981

Closed

jdavcs mentioned this pull request Feb 19, 2021

Pin mypy to 0.790 to prevent linting errors when running v 0.8 galaxyproject/galaxy#11422

Merged

matthieudesprez mentioned this pull request May 21, 2021

<directory> is not a valid python package name #10515

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

find_sources: find build sources recursively #9614

find_sources: find build sources recursively #9614

hauntsaninja commented Oct 19, 2020 •

edited

Loading

JukkaL left a comment

JukkaL Oct 23, 2020

hauntsaninja Oct 23, 2020

hauntsaninja Oct 23, 2020

hauntsaninja Oct 23, 2020

intgr commented Jan 27, 2021

find_sources: find build sources recursively #9614

find_sources: find build sources recursively #9614

Conversation

hauntsaninja commented Oct 19, 2020 • edited Loading

JukkaL left a comment

Choose a reason for hiding this comment

JukkaL Oct 23, 2020

Choose a reason for hiding this comment

hauntsaninja Oct 23, 2020

Choose a reason for hiding this comment

hauntsaninja Oct 23, 2020

Choose a reason for hiding this comment

hauntsaninja Oct 23, 2020

Choose a reason for hiding this comment

intgr commented Jan 27, 2021

hauntsaninja commented Oct 19, 2020 •

edited

Loading