Skip to content

Implement PEP 561 searching #4403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 105 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
e4b5274
Simplify find_module cache and document PEP 561
emmatyping Dec 14, 2017
6b6a97d
More work on PEP 561 impl
emmatyping Dec 15, 2017
2c936e1
Scaffold testing and fix bugs.
emmatyping Dec 16, 2017
8f64f81
Add python arg for find_module
emmatyping Dec 17, 2017
ddc526f
Get packages from Python executables
emmatyping Dec 17, 2017
329dc68
Add support for non-running Python, fix bug in impl
emmatyping Dec 21, 2017
eba98d4
Merge branch 'master' of https://github.com/python/mypy into pep561-impl
emmatyping Dec 21, 2017
5576629
Fix mypy self check
emmatyping Dec 21, 2017
2397763
Fix weird subprocess bug that works in pydevd, but not otherwise
emmatyping Dec 21, 2017
fda2a2c
Add docs for PEP561 impl
emmatyping Dec 22, 2017
59f0c49
Add initial tests for PEP 561 checking
emmatyping Dec 22, 2017
c976008
Clean up tests a bit
emmatyping Dec 22, 2017
8df8b8d
Get tests passing
emmatyping Dec 22, 2017
4ca6939
Add note about tests
emmatyping Dec 22, 2017
a96601c
Clear site-packages from cache.
emmatyping Jan 3, 2018
336fb6b
Show sample package layout, change help text
emmatyping Jan 3, 2018
8492d8b
Merge branch 'master' into pep561-impl
emmatyping Jan 3, 2018
c41eef2
Merge branch 'master' into pep561-impl
emmatyping Jan 3, 2018
1f005d7
Fix deletion of site-packages from cache
emmatyping Feb 1, 2018
45afe58
Simplify testcase file creation/deletion
emmatyping Feb 1, 2018
e7f23bb
Don't use common path and fix lint errors
emmatyping Feb 1, 2018
d74eae8
Merge branch 'master' into pep561-impl
emmatyping Feb 14, 2018
5822884
Update PEP561 implementation, docs
emmatyping Feb 15, 2018
c734bc2
Clarify python-executable purpose
emmatyping Feb 15, 2018
1d7cc7e
Document --python-executable
emmatyping Feb 15, 2018
1ac4253
Assure dir cache is always set for a key
emmatyping Feb 15, 2018
f681bf9
Fix indentation of code and command line
emmatyping Feb 15, 2018
85f233c
Fix silly indentation mistake (again)
emmatyping Feb 16, 2018
3726f18
Add check for Python packages if executable given
emmatyping Feb 16, 2018
59a2004
Finish implementation and tests for alternate executable
emmatyping Feb 16, 2018
77b3b66
Add note about advanced import methods
emmatyping Feb 16, 2018
1da6e58
Remove undocumented flag
emmatyping Feb 16, 2018
e54025f
Maintain source information in traceback
emmatyping Feb 16, 2018
056dc8f
Clean up testpackages docstrings
emmatyping Feb 16, 2018
297fac6
Even better tracebacks for package tests
emmatyping Feb 16, 2018
e35e7aa
User install on Python not executing tests
emmatyping Feb 16, 2018
ecb702e
Better debug info on failed typecheck
emmatyping Feb 16, 2018
1a41262
Check if package was installed user
emmatyping Feb 16, 2018
31dab8d
Check if user site exits
emmatyping Feb 16, 2018
ff15057
Parentheses have haunted me since Scheme
emmatyping Feb 16, 2018
9442765
Don't unconditionally set python
emmatyping Feb 16, 2018
a34e7e8
Cleanup tests
emmatyping Feb 16, 2018
0fd592a
Fix for mypy check
emmatyping Feb 16, 2018
4cbe21a
Remove debugging info
emmatyping Feb 16, 2018
f566ed0
Try to delete simple.py after all tests in case have run
emmatyping Feb 16, 2018
206f70b
Make tearDownClass a classmethod
emmatyping Feb 16, 2018
2dbff97
Don't clean up twice, as it can break stubgen tests
emmatyping Feb 16, 2018
e33ceb7
Set python version from executable
emmatyping Feb 16, 2018
644e9a5
Fix mypy self check and doc indentation
emmatyping Feb 16, 2018
af38a12
Dedent docs
emmatyping Feb 18, 2018
b7f0788
Add missed flag back
emmatyping Feb 18, 2018
0564a5c
Remove silencing of errors in installed packages
emmatyping Feb 20, 2018
d17582b
Clean up get_package_dirs
emmatyping Feb 20, 2018
085d002
Make assert a user facing error
emmatyping Feb 21, 2018
8958dc7
Refactor get_package_dirs
emmatyping Feb 21, 2018
84fd527
Make test package typesafe to avoid error
emmatyping Feb 21, 2018
4c0ff1b
Clarify docs of unsupported import features
emmatyping Feb 21, 2018
a917ff5
Reverse equality order
emmatyping Feb 21, 2018
83ab973
Correct to more sensible key type for package_dir_cache
emmatyping Feb 21, 2018
cfac326
python -> python_executable
emmatyping Feb 21, 2018
aefcb96
Set python_executable based on python_version
emmatyping Feb 21, 2018
380e301
Fix weird indentation
emmatyping Feb 21, 2018
47d8e0d
Refactor and clarify checks for package information
emmatyping Feb 21, 2018
5c19e8c
Simplify check
emmatyping Feb 21, 2018
56dfcd4
Last python -> python_executable
emmatyping Feb 21, 2018
4293dad
Use lru_cache on get_package_dirs
emmatyping Feb 21, 2018
b810d3e
Fix lint
emmatyping Feb 21, 2018
17488ad
Refactor version setting from executable
emmatyping Feb 21, 2018
e348471
Handle python executable with spaces
emmatyping Feb 21, 2018
8184937
Return early in get_package_dirs
emmatyping Feb 21, 2018
97f29cc
Refactor and clean up testpackages
emmatyping Feb 21, 2018
928683a
Fix executable flag and version inference
emmatyping Feb 21, 2018
ae65564
Clean up Python version/executable inference
emmatyping Feb 21, 2018
f6c4327
Make typedpkg python2 compatible
emmatyping Feb 21, 2018
d475d4b
Overwrite default Python version if python_executable set
emmatyping Feb 21, 2018
03cd378
Move returns into try blocks
emmatyping Feb 21, 2018
3a8ca60
Refactor python-version/executable inference
emmatyping Feb 22, 2018
e1f9494
Clarify searching for packages and mixed packages
emmatyping Feb 22, 2018
b5cbd09
Clarify default behaviour of python-executable flag
emmatyping Feb 22, 2018
fc58e4c
Comment and refactor getsitepackage usage
emmatyping Feb 22, 2018
96e2f55
Make python_executable mandatory for find_packages
emmatyping Feb 23, 2018
80f4a35
Remove py.typed file from -stubs test package
emmatyping Feb 23, 2018
dc29e51
Write testpackages to test folder
emmatyping Feb 23, 2018
be733c5
Refactor tests to not be run with package check
emmatyping Feb 23, 2018
c1264a0
Disable pep561 searching with --no-site-packages
emmatyping Feb 24, 2018
ceacbff
Fix return type to be non-optional
emmatyping Feb 24, 2018
f389f64
Set python_executable to None if not inferred
emmatyping Feb 24, 2018
1bd9f66
Set flags for tests needing 3.6
emmatyping Feb 24, 2018
806fbef
Document no-site-packages
emmatyping Feb 24, 2018
c9e3361
Fix bugs with running mypy on 3.4
emmatyping Feb 24, 2018
7be3555
Merge branch 'master' of https://github.com/python/mypy into pep561-impl
emmatyping Feb 24, 2018
5fc9d52
Disable PEP561 searching if --no-site-packages is passed
emmatyping Feb 25, 2018
ae86abd
Clean up tests
emmatyping Feb 26, 2018
0bf024f
Move no-site-packages to special-opts
emmatyping Feb 26, 2018
c9c35c1
Remove duplicate exit and error message
emmatyping Feb 26, 2018
dca9c54
Refactor name, add missing paren
emmatyping Feb 26, 2018
a929b46
Fix test import
emmatyping Feb 26, 2018
e56ffe3
Fix silly name mistake
emmatyping Feb 26, 2018
72d7b0a
Link to PEP 561 and add note about inference
emmatyping Feb 26, 2018
c9f6bc2
Fixup docs
emmatyping Feb 26, 2018
4698d23
Refactor find_module and clarify comment
emmatyping Mar 1, 2018
db0680a
Refactor find_module to not be monolithic
emmatyping Mar 5, 2018
8367e57
Key find_module_dir_cache with lib_path as well
emmatyping Mar 5, 2018
24b6742
Don't mutate dir cache with site packages
emmatyping Mar 5, 2018
afb80ea
Refactor stub package searching
emmatyping Mar 5, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 37 additions & 12 deletions docs/source/command_line.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,21 @@ summary of command line flags can always be printed using the ``-h``
flag (or its long form ``--help``)::

$ mypy -h
usage: mypy [-h] [-v] [-V] [--python-version x.y] [--platform PLATFORM] [-2]
[--ignore-missing-imports]
usage: mypy [-h] [-v] [-V] [--python-version x.y]
[--python-executable PYTHON_EXECUTABLE] [--platform PLATFORM] [-2]
[--ignore-missing-imports] [--no-site-packages]
[--follow-imports {normal,silent,skip,error}]
[--disallow-any-{unimported,expr,decorated,explicit,generics}]
[--disallow-untyped-calls] [--disallow-untyped-defs]
[--disallow-any-unimported] [--disallow-any-expr]
[--disallow-any-decorated] [--disallow-any-explicit]
[--disallow-any-generics] [--disallow-untyped-calls]
[--disallow-untyped-defs] [--disallow-incomplete-defs]
[--check-untyped-defs] [--disallow-subclassing-any]
[--warn-incomplete-stub] [--warn-redundant-casts]
[--no-warn-no-return] [--warn-return-any] [--warn-unused-ignores]
[--warn-incomplete-stub] [--disallow-untyped-decorators]
[--warn-redundant-casts] [--no-warn-no-return] [--warn-return-any]
[--warn-unused-ignores] [--warn-unused-configs]
[--show-error-context] [--no-implicit-optional] [-i]
[--quick-and-dirty] [--cache-dir DIR] [--skip-version-check]
[--strict-optional]
[--quick-and-dirty] [--cache-dir DIR] [--cache-fine-grained]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where did this option come from, and if it's introduced in this patch, should it be documented too?

Copy link
Member Author

@emmatyping emmatyping Feb 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was introduced in #4526 but not added to the docs it seems. I suppose it would be better to have it formally documented in a seperate PR. (Guido wants me to add it)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(FWIW the reason it didn't appear in the docs right away was simply that we don't regenerate this list every time we add a flag. But we should.)

[--skip-version-check] [--strict-optional]
[--strict-optional-whitelist [GLOB [GLOB ...]]]
[--junit-xml JUNIT_XML] [--pdb] [--show-traceback] [--stats]
[--inferstats] [--custom-typing MODULE]
Expand All @@ -28,9 +32,9 @@ flag (or its long form ``--help``)::
[--shadow-file SOURCE_FILE SHADOW_FILE] [--any-exprs-report DIR]
[--cobertura-xml-report DIR] [--html-report DIR]
[--linecount-report DIR] [--linecoverage-report DIR]
[--memory-xml-report DIR]
[--txt-report DIR] [--xml-report DIR] [--xslt-html-report DIR]
[--xslt-txt-report DIR] [-m MODULE] [-c PROGRAM_TEXT] [-p PACKAGE]
[--memory-xml-report DIR] [--txt-report DIR] [--xml-report DIR]
[--xslt-html-report DIR] [--xslt-txt-report DIR] [-m MODULE]
[-c PROGRAM_TEXT] [-p PACKAGE]
[files [files ...]]

(etc., too long to show everything here)
Expand Down Expand Up @@ -366,11 +370,29 @@ Here are some more useful flags:
updates the cache, but regular incremental mode ignores cache files
written by quick mode.

- ``--python-executable EXECUTABLE`` will have mypy collect type information
from `PEP 561`_ compliant
packages installed for the Python executable ``EXECUTABLE``. If not provided,
mypy will use PEP 561 compliant packages installed for the Python executable
running mypy. See :ref:`installed-packages` for more on making PEP 561
compliant packages. This flag will attempt to set ``--python-version`` if not
already set.

- ``--python-version X.Y`` will make mypy typecheck your code as if it were
run under Python version X.Y. Without this option, mypy will default to using
whatever version of Python is running mypy. Note that the ``-2`` and
``--py2`` flags are aliases for ``--python-version 2.7``. See
:ref:`version_and_platform_checks` for more about this feature.
:ref:`version_and_platform_checks` for more about this feature. This flag
will attempt to find a Python executable of the corresponding version to
search for `PEP 561`_ compliant
packages. If you'd like to disable this, see ``--no-site-packages`` below.

- ``--no-site-packages`` will disable searching for
`PEP 561`_ compliant packages.
This will also disable searching for a usable Python executable. Use this
flag if mypy cannot find a Python executable for the version of Python being
checked, and you don't need to use PEP 561 typed packages. Otherwise, use
``--python-executable``.
Copy link
Contributor

@eric-wieser eric-wieser Feb 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this last sentence mean "If you do need them and mypy cannot find the executable, specify it explicitly with --python-executable"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes


- ``--platform PLATFORM`` will make mypy typecheck your code as if it were
run under the the given operating system. Without this option, mypy will
Expand Down Expand Up @@ -453,6 +475,9 @@ For the remaining flags you can read the full ``mypy -h`` output.

Command line flags are liable to change between releases.


.. _PEP 561: https://www.python.org/dev/peps/pep-0561/

.. _integrating-mypy:

Integrating mypy into another Python application
Expand Down
1 change: 1 addition & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ Mypy is a static type checker for Python.
command_line
config_file
python36
installed_packages
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't you add checking_installed_packages too?

Copy link
Member Author

@emmatyping emmatyping Feb 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized this file is essentially a duplicate of installed_packages.rst (must have accidentally moved and checked it in somehow).

faq
cheat_sheet
cheat_sheet_py3
Expand Down
114 changes: 114 additions & 0 deletions docs/source/installed_packages.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
.. _installed-packages:

Using Installed Packages
========================

`PEP 561 <https://www.python.org/dev/peps/pep-0561/>`_ specifies how to mark
a package as supporting type checking. Below is a summary of how to create
PEP 561 compatible packages and have mypy use them in type checking.

Making PEP 561 compatible packages
**********************************

Packages that must be imported at runtime and supply type information should
put a ``py.typed`` in their package directory. For example, with a directory
structure as follows:

.. code-block:: text

setup.py
package_a/
__init__.py
lib.py
py.typed

the setup.py might look like:

.. code-block:: python

from distutils.core import setup

setup(
name="SuperPackageA",
author="Me",
version="0.1",
package_data={"package_a": ["py.typed"]},
packages=["package_a"]
)

Some packages have a mix of stub files and runtime files. These packages also require
a ``py.typed`` file. An example can be seen below:

.. code-block:: text

setup.py
package_b/
__init__.py
lib.py
lib.pyi
py.typed

the setup.py might look like:

.. code-block:: python

from distutils.core import setup

setup(
name="SuperPackageB",
author="Me",
version="0.1",
package_data={"package_b": ["py.typed", "lib.pyi"]},
packages=["package_b"]
)

In this example, both ``lib.py`` and ``lib.pyi`` exist. At runtime, ``lib.py``
will be used, however mypy will use ``lib.pyi``.

If the package is stub-only (not imported at runtime), the package should have
a prefix of the runtime package name and a suffix of ``-stubs``.
A ``py.typed`` file is not needed for stub-only packages. For example, if we
had stubs for ``package_c``, we might do the following:

.. code-block:: text

setup.py
package_c-stubs/
__init__.pyi
lib.pyi

the setup.py might look like:

.. code-block:: python

from distutils.core import setup

setup(
name="SuperPackageC",
author="Me",
version="0.1",
package_data={"package_c-stubs": ["__init__.pyi", "lib.pyi"]},
packages=["package_c-stubs"]
)

Using PEP 561 compatible packages with mypy
*******************************************

Generally, you do not need to do anything to use installed packages for the
Python executable used to run mypy. They should be automatically picked up by
mypy and used for type checking.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing that custom import hooks are not supported?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. Custom import hooks are not supported by mypy normally, and this doesn't add support for that either. Do you have a specific need for that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, but it might be nice to note in the docs exactly where mypy stops respecting the normal import mechanics. Are zip imports supported?

Copy link
Member Author

@emmatyping emmatyping Feb 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a note of that to the docs, and no, zip imports are not supported to my knowledge.

Copy link
Contributor

@eric-wieser eric-wieser Feb 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do pkg_util- and PEP420- style namespace packages fare? I suppose they can always be dealt with later, once you have a release for users in the wild to test with

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #4277 for the namespace packages implementation. That will be added later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, is there a dependency between this and #4277? Which should be merged first?


By default, mypy searches for packages installed for the Python executable
running mypy. It is highly unlikely you want this situation if you have
installed typed packages in another Python's package directory.

Generally, you can use the ``--python-version`` flag and mypy will try to find
the correct package directory. If that fails, you can use the
``--python-executable`` flag to point to the exact executable, and mypy will
find packages installed for that Python executable.

Note that mypy does not support some more advanced import features, such as zip
imports, namespace packages, and custom import hooks.

If you do not want to use typed packages, use the ``--no-site-packages`` flag
to disable searching.
Loading