Add type checking plugin support for functions #3299
Conversation
The plugins allow implementing special-case logic for inferring the return type of certain functions with tricky signatures such as `open` in Python 3. Include plugins for `open` and `contextlib.contextmanager`. Some design considerations: - The plugins have direct access to mypy internals. The idea is that most plugins will be included with mypy so mypy maintainers can update the plugins as needed. - User-maintained plugins are currently not supported but could be added in the future. However, the intention is to not have a stable plugin API, at least initially. User-maintained plugins would have to track mypy internal API changes. Later on, we may decide to provide a more stable API if there seems to be a significant need. The preferred way would still be to keep plugins in the mypy repo.
|
Note that this should not be merged yet, since this exposes some new errors in internal Dropbox codebases. |
|
Should we use a different term than "plugins" given that these are more an internal mechanism for special-casing than an way for users to special-case their own functions? I guess "extensions" is already taken by another mypy-specific dimension... |
mypy/checkexpr.py
Outdated
| formal_arg_types = [None] * num_formals # type: List[Optional[Type]] | ||
| for formal, actuals in enumerate(formal_to_actual): | ||
| for actual in actuals: | ||
| formal_arg_types[formal] = arg_types[actual] |
There was a problem hiding this comment.
Aren't there some edge cases where map_actuals_to_formals() returns overlapping mappings? (IIRC related to *args and worse.)
mypy/checkexpr.py
Outdated
| for actual in actuals: | ||
| formal_arg_types[formal] = arg_types[actual] | ||
| return self.function_plugins[fullname]( | ||
| formal_arg_types, inferred_ret_type, args, self.chk.named_generic_type) |
There was a problem hiding this comment.
Shouldn't the plugins have access to the Errors object too?
mypy/funcplugins.py
Outdated
| args: List[Expression], | ||
| named_generic_type: Callable[[str, List[Type]], Type]) -> Type: | ||
| """Infer a better return type for 'contextlib.contextmanager'.""" | ||
| arg_type = arg_types[0] |
There was a problem hiding this comment.
This would fail if there were no arguments, right? (Which would give some other error but might still get here?)
| with f('') as s: | ||
| reveal_type(s) | ||
| [out] | ||
| _program.py:13: error: Revealed type is 'def (x: builtins.int) -> contextlib.GeneratorContextManager[builtins.str*]' |
There was a problem hiding this comment.
I just realized that that class is misnamed in typeshed, it should be _GeneratorContextManager (to match what it's called at runtime). I also don't understand what its __call__ method is for (contextlib doesn't seem to have reference docs, and the source has few clues).
There was a problem hiding this comment.
The __call__ is so that @contextmanager-decorated functions can also be used as decorators themselves (executing the decorated function within the context). Nick Coghlan has said that he considers this feature a design mistake in contextlib.
|
The API for a plugin seems to take the formal argument types (subject to @gvanrossum's commentary), and also the actual arguments, but not the formal-to-actual mapping? That seems like it'd make using the actual arguments a little rough, if people were doing any kind of passing arguments you don't expect by name in an order you don't expect. I feel like it should get all the necessary information, in some kind of normalized form. (Outside this diff, we should refactor to have an easier-to-understand package that represents a mapping of actual to formal args) |
Also add comments and some defensive checks.
|
We forgot to check this against our internal codebases. There are a few errors. |
|
I took a closer look, and it seems like the internal errors are all legitimate (except for a couple cases of |
* master: (23 commits) Make return type of open() more precise (python#3477) Add test cases that delete a file during incremental checking (python#3461) Parse each format-string component separately (python#3390) Don't warn about returning Any if it is a proper subtype of the return type (python#3473) Add __setattr__ support (python#3451) Remove bundled lib-typing (python#3337) Move version of extensions to post-release (python#3348) Fix None slice bounds with strict-optional (python#3445) Allow NewType subclassing NewType. (python#3465) Add console scripts (python#3074) Fix 'variance' label. Change label for variance section to just 'variance' (python#3429) Better error message for invalid package names passed to mypy (python#3447) Fix last character cut in html-report if file does not end with newline (python#3466) Print pytest output as it happens (python#3463) Add mypy roadmap (python#3460) Add flag to avoid interpreting arguments with a default of None as Optional (python#3248) Add type checking plugin support for functions (python#3299) Mismatch of inferred type and return type note (python#3428) Sync typeshed (python#3449) ...
The plugins allow implementing special-case logic for
inferring the return type of certain functions with
tricky signatures such as
openin Python 3.Include plugins for
openandcontextlib.contextmanager.Some design considerations:
The plugins have direct access to mypy internals. The
idea is that most plugins will be included with mypy
so mypy maintainers can update the plugins as needed.
User-maintained plugins are currently not supported but
could be added in the future. However, the intention is
to not have a stable plugin API, at least initially.
User-maintained plugins would have to track mypy internal
API changes. Later on, we may decide to provide a more
stable API if there seems to be a significant need. The
preferred way would still be to keep plugins in the
mypy repo.
Work towards #2337.