Basic support for ParamSpec type checking #11594

JukkaL · 2021-11-22T15:15:29Z

Add support for type checking several ParamSpec use cases (PEP 612).

@hauntsaninja previously added support for semantic analysis of ParamSpec
definitions, and this builds on top that foundation.

The implementation has these main things going on:

ParamSpecType that is similar to TypeVarType but has three "flavors" that
correspond to P, P.args and P.kwargs
CallableType represents Callable[P, T] if the arguments are
(*args: P.args, **kwargs: P.kwargs) -- and more generally, there can also
be arbitrary additional prefix arguments
Type variables of functions and classes can now be represented using
ParamSpecType in addition to TypeVarType

There are still a bunch of TODOs. Some of these are important to address before the
release that includes this. I believe that this is good enough to merge and remaining
issues can be fixed in follow-up PRs.

Notable missing features include these:

Concatenate
Specifying the value of ParamSpec explicitly (e.g. Z[[int, str, bool]])
Various validity checks -- currently only some errors are caught
Special case of decorating a method (functools._lru_cache_wrapper should be a descriptor class, not a callable typeshed#6347)

ilevkivskyi · 2021-11-22T16:22:02Z

mypy/checker.py

-                        arg_type = self.named_generic_type('builtins.tuple',
-                                                           [arg_type])
+                        arg_type = get_proper_type(arg_type)
+                        if not isinstance(arg_type, ParamSpecType):


I think you should update the proper_plugin to not require this. ParamSpecType can never by a target of a type alias. (Note TypeVarType is already excluded).

A good idea! Done. Also removed various now-redundant get_proper_type calls.

ilevkivskyi · 2021-11-22T16:36:23Z

mypy/expandtype.py

+        param_spec = t.param_spec()
+        if param_spec is not None:
+            repl = get_proper_type(self.variables.get(param_spec.id))
+            if isinstance(repl, CallableType):


TBH it is hard to follow the logic here: could you please add a comment on why do we need to have some non-trivial logic for callable?

Added a comment.

ilevkivskyi · 2021-11-22T16:48:31Z

mypy/typeanal.py

        finally:
            if nested:
                self.nesting_level -= 1
+        # Use type(...) to ignore proper/non-proper type distinction.
+        if (not allow_param_spec
+                and type(analyzed) is ParamSpecType


Is this really needed? You can update the plugin instead as I suggested above.

Replaced with regular isinstance.

ilevkivskyi · 2021-11-22T16:53:12Z

mypy/types.py

+    def __eq__(self, other: object) -> bool:
+        if not isinstance(other, ParamSpecType):
+            return NotImplemented
+        return self.id == other.id and self.flavor == other.flavor


Why upper_bound is missing here and in various other places like is_same_type()? Add a comment about this.

It's not user-configurable and always depends on flavor (actually currently it's always object). Added some comments.

ilevkivskyi · 2021-11-22T16:54:56Z

mypy/fixup.py

@@ -247,6 +248,9 @@ def visit_type_var(self, tvt: TypeVarType) -> None:
        if tvt.upper_bound is not None:
            tvt.upper_bound.accept(self)

+    def visit_param_spec(self, p: ParamSpecType) -> None:
+        pass  # Nothing to descend into.


What about .upper_bound? Also why it is ignored in similar fine-grained related logic?

I hadn't updated some places after I added upper_bound. It's now processed here and fine-grained logic.

sobolevn

ParamSpec! 🎉 🎏

sobolevn · 2021-11-22T16:45:12Z

mypy/type_visitor.py

@@ -63,6 +63,11 @@ def visit_deleted_type(self, t: DeletedType) -> T:
    def visit_type_var(self, t: TypeVarType) -> T:
        pass

+    @abstractmethod
+    def visit_param_spec(self, t: ParamSpecType) -> T:
+        assert False


Looks like an implementation artifact. Isn't it?

I used this for testing. Removed the assert.

sobolevn · 2021-11-22T16:47:51Z

mypy/types.py

@@ -446,13 +447,63 @@ def deserialize(cls, data: JsonDict) -> 'TypeVarType':
        )


+class ParamSpecFlavor:


Maybe Enum? Like ArgKind: https://github.com/python/mypy/blob/master/mypy/nodes.py#L1636

Sorry, Enums are really my thing 🙂

I tend to avoid enums since mypyc only partially supports them, so it's possible to accidentally write code that it's pretty slow.

sobolevn · 2021-11-22T17:03:38Z

test-data/unit/check-parameter-specification.test

+    def m(self, *args: P.args, **kwargs: P.kwargs) -> int:
+        return 1
+
+c: C[Any]


Would any other non-Any type arguments work? I haven't see any examples of it in tests 🤔

This is not implemented. I've mentioned it in the commit summary as follow-up work.

JukkaL · 2021-11-23T12:37:54Z

Thanks for the feedback! I think that I've addressed all the comments (either by updating the PR or by leaving a reply).

github-actions · 2021-11-23T12:51:27Z

Diff from mypy_primer, showing the effect of this PR on open source code:

spark (https://github.com/apache/spark.git)
+ python/pyspark/shell.py:49: error: Cannot infer type of lambda  [misc]

tornado (https://github.com/tornadoweb/tornado.git)
+ tornado/util.py:55: error: Cannot infer type of lambda

JukkaL · 2021-11-23T13:13:08Z

This is still broken:

import atexit

def f() -> None: pass
atexit.register(lambda: f())  # Cannot infer type of lambda

I'll fix this in a follow-up PR.

If ParamSpec is in the context of a lambda, treat it similar to `Callable[..., Any]`. This allows us to infer at least argument counts and kinds. Types can't be inferred since that would require "backwards" type inference, which we don't support. Follow-up to #11594.

Check that the correct ParamSpec and flavor are used in `*args` and `**kwargs`. Follow-up to #11594.

@hauntsaninja

Add support for type checking several ParamSpec use cases (PEP 612). @hauntsaninja previously added support for semantic analysis of ParamSpec definitions, and this builds on top that foundation. The implementation has these main things going on: * `ParamSpecType` that is similar to `TypeVarType` but has three "flavors" that correspond to `P`, `P.args` and `P.kwargs` * `CallableType` represents `Callable[P, T]` if the arguments are (`*args: P.args`, `**kwargs: P.kwargs`) -- and more generally, there can also be arbitrary additional prefix arguments * Type variables of functions and classes can now be represented using `ParamSpecType` in addition to `TypeVarType` There are still a bunch of TODOs. Some of these are important to address before the release that includes this. I believe that this is good enough to merge and remaining issues can be fixed in follow-up PRs. Notable missing features include these: * `Concatenate` * Specifying the value of ParamSpec explicitly (e.g. `Z[[int, str, bool]]`) * Various validity checks -- currently only some errors are caught * Special case of decorating a method (python/typeshed#6347) * `atexit.register(lambda: ...)` generates an error

If ParamSpec is in the context of a lambda, treat it similar to `Callable[..., Any]`. This allows us to infer at least argument counts and kinds. Types can't be inferred since that would require "backwards" type inference, which we don't support. Follow-up to python#11594.

Check that the correct ParamSpec and flavor are used in `*args` and `**kwargs`. Follow-up to python#11594.

JukkaL added 29 commits November 22, 2021 11:54

[WIP] Include param_spec in callable types and support reveal_Type

2007165

[WIP] Simple type inference involving ParamSpec

da6d041

[WIP] Work towards using ParamSpec in classes

2506647

[WIP] Support defining decorators

56971f0

Various fixes

774a558

Fixes to mypy daemon (not tested yet)

cc8fc46

Various updates/fixes to ParamSpecType

5b8a1c6

Fix to type semantic analysis

bb928ce

Add missing get_proper_type call

f3a2596

Fixes to self check

ef52db7

Fix type check of mypy.constraints

c76ce08

Add missing get_proper_type() calls

8a69923

Move some param spec logic to CallableType

76ffb14

Fix applying types with ParamSpec

be6f8e8

Switch to simpler ParamSpec implementation + cleanup

c1686b0

Fix inference against Any

a3cb0ad

Update test cases

fd124c5

Don't wrap ParamSpec *args and **kwargs types in tuple[...] or dict[...]

22afc1d

Implement basic type checking for ParamSpec types

044ddfd

Use Callable[P, T] in error messages

6d25da4

Support subtype checks

1047f85

Implement ParamSpec join

a9b410e

Fix type erasure

f621fb7

Add test case

080049e

Disallow ParamSpec type in bad location

504267c

Fix type check and lint

2d02b62

Minor tweaks

dd28b93

Fix TODO in string conversion

01ae7d5

Update docstrings

b6b09f1

JukkaL requested a review from hauntsaninja November 22, 2021 15:15

JukkaL requested a review from ilevkivskyi November 22, 2021 15:15

This comment has been minimized.

Sign in to view

ilevkivskyi reviewed Nov 22, 2021

View reviewed changes

sobolevn reviewed Nov 22, 2021

View reviewed changes

Respond to feedback

1af097f

ilevkivskyi approved these changes Nov 23, 2021

View reviewed changes

JukkaL merged commit ce3975d into master Nov 23, 2021

JukkaL deleted the paramspec branch November 23, 2021 13:14

JukkaL mentioned this pull request Nov 23, 2021

Improve ParamSpec type inference from lambda #11602

Merged

JukkaL added a commit that referenced this pull request Nov 23, 2021

Improve type checking of ParamSpec in calls

cfc332b

Check that the correct ParamSpec and flavor are used in `*args` and `**kwargs`. Follow-up to #11594.

JukkaL mentioned this pull request Nov 23, 2021

Improve type checking of ParamSpec in calls #11603

Merged

JukkaL added a commit that referenced this pull request Nov 24, 2021

Improve type checking of ParamSpec in calls (#11603)

b47245a

Check that the correct ParamSpec and flavor are used in `*args` and `**kwargs`. Follow-up to #11594.

JukkaL mentioned this pull request Dec 10, 2021

Unexpected "need type type annotation" with contextmanager #11704

Closed

tushar-deepsource pushed a commit to DeepSourceCorp/mypy that referenced this pull request Jan 20, 2022

Improve type checking of ParamSpec in calls (python#11603)

f6dda4a

Check that the correct ParamSpec and flavor are used in `*args` and `**kwargs`. Follow-up to python#11594.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic support for ParamSpec type checking #11594

Basic support for ParamSpec type checking #11594

JukkaL commented Nov 22, 2021 •

edited

Loading

This comment has been minimized.

ilevkivskyi Nov 22, 2021

JukkaL Nov 23, 2021

ilevkivskyi Nov 22, 2021

JukkaL Nov 23, 2021

ilevkivskyi Nov 22, 2021

JukkaL Nov 23, 2021

ilevkivskyi Nov 22, 2021

JukkaL Nov 23, 2021

ilevkivskyi Nov 22, 2021

JukkaL Nov 23, 2021

sobolevn left a comment

sobolevn Nov 22, 2021

JukkaL Nov 23, 2021

sobolevn Nov 22, 2021

JukkaL Nov 23, 2021

sobolevn Nov 22, 2021

JukkaL Nov 23, 2021

JukkaL commented Nov 23, 2021

github-actions bot commented Nov 23, 2021

JukkaL commented Nov 23, 2021

		@@ -446,13 +447,63 @@ def deserialize(cls, data: JsonDict) -> 'TypeVarType':
		)


		class ParamSpecFlavor:

Basic support for ParamSpec type checking #11594

Basic support for ParamSpec type checking #11594

Conversation

JukkaL commented Nov 22, 2021 • edited Loading

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sobolevn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JukkaL commented Nov 23, 2021

github-actions bot commented Nov 23, 2021

JukkaL commented Nov 23, 2021

JukkaL commented Nov 22, 2021 •

edited

Loading