Support TypeGuard (PEP 647) #9865

gvanrossum · 2020-12-30T22:14:51Z

While PEP 647 is still in draft mode, this is (or if it isn't will soon be) a full implementation.

This depends on a typeshed PR: python/typeshed#4879.

This is a bit ugly -- in find_isinstance_check() we wrap the type in a way that is only recognized in on specific other place, narrow_type_from_binder().

hauntsaninja

This looks pretty reasonable to me!

Even though it's not in the direct path of changes, might be worth adding a test for interactions with the walrus operator.

Also, I'm guessing higher order use of type guarded functions doesn't work? E.g. something like:

def filter(fn: Callable[[T1], TypeGuard[T2]], it: Iterable[T1]) -> Iterator[T2]: ...

reveal_type(filter(...))

hauntsaninja · 2020-12-30T23:36:23Z

mypy/checker.py

+                        and isinstance(node.callee.node, SYMBOL_FUNCBASE_TYPES)
+                        and isinstance(node.callee.node.type, CallableType)
+                        and node.callee.node.type.type_guard is not None):
+                    if len(node.args) < 1:  # TODO: Is this an error?


This seems fine to me, it's similar to what happens with if isinstance() + you have a TODO in your tests to error at the definition site for functions that don't take args

- walrus - higher-order functions

gvanrossum · 2021-01-01T01:40:45Z

Also, I'm guessing higher order use of type guarded functions doesn't work?

Hm, it doesn't, but it really should. Unfortunately that's going to require a different approach, and I'm not sure how to do it. There are an awful lot of places where a Callable's ret_type attribute is used, and I'm hesitant to try and audit them all to see whether I should redirect the code to the type_guard attribute or not.

hauntsaninja · 2021-01-01T03:33:45Z

I don't have a general answer for you, but maybe add something to this line and you get lucky and it mostly just works?

mypy/mypy/constraints.py

Line 460 in aa97427

res.extend(infer_constraints(template.ret_type, cactual.ret_type,

gvanrossum · 2021-01-01T05:29:39Z

I don't have a general answer for you, but maybe add something to this line and you get lucky and it mostly just works?

mypy/mypy/constraints.py

Line 460 in aa97427

res.extend(infer_constraints(template.ret_type, cactual.ret_type,

Thanks, right on the nose. It was this and a bit in expandtype.py.

gvanrossum · 2021-01-01T05:41:14Z

However, a type guard that's a method doesn't work (no generics involved even). How are those different?

JukkaL

Thanks for working on this! Overall looks good -- the changes are clean and localized. This may not be a very commonly needed feature, but in cases where this is useful the current alternatives are all quite bad.

I did a quick review pass and left suggestions about additional test cases. I'll read the PEP in detail and play around with the implementation, and I'll do another round of review afterwards.

JukkaL · 2021-01-06T17:02:19Z

test-data/unit/check-typeguard.test

+def is_point(a: object) -> TypeGuard[points.Point]: pass
+[file points.py]
+class Point: pass
+[builtins fixtures/tuple.pyi]


Idea about additional tests (sorry if some of these already exist):

Test a type guard function with a non-empty body. Check that a non-boolean return value is flagged as an error.

Test a type guard that narrows down Dict[str, object] to a TypedDict.

Test using type guard with an assert.

Test using two type guards with 'or', such as if is_int(x) or is_float(x):.

Test narrowing down from Any using a type guard.

JukkaL · 2021-01-06T17:03:19Z

mypy/types.py

@@ -270,6 +270,16 @@ def copy_modified(self, *,
            self.line, self.column)


+class TypeGuardType(Type):
+    """Only used by find_instance_check() etc."""


Add link to the PEP in the docstring.

JukkaL · 2021-01-07T17:01:02Z

A few more ideas about tests:

Test that if not is_int(x) does not narrow Union[int, str] to str (also in the else block of if is_int(x)).
Test a type guard implemented as a class method.

gvanrossum · 2021-01-07T17:23:48Z

Also (note to self) some tests with keyword args and *args, **kwargs (not sure what's supposed to happen for those).

JukkaL · 2021-01-07T19:40:27Z

Two more things that may be worth testing:

Overloaded function where one item/all items have a type guard.
Decorated type guard function (decorator has type T -> T).

JukkaL · 2021-01-08T11:10:02Z

More possible things to test:

If a method is a type guard, do we need to ensure that any overrides in subclasses are type guards (and compatible type guards)?
If we have a higher-order function that takes a type guard, test that the passed function is a suitable type guard (currently it looks like we can pass an arbitrary function that returns bool).

(It's also okay to create follow-up issues for some of these if they don't work currently.)

These mostly come from Jukka's suggestions. Most pass; 4 are currently being skipped: [case testTypeGuardWithKeywordArgsSwapped-skip] is_float(b=1, a=a) incorrectly asserts that b is a float [case testTypeGuardWithStarArgsTuple-skip] [case testTypeGuardWithStarArgsList-skip] Horrible things happen with *args [case testTypeGuardOverload-skip] A plain bool function is accepted by a higher-order function (overload) requiring a type guard [case testTypeGuardMethodOverride-skip] A plain bool function in a subclass incorrectly is allowed to override a type guard in the base class

gvanrossum · 2021-01-08T22:26:54Z

I added tests for all your suggestions (the "overload" test doubles for overloading and higher-order-functions).

This found several problems -- for now I'm just skipping those 5 tests. I'll look into fixing them but I might take your offer of filing issues for some of them.

JukkaL

Thanks for the updates! Looks good now. Left a few minor comments, feel free to merge once you've addressed them (or decided that they are not worth it).

test-data/unit/check-typeguard.test

JukkaL · 2021-01-11T15:24:34Z

test-data/unit/check-typeguard.test

+    cc = filter(is_int, a)
+    reveal_type(cc)  # N: Revealed type is 'typing.Iterator[builtins.int]'
+    # TODO: Make this work -- this matches the third overload instead of the second
+    dd = filter(non_tg, a)


What if you switch the order of the second and third overload items?

mypy/expandtype.py

gvanrossum · 2021-01-18T17:01:22Z

Thanks, I will merge after the test pass. I hope this can be added to the 0.800 release. (Even if it's not perfect, it shouldn't change anything for users who don't use TypeGuard.)

The main significant issue is that overloads using a type guard as higher-order function don't work right. I'll file a separate issue for that.

JukkaL · 2021-01-21T17:36:50Z

Sorry, this will miss 0.800, but we are hoping to make another release fairly soon after 0.800 that switches to modular typeshed and this will be included

mortoray · 2021-02-13T12:26:51Z

This may not be a very commonly needed feature, but in cases where this is useful the current alternatives are all quite bad.

For reference, custom type guards are extremely useful in any testing library as well as any data validation and/or file loading. In those domains type validation functions are frequently used. I think we'll find this feature being used everywhere once available.

I'll be happy when this is available!

In python/typeshed#5473, I tried to switch a number of `inspect` functions to use the new `TypeGuard` functionality. Unfortunately, mypy-primer found a number of crashes in third-party libraries in places where a TypeGuard function was ANDed together with some other check. Examples: - https://github.com/sphinx-doc/sphinx/blob/4.x/sphinx/util/inspect.py#L252 - https://github.com/sphinx-doc/sphinx/blob/4.x/sphinx/ext/coverage.py#L212 - https://github.com/streamlit/streamlit/blob/develop/lib/streamlit/elements/doc_string.py#L105 The problems trace back to the decision in #9865 to make TypeGuardType not inherit from ProperType: in various conditions that are more complicated than a simple `if` check, mypy wants everything to become a ProperType. Therefore, to fix the crashes I had to make TypeGuardType a ProperType and support it in various visitors.

gvanrossum added 11 commits December 30, 2020 09:54

Tentative first steps for TypeGuard (PEP 647)

9469ffc

Rename is_type_guard to type_guard, fix repr and serialization

884d9af

Perhaps naive but passes the test

f55b284

Add more tests, tweak tests (one new test fails)

bdc97ef

Update typeshed

0ed5434

Make is_str_list() test work

47df17d

This is a bit ugly -- in find_isinstance_check() we wrap the type in a way that is only recognized in on specific other place, narrow_type_from_binder().

Add a test showing a controversial behavior

fbfeeb6

Fix mypy and lint

03e566f

Update typeshed and fix test name typo

93387bd

Sync typeshed to the version that has TypeGuard

708f2e5

Merge branch 'master' into typeguard

cf067d1

hauntsaninja reviewed Dec 30, 2020

View reviewed changes

Add two new tests

0d2eb06

- walrus - higher-order functions

Minimal changes to make filter() test pass

8264f8d

Fix mypy error in new code (corrected)

4335785

gvanrossum force-pushed the typeguard branch from 0e1776a to 4335785 Compare January 1, 2021 18:05

gvanrossum added 4 commits January 1, 2021 16:30

Make methods work (adds a field to RefExpr)

b34d2ac

Move walrus test to 3.8-only test file

9bdd779

Merge branch 'master' into typeguard

249b6e5

Add cross-module test; remove test TODOs

639b0fc

gvanrossum requested a review from JukkaL January 4, 2021 04:47

Merge remote-tracking branch 'origin/master' into typeguard

4400702

JukkaL reviewed Jan 6, 2021

View reviewed changes

gvanrossum mentioned this pull request Jan 8, 2021

Add support for simple lambda type guard #9890

Open

gvanrossum added 2 commits January 8, 2021 12:55

Merge remote-tracking branch 'origin/master' into typeguard

d108c6e

Require that a type guard's first argument is positional

370818f

JukkaL approved these changes Jan 11, 2021

View reviewed changes

gvanrossum added 4 commits January 11, 2021 13:40

Capitalize error message

9062adb

Avoid using **extra if possible

5e76923

Clean up testTypeGuardOverload -- it still fails, though

37d2a5f

Merge remote-tracking branch 'origin/master' into typeguard

9d7b6c6

Fix lint

896c90e

This was referenced Jan 18, 2021

Release 0.800 planning #9926

Closed

PEP 647 support mishandles higher-order functions #9927

Closed

gvanrossum merged commit fffbe88 into master Jan 18, 2021

gvanrossum deleted the typeguard branch January 18, 2021 18:13

gvanrossum mentioned this pull request Jan 18, 2021

PEP 647: Error if overriding typeguard with non-typeguard in subclass #9928

Closed

JelleZijlstra mentioned this pull request May 18, 2021

Fix crash on TypeGuard plus "and" #10496

Merged

mcmasn-amzn mentioned this pull request Sep 8, 2021

Docs: walrus TypeGuard #11075

Closed

justinchuby mentioned this pull request Sep 8, 2022

[ONNX] Make _is_value a TypeGuard pytorch/pytorch#84649

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support TypeGuard (PEP 647) #9865

Support TypeGuard (PEP 647) #9865

gvanrossum commented Dec 30, 2020

hauntsaninja left a comment •

edited

Loading

hauntsaninja Dec 30, 2020 •

edited

Loading

gvanrossum commented Jan 1, 2021

hauntsaninja commented Jan 1, 2021

gvanrossum commented Jan 1, 2021

gvanrossum commented Jan 1, 2021 via email

JukkaL left a comment

JukkaL Jan 6, 2021

JukkaL Jan 6, 2021

JukkaL commented Jan 7, 2021

gvanrossum commented Jan 7, 2021 via email

JukkaL commented Jan 7, 2021

JukkaL commented Jan 8, 2021

gvanrossum commented Jan 8, 2021

JukkaL left a comment

JukkaL Jan 11, 2021

gvanrossum commented Jan 18, 2021

JukkaL commented Jan 21, 2021

mortoray commented Feb 13, 2021

Support TypeGuard (PEP 647) #9865

Support TypeGuard (PEP 647) #9865

Conversation

gvanrossum commented Dec 30, 2020

hauntsaninja left a comment • edited Loading

Choose a reason for hiding this comment

hauntsaninja Dec 30, 2020 • edited Loading

Choose a reason for hiding this comment

gvanrossum commented Jan 1, 2021

hauntsaninja commented Jan 1, 2021

gvanrossum commented Jan 1, 2021

gvanrossum commented Jan 1, 2021 via email

JukkaL left a comment

Choose a reason for hiding this comment

JukkaL Jan 6, 2021

Choose a reason for hiding this comment

JukkaL Jan 6, 2021

Choose a reason for hiding this comment

JukkaL commented Jan 7, 2021

gvanrossum commented Jan 7, 2021 via email

JukkaL commented Jan 7, 2021

JukkaL commented Jan 8, 2021

gvanrossum commented Jan 8, 2021

JukkaL left a comment

Choose a reason for hiding this comment

JukkaL Jan 11, 2021

Choose a reason for hiding this comment

gvanrossum commented Jan 18, 2021

JukkaL commented Jan 21, 2021

mortoray commented Feb 13, 2021

hauntsaninja left a comment •

edited

Loading

hauntsaninja Dec 30, 2020 •

edited

Loading