False positive for @overload and type variable

[mxr]

This issue seems similar to #4619. My environment:

$ python --version
Python 3.6.5
$ mypy --version
mypy 0.620

Example file:

from random import random
from typing import Optional
from typing import overload
from typing import TypeVar

T = TypeVar("T")


@overload
def sometimes(o: T, o2: T) -> T:
    pass


@overload
def sometimes(o: None, o2: T) -> Optional[T]:
    pass


def sometimes(o: Optional[T], o2: T) -> Optional[T]:
    if o is not None:
        return o

    return None if random() < 0.5 else o2

mypy fails with:

$ mypy example.py
example.py:15: error: Overloaded function signature 2 will never be matched: signature 1's parameter type(s) are the same or broader

If I replace T with an explicit type such as List[str]

--- example.py	2018-07-31 18:19:45.000000000 -0400
+++ example-explicit.py	2018-07-31 18:19:47.000000000 -0400
@@ -1,22 +1,20 @@
 from random import random
+from typing import List
 from typing import Optional
 from typing import overload
-from typing import TypeVar
-
-T = TypeVar("T")
 
 
 @overload
-def sometimes(o: T, o2: T) -> T:
+def sometimes(o: List[str], o2: List[str]) -> List[str]:
     pass
 
 
 @overload
-def sometimes(o: None, o2: T) -> Optional[T]:
+def sometimes(o: None, o2: List[str]) -> Optional[List[str]]:
     pass
 
 
-def sometimes(o: Optional[T], o2: T) -> Optional[T]:
+def sometimes(o: Optional[List[str]], o2: List[str]) -> Optional[List[str]]:
     if o is not None:
         return o
 

then mypy passes

[Michael0x2a]

I think this error message is correct, though it unfortunately doesn't give a lot of clues why.

Basically, the issue is that if we try applying the two overloads in order, the second one will never be matched.

For example, if we try calling sometimes(None, 3), for example, mypy will try applying the first overload. In this case, the first overload matches if T is of type Optional[int]. Since we found a matching overload, mypy will not check the second one (and never will).

While mypy technically doesn't need to flag this, having overloads that can never match is probably not what a user wants, so mypy reports an error.

The two workarounds are either to:

1. Swap the order of your two overloads so sometimes(o: None, o2: T) -> Optional[T] comes first.
2. Simplify your code so it doesn't use overloads at all.

Probably option 1 will result in the most understandable code in your specific case.

[ilevkivskyi]

Is there any action item for us here or we can just close this? (This seems to behave this way by design.)

[Michael0x2a]

@ilevkivskyi -- the error message could probably be improved. Maybe add an extra line that says note: Try swapping these two overload so that signature 2 appears before signature 1 or something?

We could either track this here or close this and add another TODO to the general overloads tracking post, up to you.

[mxr]

@Michael0x2a thanks for the explanation. Swapping fixes the issue. Can you clarify why this error happens when using TypeVar but not using something like List[str]?

[Michael0x2a]

@mxr -- so, when you call an overloaded function, mypy will check each possible variant in the order you defined them and use whatever variant ended up matching first. This error occurs when it ends up being the case that it's impossible for some variant to ever be selected.

For example, in your TypeVar example the second overload will never be selected, no matter what. If we pass in any two arbitrary types X and Y as arguments into sometimes, the first overload sometimes(o: T, o2: T) -> T will always match since we can infer T to be of type Union[X, Y].

This is not the case for your list example: the arguments None and List[str] do not match your first signature sometimes(o: List[str], o2: List[str]) -> List[str] since we do not consider None to be a type or subtype of List[str]. So, mypy will move on to checking your second variant, which does match the arguments None and List[str]. Both variants can potentially be selected, depending on what inputs you provide, so there's no problem.

(Note: Older versions of mypy did use to consider None to be a subtype of all types by default. If you were to re-enable this old behavior by adding the --no-strict-optional flag, your second example will also start reporting the error.)

[ilevkivskyi]

OK, we can keep this one open. I will label this as usability issue.

[mxr]

@Michael0x2a I see, thanks. I think my biggest misunderstanding was that "we can infer T to be of type Union[X, Y]" but now the behavior makes sense