mypy shows a somewhat confusing error message for functions returning a TypeVar value

[apirogov]

Not sure if this should be considered a bug, or an enhancement proposal...

I had some code looking like this:

CallbackResult = TypeVar("CallbackResult")
VisitCallback = Callable[[str], Optional[CallbackResult]]

def visititems(self, func: VisitCallback) -> CallbackResult:
    return func("test")

Actual Behavior

mypy says about this code:

error: A function returning TypeVar should receive at least one argument containing the same TypeVar  [type-var]

Expected Behavior

First, I was confused, and thought that mypy is not able to infer that CallbackResult is hidden in the function input as the return value of VisitCallback. Actually I was initially going to open an issue because of that.

Then I noticed that in a sense, mypy is correct - because I spotted that VisitCallback returns Optional[CallbackResult], while the visititems function returns a CallbackResult.

Once I fixed that, the error went away. But in this case, mypy did not really "pinpoint" the problem as well as I feel it could.

Not sure how deeply mypy plays out the type algebra to figure out what types are constructible in a context, but I feel like this specific case could be a rather "common" one (forgetting the Optional), so maybe this can be catched with a better error message.

Maybe something of the form "the function returns typevar value X, but based on its inputs can only construct Y[X, ...]" could be possible? So that it would cover Optional, Union and other simple cases like List, etc.

Your Environment

• Mypy version used: 1.1.1

[sobolevn]

The correct type for this case would be:

from typing import Callable, TypeVar, Optional

CallbackResult = TypeVar("CallbackResult")
VisitCallback = Callable[[str], Optional[CallbackResult]]

def visititems(self, func: VisitCallback[CallbackResult]) -> Optional[CallbackResult]:
    return func("test")

[apirogov]

Yes, as I said above, I am aware of that. This issue is about the error message mypy should be showing.

[cainmagi]

This warning message you met was not raised by not using Optional[...] in your returned value annotation. It was raised for the following reason:

Specifying VisitCallback without argument is equivalent to VisitCallback[Unknown], which has the same effect as specifying VisitCallback[Any].

In other words, CallbackResult will not be automatically put into the type alias, if you do not explicitly do that. VisitCallback is not the same as VisitCallback[CallbackResult].

[apirogov]

This warning message you met was not raised by not using Optional[...] in your returned value annotation. It was raised for the following reason:

Specifying VisitCallback without argument is equivalent to VisitCallback[Unknown], which has the same effect as specifying VisitCallback[Any].

In other words, CallbackResult will not be automatically put into the type alias, if you do not explicitly do that. VisitCallback is not the same as VisitCallback[CallbackResult].

Ok so alias VisitCallback is automatically parametrized by the typevar, just by using it in the definition? Maybe I had a foggy understanding of how type variables actually work in Python.

I'm still not sure whether this is actually what was going on, because adding the "missing" Optional in the return actually removed the mypy error. If your explanation is correct, shouldn't it remain mismatched until I provide the typevar? Also, isn't it the case that usually leaving our the type parameters yields more "lenient"/"coarse" checking?

But in any case, thanks for pointing out the interaction of TypeVar and type aliases!

[cainmagi]

My understanding is that when defining a type alias, the usage of TypeVar is something like using it to define a "function". TypeVar will not be automatically shared by defining two different functions.

I am not a mypy user because I am using pyright. Pyright still raises this warning even if Optional[...] is added in the returned value annotation.

Furthermore, I find the following interesting thing:

from typing import TypeVar, Optional

CallbackResult = TypeVar("CallbackResult")


def visititems() -> Optional[CallbackResult]:
    return None

The above codes will not raise any error with mypy. But pyright can recognize the misusage correctly. I think maybe mypy has a loophole here.

[sobolevn]

Yes, I think that I can fix the case where warning is gone when Optional is added.

[sobolevn]

See discussion of Optional[T] in #13061
For now, I am not going to make the change, because looks like this is intentional with respect to some corner-cases.