Typing of return type of a generic function

[PeterLaudel]

Note: if you are reporting a wrong signature of a function or a class in
the standard library, then the typeshed tracker is better suited
for this report: https://github.com/python/typeshed/issues

Please provide more information to help us understand the issue:

• Are you reporting a bug, or opening a feature request?
  Reporting Bug
• Please insert below the code you are checking with mypy,
  or a mock-up repro if the source is private. We would appreciate
  if you try to simplify your case to a minimal repro.

from typing import TypeVar, Type, Optional, List, cast

T = TypeVar("T")


def some_func(something: Type[T]) -> T:
    return "something"


some_func(Optional[str])
some_func(Type[str])

• What is the actual behavior/output?

typing_test.py:7: error: Incompatible return value type (got "str", expected "T")
typing_test.py:10: error: Argument 1 to "some_func" has incompatible type "object"; expected "Type[<nothing>]"
typing_test.py:11: error: Argument 1 to "some_func" has incompatible type "object"; expected "Type[<nothing>]"

• What is the behavior/output you expect?
  Actually since this answer https://stackoverflow.com/a/42226930 I would expect it work. It also worked in previsous versions.
• What are the versions of mypy and Python you are using?
  mypy 0.780
  python 3.7.7

(You can freely edit this text, please remove all the lines
you believe are unnecessary.)

[tarcisioe]

I have made a very similar observation on #8941, but since Optional[str] and Type[str] are not reified types, they shouldn't be used like that. I find that somewhat troubling, because:

1 - It impedes the use you are showing, of specializing the return type of a function, unless T is a runtime-valid type (List[...] works, for example).
2 - Makes code that use introspection much harder to type, leaving it to be very sparsely type-checked (if at all), having to rely on other clunky techniques.

In my case, I ended up (for Optional), using an overload with optional as a bool argument, and overloading on Literal[True] and Literal[False].

And by the way, 0.770 deduced the type as Any, not as Optional[str], in your first example. I found that out by using reveal_type after 0.780 started showing the error.

In the end, I still need to use Optional in runtime to be able to validate input into dataclasses, (using typing_inspect to help), but I can't write tests with code that calls functions literally with Optional[Something] unless I #type: ignore the line.

[JukkaL]

As explained by @tarcisioe, the new behavior better aligns with runtime Python behavior.

[ltworf]

So just out of curiosity, forgetting the entire Optional thing, say that I have this function:

def create(t: Type[T]) -> T:
    return t()

a = create(int)
b = create(str)

What should the types be to let mypy know that a is an int and b is a str? I had always assumed this to work but as @tarcisioe it does not.

[andersk]

@ltworf def create(t: Callable[[], T]) -> T.

[ltworf]

It seems incorrect, or well it works for those but not if I start passing like List[int] which works with Type[T].

from typing import *
T = TypeVar('T')
def create(t: Type[T]) -> T:
    ...

reveal_type(create(int))
reveal_type(create(str))
reveal_type(create(List[int]))
reveal_type(create(Dict[int, int]))
reveal_type(create(Tuple[int]))
reveal_type(create(Union[int,str]))
reveal_type(create(Optional[int]))

Result:

testo2.py:7: note: Revealed type is 'builtins.int*'
testo2.py:8: note: Revealed type is 'builtins.str*'
testo2.py:9: note: Revealed type is 'builtins.list*[builtins.int*]'
testo2.py:10: note: Revealed type is 'builtins.dict*[builtins.int*, builtins.int*]'
testo2.py:11: note: Revealed type is '<nothing>'
testo2.py:11: error: Argument 1 to "create" has incompatible type "object"; expected "Type[<nothing>]"
testo2.py:12: note: Revealed type is '<nothing>'
testo2.py:12: error: Argument 1 to "create" has incompatible type "object"; expected "Type[<nothing>]"
testo2.py:13: note: Revealed type is '<nothing>'
testo2.py:13: error: Argument 1 to "create" has incompatible type "object"; expected "Type[<nothing>]"
Found 3 errors in 1 file (checked 1 source file)

the Type[T] thing works but only for some types and not others. It is very strange really.

I could understand it not working for union types, but why does it also fail for Tuple, but work for List and Dict???

I was using this for my typedload module with a load(data, type: Type[T]) -> T signature that turns out works for some types but not for some others, and the new mypy version revealed this.

Should I open a different bug to track this?

[tarcisioe]

@ltworf I think this won't be treated as a bug. Since #8941 and this issue were closed for the same reason, this seems like the intended behaviour. But it is a missing feature.

Basically what we are looking for is something mirroring what could be written in many other languages with generics as something along the lines of (pardon the C++)

template <typename T>
T load(std::string data) {}  // std::string for lack of a better type here

Unfortunately, there is no real syntax for that. The closest you can get is by using a class just to get the generic:

T = TypeVar('T')

class Converter(Generic[T]):
    @staticmethod
    def convert(data: Any) -> T:
        ...

But there are two issues here: this is pretty verbose, and there is no (easy, at least) runtime access to whatever T was, as far as I am aware, so I don't see that as a very good solution. If there is a way to get the real T at runtime I would find it acceptable, though I'd rather look at something terser.