"Missing return statement" error for functions having infinite loop unconditionally by looping on non-exhaustive generator

[yxliang01]

When having functions having infinite loop unconditionally by looping on non-exhaustive generator, there should be no return statements after the infinite looping block since all statements after the looping block are unreachable. Therefore, this "Missing return statement" error is a false-positive. The test case exposing this false-positive is implemented in #7373 .

Thanks

[JelleZijlstra]

The type system does not know whether a generator is finite (or exhaustive as you call it), so there's no way for mypy to know this. We could add a new type system feature, like an "infinite iterator" type, but I'm not sure that's worth the hassle.

I would recommend putting something like assert False, "unreachable" in functions affected by this bug, which also makes it clear to human readers why there appears to be a missing return statement.

[yxliang01]

Yes. The mitigation is what I'm currently doing. But, I think that for built-in infinite generators, at least white-listing is a possibility?

[ilevkivskyi]

This is actually a duplicate of #5992 that was closed as wont fix. So I would propose to close this as well, unless you want to provide a working patch.

[yxliang01]

@ilevkivskyi I actually saw that issue when I was creating this. However, I didn't see this as a duplicate because that issue mentioned creating new types for infinite generator while this describes false-positive for the no return statement rule. But, #5992 indeed partially covers this issue. What do you think?

[ilevkivskyi]

It if fine to keep this open, but it I don't think it is high priority. PRs are welcome however.

[yxliang01]

I am not familiar with mypy codebase and, unfortunately, my schedule doesn't allow me to master this codebase and do this... But, thanks for keeping this issue. How should we do about #7373 ?

[ilevkivskyi]

Honestly, I would just close that PR for now. Then later we can re-open it after this issue is fixed.

[sbrudenell]

Reading the typing module docs, I want to believe that infinite generators could be annotated as Generator[X, Y, NoReturn].

Is that a stupid idea? Or just stupid enough to work? 😄

[gvanrossum]

@sbrudenell Why don't you try it and report back here?

[sbrudenell]

from typing import Generator
from typing import NoReturn

def inf() -> Generator[None, None, NoReturn]:
    while True:
        yield

def consume_inf() -> int:
    for _ in inf():
        return 0

def consume_while_true() -> int:
    while True:
        return 0

$ mypy --version
mypy 0.790
$ mypy test.py
test.py:8: error: Missing return statement
Found 1 error in 1 file (checked 1 source file)

I'd expect consume_inf() to type-check, as consume_while_true() does. Today it complains only about consume_inf().

(In real usage, my inf()s are often something like this comment, especially for writing unit tests to wait for some condition with a timeout)

I'm trying to work out whether Generator[X, Y, NoReturn] is a good idea.

Pro:

• It's easy to use, in the use cases I can find. One could make it even more readable with something like InfiniteIterator = Generator[_T, None, NoReturn]
• I can't think of use cases it would interfere with (actually, I can't figure out what the ReturnType of Generator is ever used for)
• It's intuitive from the perspective of writing a generator function: the function should never return
• It's intuitive from the perspective of loops: the loop never exits on its own, just like a while True

Con:

• It's unintuitive / an impedence mismatch from the perspective of the low-level generator API: a constraint on ReturnType really means that .__next__(), .send() and .throw() can't raise StopIteration. Constraining exceptions is a new idea in the type system
  • (however, I have not seen the generator API widely used, except for implementing event loops. If mypy only naively supported this change, in the context of for and while control flow, I think it would cover a huge majority of the use cases)

I can't quite work out how to change mypy to implement this for further testing. I guess there ought to be some change around TypeChecker.accept_loop() or its callers. But I haven't figured out what the change should be, and I've run out of patience to shave this particular yak.

[sbrudenell]

Adding to the Cons:

• I originally thought Generator[X, Y, NoReturn] wasn't a new feature, just adding support for NoReturn in a context it wasn't supported before. But the return type of Generator isn't as simple as that.

[idmitrievsky]

I would also like to add that while mypy correctly narrows the type of x to int in the following code:

from typing import Generator, NoReturn, Union

def stop() -> NoReturn:
    raise AssertionError("should not call me")


def raise_on_str(x: Union[str, int]) -> int:
    if isinstance(x, str):
        stop()

    return x  # mypy understands that x can only contain a value of type `int` by this point

it fails to do the same when working with generators like this:

from typing import Generator, NoReturn, Union

def inf() -> Generator[str, None, NoReturn]:
    while True:
        yield "xyz"


def generate(x: Union[str, int]) -> Generator[str, None, int]:
    if isinstance(x, str):
        yield from inf()  # `yield from` will never return, because the generator never returns 

    return x  # error: Incompatible return value type (got "Union[str, int]", expected "int")

I am not familiar with mypy internals enough to know to what degree these cases are similar, but I feel like the usage pattern is the same and seems relevant to this issue.