TypeGuard typechecking on filter is too strict

[SmedbergM]

Bug Report

NB: This may be more properly an issue in Typeshed, or require changes in that project to fix, but our team encountered it via MyPy.

The type annotations on builtins.filter cause correct and idiomatic usages to fail to typecheck:

To Reproduce

# filter-mypy.py
path_parts: List[Optional[str]] = ["foo", None, "bar"]

"/".join(filter(None, path_parts))
"/".join(filter(bool, path_parts))
"/".join(filter(lambda s: isinstance(s, str), path_parts))

Expected:

$> mypy filter-mypy.py
Success: no issues found in 1 source file

Actual result:

$> mypy filter-mypy.py
filter-mypy.py:6: error: Argument 1 to "filter" has incompatible type "Type[bool]"; expected "Callable[[Optional[str]], TypeGuard[str]]"
filter-mypy.py:7: error: Argument 1 to "filter" has incompatible type "Callable[[Any], bool]"; expected "Callable[[Optional[str]], TypeGuard[str]]"
Found 2 errors in 1 file (checked 1 source file)

Your Environment

• Mypy version used:

$> mypy --version
mypy 0.950 (compiled: yes)

• Python version used:

$> python --version
Python 3.10.4

• Operating system and version: Ubuntu 20.04

The following is what my editor (VSCode) pulls up when I investigate the type hints for filter, though I doubt this is the same file that MyPy itself is consulting:

class filter(Iterator[_T], Generic[_T]):
    @overload
    def __init__(self, __function: None, __iterable: Iterable[_T | None]) -> None: ...
    @overload
    def __init__(self, __function: Callable[[_S], TypeGuard[_T]], __iterable: Iterable[_S]) -> None: ...
    @overload
    def __init__(self, __function: Callable[[_T], Any], __iterable: Iterable[_T]) -> None: ...

Note that both offending lines conform to the third type signature, but fail the second.

[SmedbergM]

NB: the new behavior was introduced in python/typeshed#5661

[JelleZijlstra]

This seems like a mypy bug, not a typeshed bug. It's picking the wrong overload.

[hauntsaninja]

Yeah, mypy isn't happy with this on 0.942 either, I think the bug is a type context related.

[devhl-labs]

Any word on getting this fixed? Maybe the same issue - #12996

[Zandwhich]

Has a workaround been identified in the meantime?

[ilevkivskyi]

Although error message is bad (mypy sometimes randomly chooses "best" overload match to report error if none of overload matches), it is technically correct (assuming that best inferred type of the first argument in second and third call is Callable[[Any], bool]). To illustrate, this crashes at runtime:

from typing import Any, List, Optional

path_parts: List[Optional[str]] = ["foo", None, "bar"]

def always(x: Any) -> bool:
    return True

"/".join(filter(always, path_parts))

So to support this we will need both:

• Special-case bool constructor to be a typeguard against None
• Special-case lambda x: isinstance(x, ...) to infer a type with TypeGuard

Both are possible but non-trivial (and still a bit hacky).

[anden-akkio]

I was personally getting this error whenever I was attempting to filter a list[str] down to just the elements that contained a substring. However, my problem was that substring was from a dict.get() that could resolve to None, so this was actually a good catch from mypy.

Narrowing the type with an assertion was enough to stop mypy from producing this error.

table_name = self.table_specifier.specifier.get("table")
if not table_name:
  raise ...
filtered = list(filter(lambda x: x in <>))

[ffissore]

Another example snippet that triggers the issue.

Here mypy is happy

from typing_extensions import reveal_type

non_none_strings = list(filter(lambda x: x is not None, [None, "value"]))
reveal_type(non_none_strings)

the revealed type is builtins.list[Union[builtins.str, None]]. Since I need to join the non-none strings, I specified the type

from typing_extensions import reveal_type

non_none_strings: list[str] = list(filter(lambda x: x is not None, [None, "value"]))
reveal_type(non_none_strings)

print(", ".join(non_none_strings))

which makes mypy complain with

asd.py:3: error: Argument 1 to "filter" has incompatible type "Callable[[Any], bool]"; expected "Callable[[str | None], TypeGuard[str]]"  [arg-type]
asd.py:4: note: Revealed type is "builtins.list[builtins.str]"

The workaround I had to use was to NOT pass the lambda at all

from typing_extensions import reveal_type

non_none_strings: list[str] = list(filter(None, [None, "value"]))
reveal_type(non_none_strings)

print(", ".join(non_none_strings))

[gszabo]

In my use-case I needed the first item of a list of dicts where a particular key equals a particular value. At first I used next and filter like this:

item = next(filter(lambda v: v["my-key"] == "my-value", list_of_dicts), None)
if item is None:
    ...
else:
    ...

I got 3 mypy errors for the same line:

Argument 1 to "filter" has incompatible type "Callable[[Any], Any]"; expected "Callable[[Any], TypeGuard[Any]]"  [arg-type]
Argument 1 to "filter" has incompatible type "Callable[[Any], Any]"; expected "Callable[[Any], TypeGuard[dict[Any, Any] | None]]"  [arg-type]
Value of type "dict[Any, Any] | None" is not indexable  [index]

My workaround was to use next with a generator expression:

item = next((item for item in list_of_dicts if item["my-key"] == "my-value"), None)
if item is None:
    ...
else:
    ...