Make `functools.lru_cache` a descriptor class #9834

Azureblade3808 · 2023-03-03T08:20:45Z

Fixes #6347 .

github-actions · 2023-03-03T08:30:38Z

Diff from mypy_primer, showing the effect of this PR on open source code:

dacite (https://github.com/konradhalas/dacite)
+ dacite/core.py:51: error: Argument "localns" to "get_type_hints" has incompatible type "Optional[FrozenDict]"; expected "Optional[Dict[str, Any]]"  [arg-type]

pip (https://github.com/pypa/pip)
+ src/pip/_internal/models/link.py:220: error: Missing positional argument "url" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ src/pip/_internal/models/link.py:220: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[LinkHash]"  [arg-type]
+ src/pip/_internal/models/link.py:408: error: Missing positional argument "url" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ src/pip/_internal/models/link.py:408: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[LinkHash]"  [arg-type]
+ src/pip/_internal/index/package_finder.py:890: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ src/pip/_internal/index/package_finder.py:909: error: Argument 1 has incompatible type "Optional[str]"; expected <nothing>  [arg-type]
+ src/pip/_internal/index/package_finder.py:910: error: Argument "specifier" has incompatible type "SpecifierSet"; expected <nothing>  [arg-type]
+ src/pip/_internal/index/package_finder.py:911: error: Argument "hashes" has incompatible type "Hashes"; expected <nothing>  [arg-type]
+ src/pip/_internal/self_outdated_check.py:177: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ src/pip/_internal/resolution/resolvelib/factory.py:280: error: Argument "project_name" has incompatible type "NormalizedName"; expected <nothing>  [arg-type]
+ src/pip/_internal/resolution/resolvelib/factory.py:281: error: Argument "specifier" has incompatible type "SpecifierSet"; expected <nothing>  [arg-type]
+ src/pip/_internal/resolution/resolvelib/factory.py:282: error: Argument "hashes" has incompatible type "Hashes"; expected <nothing>  [arg-type]
+ src/pip/_internal/resolution/resolvelib/factory.py:604: error: Argument 1 has incompatible type "NormalizedName"; expected <nothing>  [arg-type]
+ src/pip/_internal/commands/list.py:236: error: Argument 1 has incompatible type "NormalizedName"; expected <nothing>  [arg-type]

jax (https://github.com/google/jax)
+ jax/_src/sharding.py:138: error: Signature of "devices_indices_map" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding.py:149: error: Signature of "shard_shape" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding.py:353: error: Signature of "_to_xla_op_sharding" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding.py:507: error: Signature of "devices_indices_map" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding.py:507: error: Signature of "devices_indices_map" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding.py:519: error: Signature of "shard_shape" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding.py:519: error: Signature of "shard_shape" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding.py:604: error: Signature of "_to_xla_op_sharding" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding.py:700: error: Signature of "devices_indices_map" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding.py:700: error: Signature of "devices_indices_map" incompatible with supertype "Sharding"  [override]
+ jax/_src/pjit.py:2219: error: Argument 1 has incompatible type "int"; expected <nothing>  [arg-type]
+ jax/experimental/gda_serialization/serialization.py:51: error: Argument 1 has incompatible type "Tuple[int, ...]"; expected <nothing>  [arg-type]

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/tools.py:30: error: Unused "type: ignore" comment

rich (https://github.com/Textualize/rich)
+ rich/theme.py:25: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/theme.py:25: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"  [arg-type]
+ rich/theme.py:54: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/theme.py:54: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"  [arg-type]
+ rich/style.py:147: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/style.py:147: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"  [arg-type]
+ rich/style.py:368: error: Argument 1 has incompatible type "ColorSystem"; expected <nothing>  [arg-type]
+ rich/style.py:371: error: Argument 1 has incompatible type "ColorSystem"; expected <nothing>  [arg-type]
+ rich/style.py:372: error: Argument "foreground" has incompatible type "bool"; expected <nothing>  [arg-type]
+ rich/style.py:391: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/style.py:391: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"  [arg-type]
+ rich/style.py:527: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/style.py:527: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"  [arg-type]
+ rich/style.py:554: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/style.py:554: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"  [arg-type]
+ rich/style.py:735: error: Argument 1 has incompatible type "Optional[Style]"; expected <nothing>  [arg-type]
+ rich/segment.py:169: error: Argument 1 has incompatible type "Segment"; expected <nothing>  [arg-type]
+ rich/segment.py:169: error: Argument 2 has incompatible type "int"; expected <nothing>  [arg-type]
+ rich/markup.py:157: error: Missing positional argument "style" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/markup.py:157: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"  [arg-type]
+ rich/markup.py:214: error: Missing positional argument "style" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/markup.py:214: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"  [arg-type]
+ rich/jupyter.py:72: error: Argument 1 has incompatible type "TerminalTheme"; expected <nothing>  [arg-type]
+ rich/console.py:1471: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/console.py:1471: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"  [arg-type]
+ rich/console.py:2207: error: Argument 1 has incompatible type "TerminalTheme"; expected <nothing>  [arg-type]
+ rich/console.py:2219: error: Argument 1 has incompatible type "TerminalTheme"; expected <nothing>  [arg-type]
+ rich/color.py:552: error: Argument 1 has incompatible type "ColorTriplet"; expected <nothing>  [arg-type]
+ rich/color.py:565: error: Argument 1 has incompatible type "ColorTriplet"; expected <nothing>  [arg-type]
+ rich/ansi.py:172: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/ansi.py:172: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"  [arg-type]
+ rich/progress_bar.py:145: error: Argument 1 has incompatible type "Style"; expected <nothing>  [arg-type]
+ rich/progress_bar.py:145: error: Argument 2 has incompatible type "Style"; expected <nothing>  [arg-type]
+ rich/progress_bar.py:145: error: Argument 3 has incompatible type "Optional[str]"; expected <nothing>  [arg-type]
+ rich/progress_bar.py:145: error: Argument 4 has incompatible type "bool"; expected <nothing>  [arg-type]
+ rich/progress_bar.py:145: error: Argument "ascii" has incompatible type "bool"; expected <nothing>  [arg-type]

manticore (https://github.com/trailofbits/manticore)
+ manticore/core/smtlib/solver.py:505: error: Signature of "can_be_true" incompatible with supertype "Solver"  [override]
+ manticore/core/smtlib/solver.py:605: error: Signature of "get_all_values" incompatible with supertype "Solver"  [override]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/dtypes/cast.py:601: error: Unused "type: ignore" comment
+ pandas/core/groupby/ops.py:541: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ pandas/core/groupby/ops.py:541: error: Argument 2 has incompatible type "str"; expected <nothing>  [arg-type]
+ pandas/core/groupby/ops.py:541: error: Argument 4 has incompatible type "bool"; expected <nothing>  [arg-type]

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/polars/__init__.py:240: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ ibis/backends/datafusion/__init__.py:283: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ ibis/backends/pandas/__init__.py:187: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ ibis/backends/base/sql/__init__.py:376: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ ibis/backends/duckdb/tests/conftest.py:19: error: Argument 1 has incompatible type "Path"; expected <nothing>  [arg-type]
+ ibis/backends/snowflake/tests/conftest.py:35: error: Argument 1 has incompatible type "Path"; expected <nothing>  [arg-type]

mypy (https://github.com/python/mypy)
+ mypy/find_sources.py:161: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ mypy/find_sources.py:203: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]

pylint (https://github.com/pycqa/pylint)
+ pylint/utils/file_state.py:93: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ pylint/message/message_definition_store.py:79: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ pylint/message/message_definition_store.py:89: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ pylint/lint/message_state_handler.py:131: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ pylint/lint/message_state_handler.py:159: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ pylint/lint/pylinter.py:1339: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ pylint/lint/pylinter.py:1366: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ pylint/checkers/format.py:452: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]

aiohttp (https://github.com/aio-libs/aiohttp)
+ aiohttp/connector.py:931: error: Argument 1 has incompatible type "bool"; expected <nothing>  [arg-type]
+ aiohttp/connector.py:937: error: Argument 1 has incompatible type "bool"; expected <nothing>  [arg-type]
+ aiohttp/connector.py:938: error: Argument 1 has incompatible type "bool"; expected <nothing>  [arg-type]

paroxython (https://github.com/laowantong/paroxython)
+ paroxython/assess_costs.py:85: error: Incompatible types in assignment (expression has type "_lru_cache_wrapper[[int, int], float]", variable has type "Callable[[VarArg(<nothing>), KwArg(<nothing>)], float]")  [assignment]
+ paroxython/assess_costs.py:85: note: "_lru_cache_wrapper[[int, int], float].__call__" has type "Callable[[Arg(int, 'start'), Arg(int, 'stop')], float]"
+ paroxython/assess_costs.py:89: error: "Callable[[VarArg(<nothing>), KwArg(<nothing>)], float]" has no attribute "cache_clear"  [attr-defined]
+ paroxython/assess_costs.py:128: error: Argument 1 has incompatible type "TaxonName"; expected <nothing>  [arg-type]
+ paroxython/recommend_programs.py:296: error: Argument 1 has incompatible type "TaxonName"; expected <nothing>  [arg-type]
+ paroxython/map_taxonomy.py:243: error: Argument 1 has incompatible type "LabelName"; expected <nothing>  [arg-type]

isort (https://github.com/pycqa/isort)
+ isort/deprecated/finders.py:296: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]
+ isort/deprecated/finders.py:327: error: Argument 1 has incompatible type "str"; expected <nothing>  [arg-type]

psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/rows.py:146: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "*Generator[Optional[bytes], None, None]"; expected "bytes"  [arg-type]

jinja (https://github.com/pallets/jinja)
+ src/jinja2/environment.py:1186: error: Unused "type: ignore" comment
+ src/jinja2/environment.py:1201: error: Unused "type: ignore" comment

tmke8 · 2023-03-03T14:16:30Z

It seems mypy gets confused when lru_cache is applied to a method instead of a function:

from functools import lru_cache

class C:
    @lru_cache
    def count_vowels(self, sentence: str) -> int:
        return sum(sentence.count(vowel) for vowel in 'AEIOUaeiou')

c = C()
reveal_type(C.count_vowels)
reveal_type(c.count_vowels)

results in this mypy output:

main.py:9: note: Revealed type is "functools._lru_cache_wrapper[[self: main.C, sentence: builtins.str], builtins.int]"
main.py:10: note: Revealed type is "def (*<nothing>, **<nothing>) -> builtins.int"

whereas pyright handles it correctly:

main.py:9:13 - information: Type of "C.count_vowels" is "_lru_cache_wrapper[(self: C, sentence: str), int]"
main.py:10:13 - information: Type of "c.count_vowels" is "(sentence: str) -> int"

AlexWaygood · 2023-03-03T14:17:38Z

It seems mypy gets confused when lru_cache is applied to a method instead of a function:

from functools import lru_cache

class C:
    @lru_cache
    def count_vowels(self, sentence: str) -> int:
        return sum(sentence.count(vowel) for vowel in 'AEIOUaeiou')

c = C()
reveal_type(C.count_vowels)
reveal_type(c.count_vowels)

Is there a mypy issue open about this, do you know? If not, it might be useful to open one to track this. It's come up many times over here at typeshed.

tmke8 · 2023-03-03T14:25:53Z

Is there a mypy issue open about this, do you know? If not, it might be useful to open one to track this. It's come up many times over here at typeshed.

This is the closest I could find: python/mypy#13911

AlexWaygood · 2023-03-03T14:29:36Z

Is there a mypy issue open about this, do you know? If not, it might be useful to open one to track this. It's come up many times over here at typeshed.

This is the closest I could find: python/mypy#13911

Hmm, not really a great summary of the issue. I can try and write something up later in a new issue if you don't beat me to it :-)

tmke8 · 2023-03-03T14:55:07Z

I think I'll leave it to you as you have more context, but here is my attempt at a standalone reproduction:

from __future__ import annotations
from typing import Any, Callable, Generic, TypeVar, overload, cast
from typing_extensions import Concatenate, ParamSpec, Self

P = ParamSpec("P")
P1 = ParamSpec("P1")
R = TypeVar("R", covariant=True)
S = TypeVar("S")

class Wrapper(Generic[P, R]):
    @overload
    def __get__(self, instance: None, owner: type[Any] | None) -> Self: ...
    @overload
    def __get__(
        self: Wrapper[Concatenate[S, P1], R], instance: S, owner: type[Any] | None
    ) -> Callable[P1, R]: ...

def decorator(f: Callable[P, R]) -> Wrapper[P, R]:
    return cast(Wrapper[P, R], f)

class C:
    @decorator
    def m(self, x: int) -> int:
        return 2 * x

c = C()
reveal_type(C.m)  # N: Revealed type is "main.Wrapper[[self: main.C, x: builtins.int], builtins.int]"
reveal_type(c.m)  # N: Revealed type is "def (*<nothing>, **<nothing>) -> builtins.int"

It got quite a bit more complicated than I expected. I'm actually confused why you have to use a descriptor class when decorating a method... I can see why people are complaining about this: python/typing#1357

tmke8 · 2023-03-04T14:05:27Z

After thinking about this more, I came to think that the proper solution to this problem is to have a typing mechanism for functions to have attributes, so that type checkers can just apply the normal method bounding mechanism and the descriptor class will be unnecessary. I posted the proposal to typing-sig: https://mail.python.org/archives/list/typing-sig@python.org/thread/35FTOYUG2IPCRIIH3MQKEVV7XW3V7ASB/

The proposal would solve this and also the functools.wraps issue.

AlexWaygood · 2023-03-16T15:08:18Z

@Azureblade3808, I haven't looked too deeply at this, but I'm afraid we can't merge this as-is -- the mypy_primer diff above indicates it would be far too disruptive.

If you'd like to continue working on this and try to get the mypy_primer diff down, please feel free to open a new PR!

Azureblade3808 and others added 2 commits March 3, 2023 16:01

Make functools.lru_cache a descriptor class

3bdd0f1

[pre-commit.ci] auto fixes from pre-commit.com hooks

41df0f0

AlexWaygood closed this Mar 16, 2023

tmke8 mentioned this pull request May 6, 2023

callables are inferred to be descriptors when they shouldn't always be, protocols are inferred to not be descriptors when maybe they should, and mypy converts between them with no error python/mypy#15189

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `functools.lru_cache` a descriptor class #9834

Make `functools.lru_cache` a descriptor class #9834

Azureblade3808 commented Mar 3, 2023

github-actions bot commented Mar 3, 2023

tmke8 commented Mar 3, 2023

AlexWaygood commented Mar 3, 2023 •

edited

Loading

tmke8 commented Mar 3, 2023 •

edited

Loading

AlexWaygood commented Mar 3, 2023 •

edited

Loading

tmke8 commented Mar 3, 2023

tmke8 commented Mar 4, 2023

AlexWaygood commented Mar 16, 2023

Make functools.lru_cache a descriptor class #9834

Make functools.lru_cache a descriptor class #9834

Conversation

Azureblade3808 commented Mar 3, 2023

github-actions bot commented Mar 3, 2023

tmke8 commented Mar 3, 2023

AlexWaygood commented Mar 3, 2023 • edited Loading

tmke8 commented Mar 3, 2023 • edited Loading

AlexWaygood commented Mar 3, 2023 • edited Loading

tmke8 commented Mar 3, 2023

tmke8 commented Mar 4, 2023

AlexWaygood commented Mar 16, 2023

Make `functools.lru_cache` a descriptor class #9834

Make `functools.lru_cache` a descriptor class #9834

AlexWaygood commented Mar 3, 2023 •

edited

Loading

tmke8 commented Mar 3, 2023 •

edited

Loading

AlexWaygood commented Mar 3, 2023 •

edited

Loading