type hint: how should type of lru_cache be defined?

[david-gang]

Note: if you are reporting a wrong signature of a function or a class in
the standard library, then the typeshed tracker is better suited
for this report: https://github.com/python/typeshed/issues

Please provide more information to help us understand the issue:

• Are you reporting a bug, or opening a feature request? bug
• Please insert below the code you are checking with mypy,
  or a mock-up repro if the source is private. We would appreciate
  if you try to simplify your case to a minimal repro.

@lru_cache(maxsize=1)
def get_response_from_api() -> List[ApiObject]:
    url = _get_api_url()
    response = requests.get(url).text
    return json.loads(response, object_hook=_create_api_obj)

• What is the actual behavior/output?

error: Untyped decorator makes function "get_response_from_api" untyped

• What is the behavior/output you expect?
  it should not emit an error, alternatively there should be instructions what to do.

• What are the versions of mypy and Python you are using?
  mypy 0.600
  python 3.6.5
  Do you see the same issue after installing mypy from Git master? did not try

• What are the mypy flags you are using? (For example --strict-optional)
  mypy predictor --ignore-missing-imports --strict

• If mypy crashed with a traceback, please paste
  the full traceback below.

Also asked in SO: https://stackoverflow.com/questions/50495759/type-hint-how-should-type-of-lru-cache-be-defined

[emmatyping]

The underlying issue here is that the type variable for the lru_cache decorator becomes Any since there are no arguments to the function. I think it would be good to handle this case. For now, I'd recommend type ignoring it.

I believe variadic function parameters could help with this.

[gvanrossum]

Well, almost. The type variable only represents the return type; the decorator leaves the arguments unspecified, and the "Untyped decorator makes function "{}" untyped" error (generated by --disallow-untyped-decorators) considers that untyped. Since the function in this case has no arguments it is arguably wrong, but in general the lru_cache decorator does drop the types of the arguments (alas), so `# type: ignore` is still the best you can do ATM.

[ssbarnea]

I don't know if I was just unlucky but I encountered this bug few minutes after attempting to start using mypy for the first time on a project. Lucky me..., I had one function to check, and that one was using @lru_cache.

What is the workaround? (code still supports py27).

[gvanrossum]

Since lru_cache is mostly transparent, maybe this?

from typing import TYPE_CHECKING, TypeVar
if TYPE_CHECKING:
    F = TypeVar('F', Callable)
    def lru_cache(f: F) -> F: pass
else:
    from functools import lru_cache

[OJFord]

Or for the parameterised variant:

    def lru_cache(maxsize: int = 128, typed: bool = False) -> Callable[[F], F]:
        pass

(And note also that F should be bound=Callable.)

[asottile]

no idea how correct this is, but I took a stab at improving typeshed to "fix" this python/typeshed#3880 (didn't work for classes, nevermind)

[EldarSehayekZenity]

What about the ability to access the decorated function in order to access cache info, clear the cache, etc.? The type hints for lru_cache decorator should reflect those are available as well so mypy won't emit errors when accessing them. What do you think?

[JakeSummers]

Is there a workaround for this issue?

I tried using a type: ignore on the line with @lru_cache but that didn't work. Can I add a type shed for this to my code?

[chadrik]

The problem with the lru_cache stubs is that we can't get all of the features at the same time. We have to choose one or two of the following:

• A) preserving the type signature of the decorated function
• B) support for cache_clear methods, etc
• C) warning about non-hashable arguments.

The stubs in the typeshed chose B and C, which are arguably valuable, but it's a shame that we have to choose.

The suggestion from @gvanrossum chooses A, which is also valuable:

from typing import TYPE_CHECKING, TypeVar
if TYPE_CHECKING:
    F = TypeVar('F', Callable)
    def lru_cache(f: F) -> F: pass
else:
    from functools import lru_cache

It would be nice if ParamSpec had a basic mechanism to enforce types of arguments, like TypeVar(bound=...):

The end result might look like this:

_P = ParamSpec('P', bound=Hashable)  # <-- enforce all args are hashable
_T = TypeVar('T')

@final
class _lru_cache_wrapper(Generic[_P, _T]):
    def __call__(self, *args: _P.args, **kwargs: _P.kwargs) -> _T: ...
    def cache_info(self) -> _CacheInfo: ...
    def cache_clear(self) -> None: ...
    def __copy__(self) -> _lru_cache_wrapper[_P, _T]: ...
    def __deepcopy__(self, __memo: Any) -> _lru_cache_wrapper[_P, _T]: ...

def lru_cache(maxsize: int | None) -> Callable[Callable[_P, _T], Callable[_P, _T]]
    ...

[gvanrossum]

Shouldn't this be moved to the typeshed repo?

[chadrik]

Shouldn't this be moved to the typeshed repo?

We don't currently have the features we need to make the stubs any better. We have to choose the least bad solution, and I think the case could be made in either direction, so there's not a strong argument to break backward compatibility of the stubs.

We either need a new feature (maybe something for ParamSpec) or we need a mypy plugin for lru_cache/cache.

Does anyone have any good ideas about what a new feature to solve this problem would look like? Does my ParamSpec(bound=...) idea make sense? It's limited in usefulness but it seems like it would work.

[JotaRata]

In Python 3.11 this feature appears to be working correctly

[JelleZijlstra]

I don't think there's anything actionable for mypy here in any case.

[gandhis1]

I don't think there's anything actionable for mypy here in any case.

functools.partial was recently special-cased, is there any appetite to do the same for lru_cache, given it's comparably widespread usage?