Decorated function leaks type variables

[NeilGirdhar]

from typing import Callable, TypeVar

R = TypeVar('R')
Carry = TypeVar('Carry')
X = TypeVar('X')
Y = TypeVar('Y')

def api_boundary(f: Callable[..., R]) -> Callable[..., R]:
    return f

@api_boundary  # Removing this decorator fixes things.
def scan(f: Callable[[Carry, X], tuple[Carry, Y]],
         init: Carry,
         xs: X) -> tuple[Carry, Y]:
    ...

def f(c: int, x: bool) -> tuple[int, float]:
    return (c, 1.0)

reveal_type(scan(f, 1, True))   # Tuple[Carry`-1, Y`-3]

This is on Mypy 0.921.

Or maybe I've misunderstood type inference, and just have to wait for PEP 612 to finish being implemented?

[erictraut]

The api_boundary decorator returns a Callable[..., R] which loses the input parameters of the decorated function.

I think what you want is this:

C = TypeVar("C", bound=Callable[..., Any])

def api_boundary(f: C) -> C:
    return f

[NeilGirdhar]

@erictraut Okay, thanks! Out of curiosity, would I have been able to do it my way after PEP 612 is completely implemented?

[erictraut]

Yes, you could have used a ParamSpec:

P = ParamSpec("P")

def api_boundary(f: Callable[P, R]) -> Callable[P, R]:
    return f

This works today in pyright. I think I read that the next release of mypy will contain support for this case.

[NeilGirdhar]

I'm going to reopen this until mypy supports it. Here's another example:

from functools import partial
from typing import Callable, Generic, TypeVar

from typing_extensions import ParamSpec

R = TypeVar('R')
X = TypeVar('X')
P = ParamSpec('P')

class custom_vjp(Generic[P, R]):
    def __init__(self, fun: Callable[P, R], x: int = 0):
        self.fun = fun

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
        return self.fun(*args, **kwargs)

@custom_vjp
def replace_cotangent(x: X, new_cotangent: X) -> X:
    return x

reveal_type(replace_cotangent(1, 2))

a.py:21: note: Revealed type is "X`-1"
a.py:21: error: Argument 1 to "__call__" of "custom_vjp" has incompatible type "int"; expected "X"  [arg-type]
a.py:21: error: Argument 2 to "__call__" of "custom_vjp" has incompatible type "int"; expected "X"

[bbatliner]

More examples discovered today from my use of custom context manager decorators: https://mypy-play.net/?mypy=0.930&python=3.10&gist=06b8b06ca6bf78674c19c123b98c47ce

Interestingly, importing the contextmanager decorator directly from contextlib seems to avoid this issue, so I'm not sure what's special about their code.

The issue seems to be that using ParamSpec with generic decorators/callables seem to lose the ability to determine generic types through calls to the decorated function.

I debated making this example a separate bug ticket but this issue seems to capture it.

[bbatliner]

My example above started working on mypy 0.920 with initial support for ParamSpec and broke in 0.930, and is still broken on master, if that helps narrow down a diff.

[cdce8p]

I believe this is now fixed with the last ParamSpec PR.

• Basic ParamSpec Concatenate and literal support #11847

[AlexWaygood]

I believe this is now fixed with the last ParamSpec PR.

• Basic ParamSpec Concatenate and literal support #11847

Yes, I can't immediately reproduce any of the ParamSpec examples using the master branch. Hooray!

[bbatliner]

Hey Alex, how did you test using the master branch? mypy-play.net still produces the errors in this thread with the master branch option but I don't know if it's using the most up-to-date "master branch." Do you build mypy locally and test with that copy?

[AlexWaygood]

Hey Alex, how did you test using the master branch? mypy-play.net still produces the errors in this thread with the master branch option but I don't know if it's using the most up-to-date "master branch." Do you build mypy locally and test with that copy?

Yeah, the "mypy master" option on mypy playground isn't a reliable tool really — it often isn't quite up to date 😕

There's setup instructions here for how to get mypy running locally: https://github.com/python/mypy/blob/master/CONTRIBUTING.md. Feel free to ping me if you run into any difficulties! After following those instructions, just run python -m mypy test.py from inside your local clone.