Non-consistent type-hinting for signature of func with variable amount of dynamically added params

[vladislav-krastev]

**EDIT: for completeness, link to the discussion for this in the Python Forum, in case anyone ever suggests a solution there 71432

Hello everyone!

Using VSCode:
Python 3.12.7
Pylance v2024.11.1
pyright v1.1.386 (from how I understand it Pylance CHANGELOG)

There are different type-hinting results for logically (almost) the same ways of prepending a function signature with new parameters.

Example setup (simplified as much as I could):

from typing import cast, Callable

class MyBase:
    pass

def my_func(p_1 = None, *, p_2 = None):
    pass

class MyClass_1(MyBase):
    def __init__(self, *, f_1 = None, f_2 = None):
        self.f_1, self.f_2 = f_1, f_2

class MyClass_2(MyBase):
    def __init__(self, *, f_1 = None, f_2 = None):
        self.f_1, self.f_2 = f_1, f_2

The idea is to be able to call my_func with some statically predefined count and type of instances of subclasses of MyBase.

First option:

1. define a "decorator" that accepts only a single new_arg_type argument at a time:

def my_wrapper[**ParamsT, ReturnT, MetaT: MyBase](
    some_func: Callable[ParamsT, ReturnT],
    new_arg_type: type[MetaT],
):
    if not issubclass(new_arg_type, MyBase): 
        raise TypeError()
    def wrapped(new_arg: MetaT | None = None, /, *args: ParamsT.args, **kwargs: ParamsT.kwargs) -> ReturnT:
        f = some_func(*args, **kwargs)
        if new_arg is not None:
            if not isinstance(new_arg, new_arg_type):
                raise ValueError(f"expected: {new_arg_type} <-> received: {type(new_arg)}")
            # do something ...
        return cast(ReturnT, f)
    return wrapped

2. and call that decorator as many times as needed, when the decorated my_func would be used:

wrapped = my_wrapper(my_func, MyClass_1)
wrapped = my_wrapper(wrapped, MyClass_2)
wrapped(MyClass_2(), MyClass_1(), ... other args for my_func)

Result:

• PROS: type-hinting is what I would expect, correctly shows two positional args expected for wrapped, both with the correct default values of None and expected correct types, and both with the same name new_arg (for which we don't care as they are both positional-only)
• CONS: very non-dynamic (you have to explicitly call the decorator multiple times and that gets even clunkier if my_wrapper also returns other things) and the order of the new arguments is reversed from the logical order in which they were defined (which by itself is correct and expected, I know).

Second option:

1. define a "decorator" that accepts at least one and var-count more types for the new parameters and define them in a loop from inside the "decorator" itself:

def my_wrapper[**ParamsT, ReturnT, MetaT: MyBase](
    some_func: Callable[ParamsT, ReturnT],
    new_arg_type: type[MetaT],
    *new_arg_types: type[MetaT],
):
    def wrapper(
        some_func: Callable[ParamsT, ReturnT],
        new_arg_type: type[MetaT],
    ):
        def wrapped(new_arg: MetaT | None = None, /, *args: ParamsT.args, **kwargs: ParamsT.kwargs) -> ReturnT:
            f = some_func(*args, **kwargs)
            if new_arg is not None:
                if not isinstance(new_arg, new_arg_type):
                    raise ValueError(f"expected: {new_arg_type} <-> received: {type(new_arg)}")
                # do something ...
            return cast(ReturnT, f)
        return wrapped
    res = wrapper(
         some_func,
         new_arg_types[-1] if new_arg_types else new_arg_type,
    )
    for arg_type in reversed([new_arg_type, *new_arg_types[:len(new_arg_types)-2]]):
        res = wrapper(res, arg_type)
    return res

2. call the "decorator" once with as many types for additional params as needed:

wrapped = my_wrapper(my_func, MyClass_1, MyClass_2)
wrapped(MyClass_1(), MyClass_2(), ... other args for my_func)

Result:

• PROS: usage looks way cleaner, more understandable and natural then multiple calls to the same function in a row, also the order in which the additional types were defined is preserved
• CONS: type-hinting gets totally broken (both for the new params and the original func's params)

Other tested options:

• recursion in my_wrapper instead of a for-loop, which you'd think is the most correct way, but no
• new local-bounded TypeVars for wrapper both when using loop or recursion in my_wrapper -> produces the worst results

All of the options produce working code with the only difference on runtime being the expected order of the new args.

So, is there a way to have the dynamic way of doing it, but still preserving the correct and useful type-hinting from the First option?

I guess the issue is that every time the decorator is called, it appends a new param with a duplicate name of an existing param? Tried figuring out how to do that insane thing as well (dynamic name of a function parameter) but couldn't figure out how to make it work, not to mention type it.....

Any ideas, comments, suggestions how this or something similar can be achieved?
I don't have issues with using any hack inside the my_wrapper function, or # typeignores etc.

BTW, just spamming n-count of decorator calls over the my_func definition is not an option in this case, as it's from a third party lib AND it's supposed to be any function from a family of un-related funcs (that still produce instance of the same class).

Thank you in advance!!!

[erictraut]

If you have general questions about the Python typing spec or how to use type annotations for specific use cases, the Python typing forum is a good place to post. That forum is frequented by experts in the Python typing community, and they can help you with such questions.

If you think that pyright is behaving incorrectly (contrary to the typing spec), I'd appreciate it if you could post a minimal, self-contained sample along with the observed and expected behavior.

[vladislav-krastev]

Thank you for answering!

I literally cannot think of a more minified example then the following - both executable on just a copy-paste and when hovering over example_1 and example_2 in VSCode reproduces the issue.

from typing import cast, Callable

class MyBase:
    pass

class MyClass_1(MyBase):
    def __init__(self, *, f_1 = None, f_2 = None):
        self.f_1, self.f_2 = f_1, f_2

class MyClass_2(MyBase):
    def __init__(self, *, f_1 = None, f_2 = None):
        self.f_1, self.f_2 = f_1, f_2

def my_func(p_1: int | None = None, *, p_2: str | None = None) -> str:
    print("OK")
    return "OK"

def my_wrapper[**ParamsT, ReturnT, MetaT: MyBase](
    some_func: Callable[ParamsT, ReturnT],
    new_arg_type: type[MetaT],
    *new_arg_types: type[MetaT],
):
    def wrapper[**InternalParamsT](
        some_func: Callable[InternalParamsT, ReturnT],
        new_arg_type: type[MetaT],
    ):
        def wrapped(new_arg: MetaT | None = None, /, *args: InternalParamsT.args, **kwargs: InternalParamsT.kwargs) -> ReturnT:
            f = some_func(*args, **kwargs)
            if new_arg is not None:
                if not isinstance(new_arg, new_arg_type):
                    raise ValueError(f"expected: {new_arg_type} <-> received: {type(new_arg)}")
                # do something ...
            return cast(ReturnT, f)
        return wrapped
    return wrapper(
        some_func,
        new_arg_types[-1] if new_arg_types else new_arg_type,
    )

example_1 = my_wrapper(my_func, MyClass_1)
example_2 = my_wrapper(my_func, MyClass_1, MyClass_2)

Calling my_wrapper with zero var-args produces the correct type hint for example_1:

(variable) def example_1(
    new_arg: MyClass_1 | None = None,
    /,
    p_1: int | None = None,
    *,
    p_2: str | None = None
) -> str

Calling my_wrapper with non-zero var-args produces the unexpected (for me) type hint for example_2:

(variable) def example_2(
    new_arg: MyClass_1 | MyClass_2 | None = None,
    /,
    p_1: int | None = None,
    *,
    p_2: str | None = None
) -> str

Here, I would expect the example_2 type-hint to have the type-hint for its new_arg param as new_arg: MyClass_2 | None = None, and not new_arg: MyClass_1 | MyClass_2 | None = None,.
When zero var-args, the correct type is inferred, when non-zero, the correct type is not inferred, despite the 'calculation' for what that type is being the same piece of code, namely the if-else part in the return statement:

    return wrapper(
        some_func,
        new_arg_types[-1] if new_arg_types else new_arg_type,
    )

[erictraut]

Pyright is working correctly here.

Here's a minified version of your code. I've stripped out all of the extraneous and irrelevant details to make it clearer what is going on.

from typing import Concatenate, Callable

class Class1: ...
class Class2: ...

def func(p1: int) -> str: ...

def wrapper[**P, R, B](a: Callable[P, R], b: type[B], *args: type[B]) -> Callable[Concatenate[B, P], R]: ...

x = wrapper(func, Class1, Class2)

The constraint solver is being asked to find a solution for type variable B based on the constraints that the resulting type must be a subtype of both Class1 and Class2. Pyright's constraint finds the solution Class1 | Class2. The solutions for type variables P and R are straightforward. The resulting return type is therefore evaluated as: (Class1 | Class2, p1: int) -> str.

[vladislav-krastev]

It seems I'm (totally?) misunderstanding how and based on what the type-hints and type-checks are (auto) generated.
So it's not actually checking what's inside the function, only its signature? From how I understand you, the body of the function is quote "irrelevant" for what the resulting type-hinting is?
The code in the function's body

return internal_wrapper(
        f,
        args[-1] if args else b,
    )

is not actually considered at all, only the function's signature is: def wrapper[**P, R, B](a: Callable[P, R], b: type[B], *args: type[B]) -> Callable[Concatenate[B, P], R]: ... ?

**EDIT: and this resulting signature is generated only once, when the function is defined, not on places where its used -> hence, how it's used in a specific situation would not actually produce a concrete signature for this usage?
Am I understanding you correctly on why this is the correct and expected behavior?

[erictraut]

When a type checker evaluates the type of a call expression, it uses the signature of the target call. It does not use the implementation of the target call in this circumstance. Often (e.g. in the case of type stubs), the implementation is not even provided to a type checker.

If you omit the return type annotation for a function, pyright will attempt to infer its return type based on the implementation, but that inferred return type is then used to define the signature.

If the implementation of a function is provided, a type checker will check the code within that function. However, the body of a function isn't used to evaluate calls to that function. That's why I stripped out the implementation in your example. Those details are irrelevant if the intent of the sample is to focus on the evaluation of the function call.

You are correct that a function's signature is defined only once. However, the signature can be polymorphic (i.e. use generic type variables), which allows different callers to supply different argument types.

You may find that this chapter of the Python typing spec will help you better understand how the type system works with regard to call signatures and callables.

[vladislav-krastev]

Thank you for the explanation, much appreciated!