make wrapt a PEP561 typed package.

[bitranox]

not sure if the name have to be "wrapt.pyi" or "py.typed" - that needs to be tested.

[bitranox]

correction : not sure if the name have to be "wrapt.pyi" or "wrappers.pyi" - that needs to be tested.

[GrahamDumpleton]

Is the .idea directory an editor directory and should it be removed and then name added to .gitignore?

[bitranox]

oh, sorry, yes, the .idea needs to be excluded

[dimaqq]

Kind poke!

[GrahamDumpleton]

The problem is that I don't know enough about adding python type hinting to a package to know whether this patch even works. From my testing it is incomplete for a number of reasons.

The first problem is that the py.typed and *.pyi files don't get installed with the package. This is solved by adding to setup.cfg the lines:

[options.package_data]
wrapt = py.typed, *.pyi

The second problem is that the type hints were included in a file called wrapt.pyi, but that is ignored. The pyi files should have a basename which matches the particular code file they contain hints for.

Thus, should at least perhaps be named __init__.pyi instead and the type hint for wrapt.decorator is then at least recognised by pylance under VS Code when type checking is set to basic. I would have thought they should have been in decorators.pyi and wrappers.pyi respectively, although I couldn't get that working, but that may actually be because the package can optionally use a C extension, so maybe it should be _wrappers.pyi for some things. I am not really sure whether this is an issue with the type hints or how I have the editor set up, since I have never used type checking for Python in the editor before.

Anyway, for the type hint for wrapt.decorator, addressing the above issue doesn't actually seem to help though as the editor still complains about the decorated function.

There is also another problem which is that as soon as you add a .pyi file, it is taken as the authoritative source for interface information, thus the contents on the py file are ignored. This means all the other functions and types exported by the wrapt module are not recognised at all.

So it seems you can't do this half way and as soon as you start using pyi files then they need to be complete and contain interface descriptions for everything in the module they sit parallel to. So having type hints for just wrapt.decorator and wrapt.ObjectProxy is not enough and they are needed for everything else in wrapt as well, which is going to be much more work.

What I would perhaps suggest for anyone who better understands this and wants to see it done, is to create a separate Python package in a GitHub repository (not released to PyPi), which is a -stubs package per PEP 561 and try and construct a complete type hints package for wrapt which captures everything required. For those who want to test it while is worked on, it can be installed from the GitHub repository.

The only other option I can see is to wait until Python 2 support is dropped which would allow type hints to be included in the Python code itself, although separate pyi files are likely still going to be needed to define prototypes for anything implemented in the C extension. How that will work though I don't know when wrapt can depending on how it is installed use a pure Python variant of code, or the C extension variant.

[dimaqq]

I think you're correct.
If wrapt supports both py2 and py3, then separate pyi files are a must.

Typically, there's 1-1 for .py and .pyi file (technically one for each module), like in
https://github.com/python/typeshed/tree/main/stubs/redis/redis

At the same time, I think (but I can't find the reference) that stub package can be shipped in a form of a single file, and that would be wrapt/wrapt.pyi.

[bitranox]

• python2 sunset date has passed January 1, 2020, so I guess its time to clean up and get rid of the python2 code.
  The current available wrapt version will be still available on pypy for python2 users, but You can enjoy the python3 possibilities.

• I use the type hints to be included in the Python code itself in my code, and that is because my modules are not that matured, so it is a pain to keep *.py and *.pyi files in sync, but the readability of the code suffers a bit when put that type information in the *py files.

For a very matured project like wrapt I think its better to work with *.pyi files, because signatures dont change that frequently.

There is also another problem which is that as soon as you add a .pyi file, it is taken as the authoritative source for interface information, thus the contents on the py file are ignored. This means all the other functions and types exported by the wrapt module are not recognised at all.

I never worked with pylance , I use pycharm - I am almost sure with pycharm that is not the case, but i did not try.

So it seems you can't do this half way and as soon as you start using pyi files then they need to be complete and contain interface descriptions for everything in the module they sit parallel to.

I would guess that 99% of the users only use the wrapt decorator and nothing else. So, to type check that, the signature of the wrapt decorator would be enough, and You can add signatures gradually as needed.
as @dimaqq pointed out, stub files can be marked as partial, so most people will be happy with type hints for the wrap decorator only :)

Besides the "offline" type checkers, there is also a wonderful runtime type checker :

https://github.com/beartype/beartype

maybe You can team up with @leycec - he is a "Runtime type-checking aficionado. "

[dimaqq]

I would guess that 99% of the users only use the wrapt decorator and nothing else.

stubs can be marked as partial, this makes sense.
I'd be happy with type hints for the wrap decorator only :)

[leycec]

maybe You can team up with @leycec - he is a "Runtime type-checking aficionado. "

💪 🐻

@leycec has been summoned to the chat. The chat proceeds to descend into chaos.

Actually, I humbly agree with everything @bitranox has wisely spouted above – except this:

I use the type hints to be included in the Python code itself in my code, and that is because my modules are not that matured, so it is a pain to keep *.py and *.pyi files in sync...

Yes. I strongly agree with this. In 2023, I see vanishingly few .pyi files in open-source projects. Indeed, I'll strengthen this claim; the only open-source projects with .pyi files in 2023 are projects with C extensions (e.g., NumPy), which cannot be annotated with type hints in pure C and thus require external .pyi files for annotations.

Pure-Python .py files, however, are always annotated with type hints directly in those files themselves. Why? Because, as @bitranox observed:

• Desynchronization. It's all too easy for .py and .pyi files to diverge and all too hard to detect that divergence, which leads directly to...
• Testing. It's unclear whether pytest-based unit or integration tests even can programmatically detect a desynchronization between .py and .pyi files. You'll never even know when you've done wrong.
• Runtime type-checking. .pyi files are mostly incompatible with runtime type-checkers like @beartype, Pydantic, and typeguard; .py files are perfectly compatible. Indeed, @beartype is on the cusp of releasing import hooks enabling end users to type-check their entire app stack (including third-party dependencies like wrapt) at runtime in only two lines of code. But this only applies to type hints defined in .py files; type hints defined in .pyi files are basically opaque and inaccessible to runtime type-checkers. This is why my emoji sobs unconsolably. 😭

...but the readability of the code suffers a bit when put that type information in the *py files.

Actually, it's probably the opposite; well-annotated .py files tend to be more readable than unannotated .py files if anything, as the type hints annotating those files explicitly inform readers of the types accepted and returned by classes and callables in those files.

That said, your mileage may vary (YMMV). This has been a public service announcement from the @beartype Broadcasting Service. Please do let me know if there is anything @beartype can explicitly do for wrapt here. So much love for all the awesome good work you do for the Python community, wrapt devs! 🏩

[bitranox]

@leycec , always funny and precise comments, love it.
@GrahamDumpleton , the minimum You can do is to introduce a partial pyi file, and decorate at least the wrapt decorator - 99% of users (including me) will be happy with that.
As soon as You get rid of the python2 compatibility, You can start to annotate in the *.py files, enable us coding slaves to enjoy runtime type checkers like beartype

[dimaqq]

I think that pyi files should also follow pep-8, which means 2 blank lines between functions and classes declared at the module scope.

Maybe just run ruff or black to auto-format this file?

[dimaqq]

Due to microsoft/pyright#5929 the re-export in __init__.py needs to be updated too.

For example, from .decorators import decorator as decorator

The issue affects pyright but not mypy.

You can test this by e.g.

poetry add git+https://github.com/GrahamDumpleton/wrapt.git#a2dddf6e10536f4cd4bfc96ff1a706cd71b0694c
poetry add pyright
poetry add mypy

pyright some_test_file.py
mypy some_test_file.py

[dimaqq]

An alternative to from ... import A as A would be to provide explicit __all__ in wrapt/__init__.py.

[GrahamDumpleton]

Latest version of wrapt requires Python 3.6+ and in #261 someone recently suggested requiring Python 3.8+.

So maybe someone wants to have a go at a separate PR now which embeds annotations in the code. Would appreciate it starting with core functionality first so I can follow along, learn, and verify myself since am not knowledgeable about using type annotations.

[dimaqq]

This did not work for me, so I managed to type the wrapt.decorator in a different way, I think it's Py 3.7+

from typing import Any, Callable, Optional, TypeVar, Tuple, Dict, Protocol

R = TypeVar('R', covariant=True)  # Return type of the function
T = TypeVar('T')  # Type of instance (for methods)

class FuncProtocol(Protocol[R]):
    def __call__(self, *args: Any, **kwargs: Any) -> R: ...

def decorator(
    wrapper: Callable[
        [
            FuncProtocol[R],
            Optional[T],
            Tuple[Any, ...],
            Dict[str, Any]],
        R],
    enabled: Optional[bool] = None
) -> FuncProtocol[R]: ...

https://gist.github.com/dimaqq/e4b418e1b9ce6a3cd874ba9540fe42c6

[grayjk]

@bitranox give the current develop branch a look. partial typing stubs have been added: https://github.com/GrahamDumpleton/wrapt/blob/develop/src/wrapt/__init__.pyi

[GrahamDumpleton]

Note that I decided to only provide type hints for Python 3.10+. Python 3.9 will go obsolete September or October this year. Making that decision means could use full syntax for type hints.

I haven't tackled decorator yet and am going to do that last as it has some extra complexity due to adapter factory argument, thus started with FunctionWrapper and function_wrapper to start with.

Couple of things have found are they unless you tell mypy to use --strict it will not warn about some things. Further, even with strict, when using function_wrapper as a decorator on class methods (normal, class method and static method), mypy isn't able to understand that binding is occurring and so is not able to detect when calling these with wrong arguments with respect to what was decorated.

I only realised the latter when doing synchronized decorator as had same issue with it as for that I was adding tests for that case, which forgot to do with function_wrapper. For synchronized I was able to solve it by simplifying things and ignoring to some degree that function wrappers were used under the covers.

Anyway, am making progress. Have not really added type hints to code before for a package used by others, so has been a bit of a learning experience.

[GrahamDumpleton]

It isn't actually binding which is a problem but a more fundamental problem that mypy seems to have being able to handle decorators which are implemented as a class rather than a function. It just isn't able to infer arguments and return type properly in that case. Only solution have found so far is that if a user wants users of their decorator to have type checking, they need to use a function wrapper around it. Eg.,

P = ParamSpec("P")
R = TypeVar("R", covariant=True)


def wrapper(
    callable: Callable[Concatenate[Any, P], R],
) -> Callable[Concatenate[Any, P], R]:
    @function_wrapper
    def _wrapper(
        wrapped: Callable[..., Any],
        instance: Any,
        args: tuple[Any, ...],
        kwargs: dict[str, Any],
    ) -> Any:
        return wrapped(*args, **kwargs)

    return _wrapper(callable)


@wrapper
def function(x: int, y: str = "default") -> str:
    """A simple function to be wrapped."""
    return f"{x}: {y}"
    
 function() # mypy will flag this as invalid

So forces extra work on to the user of wrapt. Can't see a way around it yet. If don't do this, mypy will not pick up that call to function() is missing an argument.

I have used @function_wrapper decorator from wrapt here, but will be same issue with @decorator as well even if add type hints for it.

[GrahamDumpleton]

Possibly relevant discussion, just link here so don't forget.

• Making a type alias of a Callable that describes a decorator discards all type information python/mypy#18842

[bitranox]

maybe beartype can help to identify issues : https://github.com/beartype/beartype
You might contact @leycec to get it done in a clean way - he is a type-checking aficionado - and a very funny and nice guy.
all the best
bitranox

[dimaqq]

I wonder if smth like this would work:

class _deco: …
def _deco(): …

class Deco(Protocol):
    def __call__(self, …) -> …: pass

wrapt: Deco = _deco

[leycec]

@bitranox: Thanks so much for the ping. You're so noice. 🤗

Callable[...] type hints are, indeed, severely deficient. They never quite got fully specced out – which is a shame. They could have been super-useful. Weirdo stuff that nobody really understands like typing.ParamSpec and typing.Concatenate[...] can help in various edge cases, but never much. It's always an exercise in frustration.

Interestingly, mypy supports a kinda hacky non-standard alternative to Callable[...] type hints that it calls "Callback Protocols". The core idea is that, rather than use a Callable[...] type hint, you define a typing.Protocol subclass whose __call__() method is annotated in the way that you wanted your Callable[...] type hint to be annotated: e.g.,

from typing import Protocol

class Combiner(Protocol):
    def __call__(self, *vals: bytes, maxlen: int | None = None) -> list[bytes]: ...

def batch_proc(data: Iterable[bytes], cb_results: Combiner) -> bytes:
    for item in data:
        ...

Crucially, note that Combiner is then used as a type hint. The cb_results parameter accepted by the batch_proc() function will then match any callable with a signature resembling that of the Combiner.__call__() method.

Kinda weird, but occasionally useful. No idea whether callback protocols apply to this exact use case, though. It might be best to just "move on" and accept that strictly typing this edge case is non-trivial or even infeasible.

Sadly, PEP 677 was rejected. Here's hoping that CPython devs accept a full-blown Callable[...] PEP standard at some point. Until then, we all sigh.

Anyway, am making progress. Have not really added type hints to code before for a package used by others...

You're doing a great job, @GrahamDumpleton! Some type hints are better than no hints. What you have now is almost certainly better than what you had before (i.e., nothing). Personally, I'd just call this QA Mona Lisa a done painting at some point.

tl;dr: Ship this useful madness. 😂

[GrahamDumpleton]

I think I have pushed up now to develop branch type hints for all public APIs. Can be used by installing https://github.com/GrahamDumpleton/wrapt/archive/refs/heads/develop.zip.

As mentioned there are some limitations and also some corner cases where will give wrong errors/warnings by type checkers, but can't see they are fixable. I still need to finish up tests, and am going to create some new docs about limitations and workarounds for adding type hints to your own decorators so when used it will correctly complain. The docs might stimulate suggestions on how to improve type hints to make it work properly without workarounds.

I am going to close this issue, but since no where else, can keep discussion going here for now if find anything.

[GrahamDumpleton]

Oh, I could be a goose and me as dumb user wasn't setting up properly type hints for my decorator when using a wrapt decorator. So what I thought might be a limitation may not be after all. Still exploring.

[GrahamDumpleton]

So now know how to make type signatures propagate properly so long as decorator is designed to only work on normal functions. If want type signature to work you can't use it on an instance or class method. Applying decorators to functions which have default arguments is also a problem as knowledge of default arguments is lost. 😩

[leycec]

It's a wicked problem, to be sure. @beartype resolves this to everyone's satisfies through a combination of PEP 484 typing.TypeVar + @typing.overload. Of course, I suspect you've done already done something similar, @bitranox. If not, consider the horror that follows:

from collections.abc import Callable
from typing import Any, TypeVar, Union, overload

DecoratableT = TypeVar(
    'DecoratableT',
    # Your decorator decorates objects that are either...
    bound=Union[
        # Arbitrary class *OR*...
        type,

        # Arbitrary callable *OR*...
        Callable[..., Any],

        # Other exotic kinds of objects if you support them. For example,
        # @beartype also supports oddball stuff like C-based descriptors:
        classmethod,
        property,
        staticmethod,
    ],
)


# These overloads satisfy static type-checkers like mypy and pyright:
@overload  # type: ignore[misc,no-overload-impl]
def your_decorator(obj: DecoratableT) -> DecoratableT: ...
@overload
def your_decorator(define_any_optional_parameters_accepted_by_your_decorator_here: Any = None) -> (
    Callable[[DecoratableT], DecoratableT]): ...

# This is your actual decorator implementation, which also needs to be well-typed:
def beartype(
    # Optional positional or keyword parameters.
    obj: Optional[DecoratableT] = None,

    # Optional keyword parameters.
    define_any_optional_parameters_accepted_by_your_decorator_here: Any = None,
) -> Union[DecoratableT, Callable[[DecoratableT], DecoratableT]]:
    # Implement your decorator here, yo!

That's sufficed for @beartype users for literally years. No complaints from @beartype's userbase that sprawls over like a billion packages now. Admittedly, it's pretty ugly and nonsensical syntax. What you gonna do, though? 😮‍💨

[GrahamDumpleton]

I added docs, so you can see details of current limitations in:

• https://github.com/GrahamDumpleton/wrapt/blob/develop/docs/typing.rst

The type hints I ended up adding can be seen in:

• https://github.com/GrahamDumpleton/wrapt/blob/develop/src/wrapt/__init__.pyi

The beartype package ones look rather tame to me. 🤣

[leycec]

WOW. That's... insane. I mean, that's good insane. To be sure, it's not the bad kind of insane. But "wow." I am stunned. That's probably the most intense use of callable-oriented type hints I've ever seen. You have taken type hints to the limit and blown way past that limit into a new event horizon of unseen horror.

If that's still not good enough, then surely nothing more can be done. I bow before your QA might. 🙇

[GrahamDumpleton]

Are you able to try wrapt==2.0.0rc1 from PyPi and let me know whether works okay or have issues. Thanks.

[GrahamDumpleton]

Will be a 2.0.0rc2 now. Someone suggested something to me which wouldn't work as was suggested with the way things worked in the way I did the type hints in wrapt, but it caused me to go back and try again an approach that had failed at before. In doing that I found a quirk of how type hints was working that was causing a prior attempt to fail. Thus will be able to change things and solve the problems of optional and keyword params not working properly, but in doing so will mean the way type hint is written for wrapped argument of wrapper when declaring you own decorators has to change.