Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-94912: Added marker for non-standard coroutine function detection #99247

Merged
merged 16 commits into from
Dec 18, 2022
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 25 additions & 2 deletions Doc/library/inspect.rst
Original file line number Diff line number Diff line change
Expand Up @@ -343,15 +343,38 @@ attributes (see :ref:`import-mod-attrs` for module attributes):

.. function:: iscoroutinefunction(object)

Return ``True`` if the object is a :term:`coroutine function`
(a function defined with an :keyword:`async def` syntax).
Return ``True`` if the object is a :term:`coroutine function` (a function
defined with an :keyword:`async def` syntax), a :func:`functools.partial`
wrapping a :term:`coroutine function`, an instance of a class defining an
:keyword:`async def` ``__call__``, or a sync function marked with
:func:`markcoroutinefunction`.

.. versionadded:: 3.5

.. versionchanged:: 3.8
Functions wrapped in :func:`functools.partial` now return ``True`` if the
wrapped function is a :term:`coroutine function`.

.. versionchanged:: 3.12
Instances of classes defining an :keyword:`async def` ``__call__``, or
sync functions marked with :func:`markcoroutinefunction` now return
``True``.


.. function:: markcoroutinefunction(func)

Decorator to mark a callable as a :term:`coroutine function` if it would not
otherwise be detected by :func:`iscoroutinefunction`.

This may be of use for sync functions that return a :term:`coroutine`, if
the function is passed to an API that requires :func:`iscoroutinefunction`.

When possible, using an :keyword:`async def` function is preferred. Also
acceptable is calling the function and testing the return with
:func:`iscoroutine`.

.. versionadded:: 3.12


.. function:: iscoroutine(object)

Expand Down
6 changes: 6 additions & 0 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,12 @@ asyncio
a custom event loop factory.
(Contributed by Kumar Aditya in :gh:`99388`.)

inspect
-------

* Add :func:`inspect.markcoroutinefunction` to mark sync functions that return
a :term:`coroutine` for use with :func:`iscoroutinefunction`.
(Contributed Carlton Gibson in :gh:`99247`.)

pathlib
-------
Expand Down
27 changes: 25 additions & 2 deletions Lib/inspect.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@
"ismodule",
"isroutine",
"istraceback",
"markcoroutinefunction",
"signature",
"stack",
"trace",
Expand Down Expand Up @@ -391,12 +392,34 @@ def isgeneratorfunction(obj):
See help(isfunction) for a list of attributes."""
return _has_code_flag(obj, CO_GENERATOR)

# A marker for markcoroutinefunction and iscoroutinefunction.
_is_coroutine = object()

def markcoroutinefunction(func):
"""
Decorator to ensure callable is recognised as a coroutine function.
"""
if hasattr(func, '__func__'):
func = func.__func__
func._is_coroutine = _is_coroutine
return func

def iscoroutinefunction(obj):
"""Return true if the object is a coroutine function.

Coroutine functions are defined with "async def" syntax.
Coroutine functions are normally defined with "async def" syntax, but may
be marked via markcoroutinefunction.
"""
return _has_code_flag(obj, CO_COROUTINE)
func = getattr(obj, "__func__", obj)
if getattr(func, "_is_coroutine", None) is _is_coroutine:
return True

if not isclass(obj) and callable(obj) and getattr(obj.__call__, "_is_coroutine", None) is _is_coroutine:
return True
Copy link
Contributor Author

@carltongibson carltongibson Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the check, we need to check plain functions and the __func__ cases, and the __call__ implementation for class instances. (Tests cover that: given that functions have a __call__ we need to be careful not to just check the one case.)

I looked at various ways of combining this into a single pass, but none that I found were particularly clear. Happy to take a suggestion, but I don't see this as being performance sensitive.

The implementation here is more sophisticated that the older asyncio one…

https://github.com/python/cpython/blob/3.11/Lib/asyncio/coroutines.py#L17-L24

… but (as per the added test cases) various extra (seemingly legitimate) cases are now covered. (So 👍 I guess)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking about this, but being distracted by other stuff. I'll try to get to it later this week.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @gvanrossum — thanks.

I thought one maybe neater re-write looks like this:

diff --git a/Lib/inspect.py b/Lib/inspect.py
index 883d0b02bf..3b9bd14a38 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -410,12 +410,13 @@ def iscoroutinefunction(obj):
     Coroutine functions are normally defined with "async def" syntax, but may
     be marked via markcoroutinefunction.
     """
-    func = getattr(obj, "__func__", obj)
-    if getattr(func, "_is_coroutine", None) is _is_coroutine:
-        return True
-
-    if not isclass(obj) and callable(obj) and getattr(obj.__call__, "_is_coroutine", None) is _is_coroutine:
-        return True
+    if not isclass(obj) and callable(obj):
+        # Test both the function and the __call__ implementation for the
+        # _is_coroutine marker.
+        f = getattr(getattr(obj, "__func__", obj), "_is_coroutine", None)
+        c = getattr(obj.__call__, "_is_coroutine", None)
+        if f is _is_coroutine or c is _is_coroutine:
+            return True
 
     return _has_code_flag(obj, CO_COROUTINE) or (
         not isclass(obj) and callable(obj) and _has_code_flag(obj.__call__, CO_COROUTINE)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So perhaps a confusing aspect is that everything that's callable has a __call__ method. Also the presence of __func__ indicates it's a class or instance method.

At least, those things seem to be obfuscating what's going on a bit.

Also, looking at the code for _has_code_flag(), I wonder if we don't need the same logic for checking the _is_coroutine flag.

What would happen if we wrote a helper function like this?

def _has_coroutine_mark(f):
    while ismethod(f):
        f = f.__func__
    f = functools._unwrap_partial(f)
    if not (isfunction(f) or _signature_is_functionlike(f)):
        return False
    return getattr(f, "_is_coroutine", None) is _is_coroutine

(FWIW I wonder if the variable _is_coroutine shouldn't be renamed _is_coroutine_marker.)

Now, I'm not entirely sure why you're testing for __call__, when none of the other is<something>function() predicates test for it. Maybe things would become simpler if we removed that feature?

We could then define our function like this:

def iscoroutinefunction(obj):
    return _has_code_flag(obj, CO_COROUTINE) or _has_coroutine_mark(obj)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, I'm not entirely sure why you're testing for call

OK, so the issue is we have classes like this:

        class Cl:
            async def __call__(self):
                pass

... where we need instances to be identified as async, and iscoroutinefunction doesn't (currently) pick it up.

Note that this is an actual, already-out-there, need. e.g. the Starlette framework has this exact check to work around this limitation. asgiref works around this limitation in its usage by marking the instance itself with the _is_coroutine marker.

Either way, I think it is desirable to return True for class instance of classes such as Cl with an async def __call__()

So that's the reason for adding the additional ... _has_code_flag(obj.__call__, ...) check.

In this PR, given that we were adding @staticmethod and @classmethod and so on, it seemed that we also should cover this kind of case:

        class Cl2:
            @inspect.markcoroutinefunction
            def __call__(self):
                pass

Which leads to needing similar in the _is_coroutine_marker block.

The latter case is somewhat hypothetical... — it comes from trying to cover all the cases here, rather than from real-use™. To that extent I'd be happy to drop it, but I guess if we do there's an issue in X months time, saying "I need to do this". 🤔

I don't know the correct thing to say. (?)

(FWIW I wonder if the variable _is_coroutine shouldn't be renamed _is_coroutine_marker.)

Yep OK. +1

What would happen if we wrote a helper function like this?

Let me have a read.

Thanks so much for your thought and input on this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, I still feel that you're sneaking in a new feature under the radar. :-) What this PR is supposed to fix is just to add the ability to mark a sync function as "morally async". IOW the behavior of

class C:
    @inspect.markcoroutinefunction
    def __call__(self): ...
print(inspect.iscoroutinefunction(C())

should be the same as

class C:
    async def __call__(self): ...
print(inspect.iscoroutinefunction(C())

and since the latter prints False, I don't think this PR has any business changing the output to True.

If you think the functionality used by Starlette deserves to be included in inspect, you should probably open a new issue for that first so it can be discussed properly. But I personally think this is overreaching (however, I am repeating myself).

Have you considered adding def _has_coroutine_mark(f): as I suggested?

(Sorry to be such a pain. But once we let this in there's no way back, so I prefer to have the minimal necessary functionality only. Other core devs may differ though.)


return _has_code_flag(obj, CO_COROUTINE) or (
gvanrossum marked this conversation as resolved.
Show resolved Hide resolved
not isclass(obj) and callable(obj) and _has_code_flag(obj.__call__, CO_COROUTINE)
)

def isasyncgenfunction(obj):
"""Return true if the object is an asynchronous generator function.
Expand Down
43 changes: 43 additions & 0 deletions Lib/test/test_inspect.py
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,49 @@ def test_iscoroutine(self):
gen_coroutine_function_example))))
self.assertTrue(inspect.isgenerator(gen_coro))

async def _fn3():
pass

@inspect.markcoroutinefunction
carltongibson marked this conversation as resolved.
Show resolved Hide resolved
def fn3():
return _fn3()

self.assertTrue(inspect.iscoroutinefunction(fn3))
self.assertTrue(
inspect.iscoroutinefunction(
inspect.markcoroutinefunction(lambda: _fn3())
)
)

class Cl:
async def __call__(self):
pass

self.assertFalse(inspect.iscoroutinefunction(Cl))
self.assertTrue(inspect.iscoroutinefunction(Cl()))

class Cl2:
@inspect.markcoroutinefunction
def __call__(self):
pass

self.assertFalse(inspect.iscoroutinefunction(Cl2))
self.assertTrue(inspect.iscoroutinefunction(Cl2()))

class Cl3:
@inspect.markcoroutinefunction
@classmethod
def do_something_classy(cls):
pass

@inspect.markcoroutinefunction
@staticmethod
def do_something_static():
pass

self.assertTrue(inspect.iscoroutinefunction(Cl3.do_something_classy))
self.assertTrue(inspect.iscoroutinefunction(Cl3.do_something_static))

self.assertFalse(
inspect.iscoroutinefunction(unittest.mock.Mock()))
self.assertTrue(
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Add :func:`inspect.markcoroutinefunction` decorator which manually marks
a function as a coroutine for the benefit of :func:`iscoroutinefunction`.