[mypyc] Wrap async functions with function-like type #20260

p-sawicki · 2025-11-19T12:01:12Z

Currently, introspection functions from the inspect module like iscoroutinefunction(fn) don't return the expected result for functions compiled with mypyc. mypyc defines the functions using PyMethodDef in C for which cpython makes objects with type PyCFunction_Type and not PyFunction_Type. The latter corresponds to types.FunctionType in python. So the isfunction(fn) check fails.

iscoroutinefunction(fn) and some others accept function-like objects as long as they are callable and have some required attributes to support compiled functions.

The most important attribute is __code__ which is a code object that stores flags for the function. For example for iscoroutinefunction(fn), the CO_COROUTINE flag is checked.

In this PR, mypyc adds a wrapper CPyFunction whose __call__ calls the same function that would be defined with PyMethodDef. The wrapper is added to __dict__ of either the module (for functions) or the containing class (for methods) with the original name of the function so the wrapper is returned in place of the PyCFunction_Type objects.

For nested functions, mypyc already creates a callable class so the function-like attributes are added as properties. A static CPyFunction object is created separately for each class that is used in the getter and setter methods of the callable class.

For now CPyFunctions are created only for async functions to support iscoroutinefunction(fn) but the implementation could be extended to support other introspection functions in future PRs. For non-async functions the wrapper does not seem necessary for now because the default behavior makes the function always return false. So the behavior is as-expected for non-async functions.

This implementation was inspired by Cython which generates much richer function wrappers.

JukkaL

Some return values aren't checked (not a full review).

JukkaL · 2025-11-20T12:11:54Z

mypyc/lib-rt/CPyFunction.c

+    PyMethodDef *method = CPyMethodDef_New(funcname, func, func_flags, func_doc);
+    PyObject *code = CPyCode_New(filename, funcname, first_line, code_flags);
+    PyObject *op = (PyObject *)CPyFunction_Init(PyObject_GC_New(CPyFunction, CPyFunctionType),
+                                                method, PyUnicode_FromString(funcname), module, code);


Check return values of all function calls that may fail due to out of memory.

JukkaL · 2025-11-20T12:12:20Z

mypyc/lib-rt/function_wrapper.c

+}
+
+static PyMethodDef* CPyMethodDef_New(const char *name, PyCFunction func, int flags, const char *doc) {
+    PyMethodDef *method = (PyMethodDef *)PyMem_Malloc(sizeof(PyMethodDef));


Check return value.

JukkaL · 2025-11-20T12:12:27Z

mypyc/lib-rt/function_wrapper.c

+}
+
+static PyObject* CPyCode_New(const char *filename, const char *funcname, int first_line, int flags) {
+    PyCodeObject *code_obj = PyCode_NewEmpty(filename, funcname, first_line);


Check return value.

JukkaL · 2025-11-20T12:14:55Z

mypyc/codegen/emitmodule.py

+            wrapper_name = emitter.emit_cpyfunction_instance(fn, filepath)
+            name_obj = f'PyUnicode_FromString("{fn.name}")'
+            emitter.emit_line(
+                f"if(PyDict_SetItem({globals}, {name_obj}, (PyObject *){wrapper_name}) < 0)"


Style nit: add space after if, before (.

JukkaL · 2025-11-20T12:15:01Z

mypyc/codegen/emitmodule.py

+
+            filepath = self.source_paths[module.fullname]
+            wrapper_name = emitter.emit_cpyfunction_instance(fn, filepath)
+            name_obj = f'PyUnicode_FromString("{fn.name}")'


Check return value.

JukkaL · 2025-11-20T12:15:25Z

mypyc/codegen/emitclass.py

+        filepath = emitter.filepath or ""
+        wrapper_name = emitter.emit_cpyfunction_instance(fn, filepath)
+        emitter.emit_line(
+            f'if (PyDict_SetItem(tp->tp_dict, PyUnicode_FromString("{fn.name}"), {wrapper_name}) < 0)'


Check return value of PyUnicode_FromString.

JukkaL · 2025-11-20T12:16:17Z

mypyc/codegen/emit.py

+
+        code_flags = "CO_COROUTINE"
+        self.emit_line(
+            f'PyObject* {wrapper_name} = CPyFunction_New({module}, "{filepath}", "{name}", {cfunc}, {func_flags}, {doc}, {fn.line}, {code_flags});'


Check the return value of CPyFunction_New.

JukkaL

Another round of review -- still not a full review.

JukkaL · 2025-11-24T15:53:11Z

mypyc/lib-rt/function_wrapper.c

@@ -0,0 +1,235 @@
+#define PY_SSIZE_T_CLEAN


Style nit: the file name is inconsistent. Use snake_case.c, such as function_wrapper.c.

JukkaL · 2025-11-24T15:55:26Z

mypyc/lib-rt/CPyFunction.c

+                          int first_line, int code_flags) {
+    if (!CPyFunctionType) {
+        CPyFunctionType = (PyTypeObject *)PyType_FromSpec(&CPyFunction_spec);
+        assert(CPyFunctionType);


If this fails, return NULL instead of failing an assertion. Another option, since this will likely only fail if running out of memory, is to call CPyError_OutOfMemory on NULL return which aborts.

JukkaL · 2025-11-24T15:55:42Z

mypyc/lib-rt/function_wrapper.c

+}
+
+static PyObject* CPyFunction_repr(CPyFunction *op) {
+    return PyUnicode_FromFormat("<function %U at %p>", op->func_name, (void *)op);


Test this in a run test.

JukkaL · 2025-11-24T15:56:19Z

mypyc/lib-rt/function_wrapper.c

+static void CPyFunction_dealloc(CPyFunction *m) {
+    PyObject_GC_UnTrack(m);
+    if (CPyFunction_weakreflist(m) != NULL)
+        PyObject_ClearWeakRefs((PyObject *) m);


Maybe test a weak reference in a function object in a run test?

JukkaL · 2025-11-24T15:57:58Z

mypyc/test-data/run-async.test

+    assert is_coroutine(identity_async)
+
+    assert not is_coroutine(wrapped)
+    assert is_coroutine(wrapped_async)


Test also calling the functions that use the new wrapper, just in case. I'm not sure if all of these use cases have a similar existing test case.

JukkaL

This concludes my first review pass. I'll do another lighter weight pass after the current round of comments has been addressed.

This is an important feature for mypyc, and I think a fairly complex implementation is the only way to address compatibility issues. However, the complexity makes this hard to test and review, but this seems unavoidable.

JukkaL · 2025-11-25T13:36:05Z

mypyc/irbuild/callable_class.py

+        "__annotations__": cpyfunction_get_annotations,
+        "__defaults__": cpyfunction_get_defaults,
+        "__kwdefaults__": cpyfunction_get_kwdefaults,
+    }


Add at least basic testing of all the new properties. Also test the attributes of the wrapper objects when relevant. Simple tests are sufficient, but it would be good that each code path has at least one test that explicitly uses it.

JukkaL

Thanks for the updates!

p-sawicki changed the title ~~[mypyc] Wrap async functions with FunctionType-like type~~ [mypyc] Wrap async functions with function-like type Nov 19, 2025

p-sawicki force-pushed the async-function-wrapper branch from 889a267 to 585f818 Compare November 19, 2025 15:31

JukkaL reviewed Nov 20, 2025

View reviewed changes

JukkaL reviewed Nov 24, 2025

View reviewed changes

JukkaL reviewed Nov 25, 2025

View reviewed changes

JukkaL approved these changes Nov 28, 2025

View reviewed changes

p-sawicki added 6 commits November 28, 2025 12:27

Wrap async functions in cpython-like objects

79f9e83

Fix compilation on older python versions

f479e48

Fix test hang on Windows

08f1e2e

Check return values

03ed512

Address review comments

34167d5

Add tests with static and class methods

816924d

p-sawicki force-pushed the async-function-wrapper branch from fdf1e97 to 816924d Compare November 28, 2025 11:34

p-sawicki merged commit 81eaa5d into python:master Nov 28, 2025
14 checks passed

p-sawicki deleted the async-function-wrapper branch November 28, 2025 13:38

RiyaPrabhakar-git added this to GC-Content-Calculator Dec 12, 2025

github-project-automation bot moved this from Todo to Done in GC-Content-Calculator Dec 12, 2025

github-project-automation bot moved this to Todo in GC-Content-Calculator Dec 12, 2025

Uh oh!

[mypyc] Wrap async functions with function-like type #20260

[mypyc] Wrap async functions with function-like type #20260

Uh oh!

Conversation

p-sawicki commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

p-sawicki commented Nov 19, 2025 •

edited

Loading