ctypes should return composite types from callbacks

[gholling]

BPO	5710
Nosy	@theller, @amauryfa, @freakboy3742, @albertz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2009-04-06.17:28:44.019>
labels = ['ctypes', 'type-feature']
title = 'ctypes should return composite types from callbacks'
updated_at = <Date 2017-11-23.02:04:38.940>
user = 'https://bugs.python.org/gholling'

bugs.python.org fields:

activity = <Date 2017-11-23.02:04:38.940>
actor = 'freakboy3742'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2009-04-06.17:28:44.019>
creator = 'gholling'
dependencies = []
files = []
hgrepos = []
issue_num = 5710
keywords = []
message_count = 8.0
messages = ['85654', '85731', '85749', '198501', '255582', '306730', '306731', '306778']
nosy_count = 7.0
nosy_names = ['theller', 'amaury.forgeotdarc', 'gholling', 'freakboy3742', 'Albert.Zeyer', 'Pam.McANulty', 'Mason.Bially']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue5710'
versions = ['Python 2.7', 'Python 3.4']

[gholling]

We have an application that calls a 3rd party library that returns a
structure (by value) from a callback. I'm including some sample code
that duplicates the behavior. The problem is that return types from
callbacks cannot be anything other than simple datatypes (c_int,
c_float, ..., c_void_p). For other datatypes (STRUCTURE, POINTER, ...),
ctypes returns the following error:

"invalid result type for callback function"

The error message comes from callback.c, in function AllocFunctionCallback.

I think this may be a duplicate of issue bpo-1574584.

I've tested this on Windows and linux; I'm including a Makefile and
source code that works on Windows/cygwin.

------ file: Makefile ----

all: hello.dll hello.exe hello_dll.exe
clean:
rm *.exe *.dll

hello.exe: hello.h hello.c hello_main.c
gcc hello.c hello_main.c -o hello.exe --save-temps

hello.dll: hello.h hello.c
gcc -mno-cygwin -shared hello.c -o hello.dll --save-temps

hello_dll.exe: hello.h hello.c
gcc hello_main.c -L. -lhello -o hello_main.exe

------ file: hello.h ----

struct helloStruct {
  int i;
  float f;
  int i2;
  int i3;
};

float fxn (struct  helloStruct callback());

------ file: hello.c ----

#include <stdio.h> 
#include "hello.h"

float fxn (struct helloStruct callback()) {
  struct  helloStruct result = callback();

printf ("i: %d\n", result.i);
printf ("f: %f\n", result.f);
return result.f * result.i;
}

------ file: hello_main.c ----

#include <stdio.h> 
#include "hello.h"

struct helloStruct callback();

int main (int argc, char **argv) {
  float f;
  struct helloStruct result;
  printf ("Hello world\n");
  f = fxn (callback);
  printf ("Callback result: %f\n", f);
}

struct helloStruct callback () {
  struct helloStruct result;
  result.i = 10;
  result.f = 3.14159;
  return result;
}

int int_callback () {
  return 42;
}

------ file: hello.py ----

from ctypes import cdll, c_char, c_int, c_float, Structure, CFUNCTYPE,
POINTER, c_char_p

class helloStruct (Structure):
    pass
helloStruct._fields_ = [
    ('i', c_int),
    ('f', c_float)    ]

def callback():
    print ("callback()")
    hs = helloStruct()
    hs.i = 10
    hs.f = 25.5
    return hs

libc = cdll.msvcrt
#helloLib = libc.load_library("hello")
#helloLib = libc.hello
helloLib = cdll.hello

helloLib.fxn.restype = helloStruct
# It looks like only simple return types are supported for
# callback functions.  simple = c_int, c_float, ...
# Python bug # 1574584 - status: closed.
#    Suggests posting to ctypes-users, but I don't see any recent activity.
#
TMP_FCN = CFUNCTYPE (POINTER(c_char))           # Error message
#TMP_FCN = CFUNCTYPE (c_char_p)                  # Runs, but invalid result
#TMP_FCN = CFUNCTYPE (c_void_p)                  # Runs, but invalid result
#TMP_FCN = CFUNCTYPE (c_int)                  # Runs, but invalid result
#TMP_FCN = CFUNCTYPE (POINTER(c_int))         # Error message
#TMP_FCN = CFUNCTYPE (POINTER(helloStruct))   # Error message
#TMP_FCN = CFUNCTYPE (helloStruct)            # Error message
callback_fcn = TMP_FCN (callback)
result = helloLib.fxn (callback_fcn)

# 2.5
#print "result: ", result
# 3.0
print ("result: ", result)

[theller]

There is a problem returning arbitrary complicated ctypes types from
callbacks. Consider that a callback function returns a structure; the
structure itself may contain 'char *' field for example. The callback
function creates the structure and fills in the fields. The 'char *'
field contains a pointer to a nul-terminated string; the ctypes
structure object must keep the Python string alive as long as the
structure itself. It does this via the private '_objects' instance
variable. If the callback function now returns the structure, the
Python ctypes object usually will go out of scope and will be garbage
collected, together with its '_objects'. So, the pointer(s) contained
in the structure will become invalid now, and even simple structure
fields will become invalid.

However, the callback function result will be used by some other code
and will surely crash.

In principle it should be possible to make code like this work; one
would have to maintain the returned objects elsewhere and clean them up
when they are no longer used anywhere; but I think this gets too
complicated.

[amauryfa]

But isn't this purely a user-side concern?

For example, if I want to use a function such as QBuffer::setBuffer in
the Qt library:
http://doc.trolltech.com/4.4/qbuffer.html#setBuffer
I must keep a reference to the buffer as long as the QBuffer is alive,
or expect a crash.

Returning a pointer from a function is always tough, even in C:
everybody has already tried to return the address of a local variable...
the pointer must belong to some container that outlives the function
call.
ctypes is not different in this aspect. The same precautions as in C
apply. And with a warning note in the documentation, there seems to be
no reason to limit the return type of a callback.

[MasonBially]

I agree with Amaury that this is purely a user side concern. While I think it's important to note the behaviour of ctypes in the case that Thomas describes, I believe it's more important to fully support the range of behaviours allowed by C function callbacks.

I see the use cases for complex return types that don't fall under the concerns raised by Thomas as the following:

• Returning a pointer to already existing memory.
• Returning complex value types (structs of ints, doubles, chars, ect). This is especially important from a compatibility standpoint for C libraries which expect such return types.

Because I need this for my current project I will work on writing a patch.

[albertz]

Any update here?

[PamMcANulty]

In the example code, the 'result' variable is declared on the stack in the callback() function. So it will be effectively when callback() returns.

Also the "result" variable in main() is never referenced. A pointer main()'s "result" variable should be passed to callback() as an argument ("&result") and callback(struct helloStruct *result) should populate it (via result->i = 10; etc)

  struct helloStruct result;
  result.i = 10;
  result.f = 3.14159;
  return result;
}```

[PamMcANulty]

oops -
"In the example code, the 'result' variable is declared on the stack in the callback() function. So it will be effectively when callback() returns." should be
"In the example code, the 'result' variable is declared on the stack in the callback() function. So it will be effectively DESTROYED when callback() returns."

I hate waking up at 4:15a.

[freakboy3742]

For those interested, we developed a workaround for this in Rubicon:

https://github.com/pybee/rubicon-objc/pull/85/files

The fix involves using ctypes to access ctypes own internals, and build a modified version of the Structure data type that is able to perform a copy when used as returned value.

Hopefully we'll be able to get this into the form of a patch for ctypes that is acceptable to Python core.

[freakboy3742]

A follow up - the Rubicon patch has been updated to account for the changes to ctypes introduced in Python 3.13.0a6, and to account for the resolution of #81061.

https://github.com/beeware/rubicon-objc/blob/main/src/rubicon/objc/ctypes_patch.py