Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Passing an array of size zero doesn't work #239

Open
marktsuchida opened this issue May 26, 2024 · 4 comments
Open

Passing an array of size zero doesn't work #239

marktsuchida opened this issue May 26, 2024 · 4 comments

Comments

@marktsuchida
Copy link
Contributor

marktsuchida commented May 26, 2024 •
edited
Loading

Calling a function with a pointer parameter with a Python object implementing the buffer protocol does not appear to work when the object has zero elements. Is this intended behavior?

>>> import cppyy, array
>>> cppyy.cppdef("void f(unsigned char const *buf) {}")
True
>>> cppyy.gbl.f(array.array('B', [1, 2, 3]))  # OK
>>> cppyy.gbl.f(array.array('B', []))         # Fails
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: void ::f(const unsigned char* buf) =>
    TypeError: could not convert argument 1 (could not convert argument to buffer or nullptr)

(My hope was that f() would get called, either with a non-null pointer that f should not dereference, or with nullptr.)

Furthermore, a memoryview of size 0 causes a crash:

>>> cppyy.gbl.f(memoryview(array.array('B', [1, 2, 3])))  # OK
>>> cppyy.gbl.f(memoryview(array.array('B', [])))         # Crashes, exit code 129
 *** Break *** segmentation violation
Full stack trace

(It looks like the same thing gets printed twice.)

>>> cppyy.gbl.f(memoryview(array.array('B', [])))
 *** Break *** segmentation violation
[/Users/mark/tmp/venv/lib/python3.12/site-packages/cppyy_backend/lib/libcppyy_backend.so] (anonymous namespace)::TExceptionHandlerImp::HandleException(int) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/cppyy_backend/lib/libCoreLegacy.so] CppyyLegacy::TUnixSystem::DispatchSignals(CppyyLegacy::ESignals) (no debug info)
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyUnicode_FromFormatV (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyUnicode_FromFormat (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::Utility::GetBuffer(_object*, char, int, void*&, bool) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::(anonymous namespace)::UCharArrayConverter::SetArg(_object*, CPyCppyy::Parameter&, CPyCppyy::CallContext*) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::CPPMethod::ConvertAndSetArgs(_object* const*, unsigned long, CPyCppyy::CallContext*) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::CPPFunction::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::(anonymous namespace)::mp_vectorcall(CPyCppyy::CPPOverload*, _object* const*, unsigned long, _object*) (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyEval_EvalCode (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] builtin_exec (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] gen_send_ex2 (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] gen_send_ex (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] method_vectorcall (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyVectorcall_Call (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyEval_EvalCode (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] run_eval_code_obj (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] run_mod (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pyrun_file (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyRun_SimpleFileObject (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyRun_AnyFileObject (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pymain_run_file_obj (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pymain_run_file (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] Py_RunMain (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] Py_BytesMain (no debug info)
[/usr/lib/dyld] start (no debug info)
 *** Break *** segmentation violation
[/Users/mark/tmp/venv/lib/python3.12/site-packages/cppyy_backend/lib/libcppyy_backend.so] (anonymous namespace)::TExceptionHandlerImp::HandleException(int) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/cppyy_backend/lib/libCoreLegacy.so] CppyyLegacy::TUnixSystem::DispatchSignals(CppyyLegacy::ESignals) (no debug info)
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyUnicode_FromFormatV (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyUnicode_FromFormat (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::Utility::GetBuffer(_object*, char, int, void*&, bool) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::(anonymous namespace)::UCharArrayConverter::SetArg(_object*, CPyCppyy::Parameter&, CPyCppyy::CallContext*) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::CPPMethod::ConvertAndSetArgs(_object* const*, unsigned long, CPyCppyy::CallContext*) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::CPPFunction::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) (no debug info)
[/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::(anonymous namespace)::mp_vectorcall(CPyCppyy::CPPOverload*, _object* const*, unsigned long, _object*) (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyEval_EvalCode (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] builtin_exec (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] gen_send_ex2 (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] gen_send_ex (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] method_vectorcall (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyVectorcall_Call (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyEval_EvalCode (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] run_eval_code_obj (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] run_mod (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pyrun_file (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyRun_SimpleFileObject (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyRun_AnyFileObject (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pymain_run_file_obj (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pymain_run_file (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] Py_RunMain (no debug info)
[/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] Py_BytesMain (no debug info)
[/usr/lib/dyld] start (no debug info)

This was with Python 3.12.3 from Homebrew, macOS 14, arm64,
cppyy 3.1.2,
cppyy-backend 1.15.2,
cppyy-cling 6.30.0,
CPyCppyy 1.12.16.

I got the same behavior with Python 3.12.2, x86-64, from conda-forge (running on Rosetta 2), same cppyy versions, except that the error printed before crashing was *** Break *** floating point exception. The stack trace was essentially identical to the one above except that some frames were omitted. The exit code was 140 (SIGSYS?) instead of 129 (SIGHUP?), FWIW.

Passing bytes() raised TypeError similarly to the zero-length array.array, and passing np.frombuffer(bytes(), dtype=np.uint8) crashed similarly to the zero-length memoryview.
@wlav
Copy link
Owner

wlav commented May 27, 2024 •
edited
Loading

The segfault should obviously not happen (will fix that); the others are neither here nor there: an empty Python container is not the same as a null pointer. The code fails b/c it looks for a compatible buffer by asking the object for one through the Python buffer interface and doesn't get any. It doesn't know/check the type to figure if it's a compatible one that could have served up a buffer if only the object had a buffer: there's no end to those cases (bytearray, array, numpy.ndarray, bytes, memoryview, LowLevelView, ctypes.c_ubyte*0, ...).

To pass a null pointer here, use cppyy.nullptr:

>>> import cppyy, array
>>> cppyy.cppdef("void f(unsigned char const *buf) {}")
True
>>> cppyy.gbl.f(cppyy.nullptr)
>>>

@wlav
Copy link
Owner

wlav commented May 27, 2024

Interesting, the crash with the memoryview is in Python, not in cppyy:

    if (PyObject_CheckBuffer(pyobject)) {
        Py_buffer bufinfo;
        memset(&bufinfo, 0, sizeof(Py_buffer));
        if (PyObject_GetBuffer(pyobject, &bufinfo, PyBUF_FORMAT) == 0) {    // <- this crashes

and oddly, it only happens on Linux.

I've put in a protection against asking sequences of length 0 for their buffer info. However, this is once more a case where it's clear that an empty container-like object just isn't a nullptr.

@marktsuchida
Copy link
Contributor Author

Thanks. I agree that an empty buffer is not the same thing as nullptr, but was approaching this from the example in the docs (void array_method(int* ad, int size)) in which a zero size would very much make sense (actually I'm trying to come up with a way to translate objects implementing the buffer protocol into std::span). The workaround of special-casing the zero-size case on the Python side will definitely work for me.

The code fails b/c it looks for a compatible buffer by asking the object for one through the Python buffer interface and doesn't get any.

I'm a little confused by this, because I'm pretty sure you do get a buffer interface from an empty buffer object:

In [3]: cppyy.include("Python.h")                                               
Out[3]: True                                                                                                            

In [20]: cppyy.cppdef("""
    ...: PyObject *i(PyObject *pyobj) {
    ...:     if (PyObject_CheckBuffer(pyobj) != 1) {
    ...:         PyErr_SetString(PyExc_TypeError, "Not a buffer");
    ...:         return NULL;
    ...:     }
    ...:     Py_buffer bufinfo;
    ...:     memset(&bufinfo, 0, sizeof(bufinfo));
    ...:     if (PyObject_GetBuffer(pyobj, &bufinfo, PyBUF_FORMAT) != 0)
    ...:         return NULL;
    ...:     PyObject *ret = PyUnicode_FromString(bufinfo.format);
    ...:     PyBuffer_Release(&bufinfo);
    ...:     return ret;
    ...: }
    ...: """)
Out[20]: True

In [21]: cppyy.gbl.i(42)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[21], line 1
----> 1 cppyy.gbl.i(42)

TypeError: _object* ::i(PyObject* pyobj) =>
    TypeError: Not a buffer

In [22]: cppyy.gbl.i(b'abc')
Out[22]: 'B'

In [23]: cppyy.gbl.i(bytes())
Out[23]: 'B'

The reason empty buffers are not treated as buffers appears to be because CPyCppyy::Utility::GetBuffer() returns 0 both for non-buffers and otherwise compatible buffers that happen to be empty. (But if you feel that maintaining this behavior (TypeError on empty buffer) is important for backward compatibility, I certainly understand.)

Interestingly, wrapping in memoryview throws an exception in this case (from the PyObject_GetBuffer() call):

In [24]: cppyy.gbl.i(memoryview(bytes()))
---------------------------------------------------------------------------
BufferError                               Traceback (most recent call last)
Cell In[24], line 1
----> 1 cppyy.gbl.i(memoryview(bytes()))

BufferError: _object* ::i(PyObject* pyobj) =>
    BufferError: memoryview: cannot cast to unsigned bytes if the format flag is present

And the exception is not limited to the empty case. So I guess memoryview just doesn't work when PyBUF_FORMAT is requested (alone). cppyy.gbl.j(memoryview(bytes())) does return 'B', where j() is identical to i() above except for PyBUF_FORMAT being replaced with PyBUF_ND | PyBUF_FORMAT.

The docs say "PyBUF_FORMAT can be |’d to any of the flags except PyBUF_SIMPLE. The latter already implies format B (unsigned bytes)." And PyBUF_SIMPLE equals 0, so PyBUF_FORMAT on its own may be problematic.

(I don't really understand why non-empty memoryview worked in my initial post. Perhaps it is handled by the fallback code after PyObject_GetBuffer() fails?)

@wlav
Copy link
Owner

wlav commented Dec 17, 2024

The crash is gone with the release of cppyy 3.5.0 and its dependencies. Haven't decided yet on what to do with passing empty arrays, so keeping this open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@marktsuchida @wlav