PyArray_Descr API is not compatible with the Python Limited API

[lpsinger]

Functions to create a new array using a PyArray_Descr * do not support the PEP 384 Python Limited API.

Reproducing code example:

foo.c

#include <Python.h>
#include <numpy/arrayobject.h>

static PyObject *bar(PyObject *NPY_UNUSED(module), PyObject *NPY_UNUSED(args))
{
    PyObject *dtype;
    PyArray_Descr *descr;
    npy_intp dims[] = {10};
    int result;

    dtype = Py_BuildValue("[(ss)(ss)]", "bat", "f8", "baz", "i8");
    if (!dtype)
        return NULL;

    result = PyArray_DescrConverter(dtype, &descr);
    Py_DECREF(dtype);
    if (result != NPY_SUCCEED)
        return NULL;
    
    return PyArray_SimpleNewFromDescr(1, dims, descr);
}

static PyMethodDef methods[] = {
    {"bar", bar, METH_NOARGS, "fill me in"},
    {NULL, NULL, 0, NULL}
};

static PyModuleDef module = {
    .m_base = PyModuleDef_HEAD_INIT,
    .m_name = "foo",
    .m_methods = methods,
    .m_size = -1
};

PyMODINIT_FUNC PyInit_foo(void); /* Silence -Wmissing-prototypes */
PyMODINIT_FUNC PyInit_foo(void)
{
    import_array();
    return PyModule_Create(&module);
}

setup.py

from setuptools import setup, Extension
import numpy as np

ext = Extension('foo', ['foo.c'], include_dirs=[np.get_include()])
setup(name='numpy-descr-limited-api-example', ext_modules=[ext])

Error message:

Failed output with limited Python API

$ env CC=gcc CFLAGS="-DPy_LIMITED_API=0x03060000" pip install . && python -c 'import foo; print(repr(foo.bar()))'
Processing /Users/lpsinger/Desktop/numpy-descr-limited-api-example
Using legacy setup.py install for numpy-descr-limited-api-example, since package 'wheel' is not installed.
Installing collected packages: numpy-descr-limited-api-example
  Attempting uninstall: numpy-descr-limited-api-example
    Found existing installation: numpy-descr-limited-api-example 0.0.0
    Uninstalling numpy-descr-limited-api-example-0.0.0:
      Successfully uninstalled numpy-descr-limited-api-example-0.0.0
    Running setup.py install for numpy-descr-limited-api-example ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/cs/pwq97bws17d2p8qcp410gs7c84mbdz/T/pip-req-build-_dx9jmw9/setup.py'"'"'; __file__='"'"'/private/var/folders/cs/pwq97bws17d2p8qcp410gs7c84mbdz/T/pip-req-build-_dx9jmw9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/cs/pwq97bws17d2p8qcp410gs7c84mbdz/T/pip-record-8ur13b4e/install-record.txt --single-version-externally-managed --compile --install-headers /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/include/site/python3.8/numpy-descr-limited-api-example
         cwd: /private/var/folders/cs/pwq97bws17d2p8qcp410gs7c84mbdz/T/pip-req-build-_dx9jmw9/
    Complete output (29 lines):
    running install
    running build
    running build_ext
    building 'foo' extension
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -DPy_LIMITED_API=0x03060000 -I/Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include -I/Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c foo.c -o build/temp.macosx-10.15-x86_64-3.8/foo.o
    In file included from /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                     from /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                     from /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from foo.c:2:
    /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
       17 | #warning "Using deprecated NumPy API, disable it with " \
          |  ^~~~~~~
    In file included from /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:21,
                     from /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from foo.c:2:
    foo.c: In function 'bar':
    /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include/numpy/__multiarray_api.h:642:23: error: dereferencing pointer to incomplete type 'PyTypeObject' {aka 'struct _typeobject'}
      642 | #define PyArray_Type (*(PyTypeObject *)PyArray_API[2])
          |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:125:31: note: in expansion of macro 'PyArray_Type'
      125 |         PyArray_NewFromDescr(&PyArray_Type, descr, nd, dims, \
          |                               ^~~~~~~~~~~~
    foo.c:19:12: note: in expansion of macro 'PyArray_SimpleNewFromDescr'
       19 |     return PyArray_SimpleNewFromDescr(1, dims, descr);
          |            ^~~~~~~~~~~~~~~~~~~~~~~~~~
    foo.c:20:1: warning: control reaches end of non-void function [-Wreturn-type]
       20 | }
          | ^
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
  Rolling back uninstall of numpy-descr-limited-api-example
  Moving to /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/foo.cpython-38-darwin.so
   from /private/var/folders/cs/pwq97bws17d2p8qcp410gs7c84mbdz/T/pip-uninstall-78_wo8n6/foo.cpython-38-darwin.so
  Moving to /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/numpy_descr_limited_api_example-0.0.0-py3.8.egg-info
   from /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/lib/python3.8/site-packages/~umpy_descr_limited_api_example-0.0.0-py3.8.egg-info
ERROR: Command errored out with exit status 1: /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/cs/pwq97bws17d2p8qcp410gs7c84mbdz/T/pip-req-build-_dx9jmw9/setup.py'"'"'; __file__='"'"'/private/var/folders/cs/pwq97bws17d2p8qcp410gs7c84mbdz/T/pip-req-build-_dx9jmw9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/cs/pwq97bws17d2p8qcp410gs7c84mbdz/T/pip-record-8ur13b4e/install-record.txt --single-version-externally-managed --compile --install-headers /Users/lpsinger/Desktop/numpy-descr-limited-api-example/env/include/site/python3.8/numpy-descr-limited-api-example Check the logs for full command output.

Successful output with full Python API

$ env CC=gcc pip install . && python -c 'import foo; print(repr(foo.bar()))'
Processing /Users/lpsinger/Desktop/numpy-descr-limited-api-example
Using legacy setup.py install for numpy-descr-limited-api-example, since package 'wheel' is not installed.
Installing collected packages: numpy-descr-limited-api-example
  Attempting uninstall: numpy-descr-limited-api-example
    Found existing installation: numpy-descr-limited-api-example 0.0.0
    Uninstalling numpy-descr-limited-api-example-0.0.0:
      Successfully uninstalled numpy-descr-limited-api-example-0.0.0
    Running setup.py install for numpy-descr-limited-api-example ... done
Successfully installed numpy-descr-limited-api-example-0.0.0
array([(3.10503618e+231, 8070450532247928832),
       (4.70372193e+000, 4613349226564724111),
       (2.73861279e+000, 4613349226564724111),
       (2.23606798e+000, 4612217596255138984),
       (2.23606798e+000,    1608428438346624),
       (4.70372193e+000, 4613349226564724111),
       (2.23606798e+000, 4616981938510815294),
       (2.73861279e+000, 4612217596255138984),
       (4.70372193e+000, 4613349226564724111),
       (2.23606798e+000,    2814754097651120)],
      dtype=[('bat', '<f8'), ('baz', '<i8')])

Numpy/Python version information:

>>> import sys, numpy; print(numpy.__version__, sys.version)
1.19.1 3.8.5 (default, Jul 21 2020, 18:31:18) 
[Clang 11.0.3 (clang-1103.0.32.62)]

[eric-wieser]

A simple reproducer is https://godbolt.org/z/dc8vaz

This seems to pass in GCC 10, and in all versions of clang.

So I'd argue this is a GCC bug.

[lpsinger]

Is this a bug in GCC? Or is this a nonstandard GCC/Clang extension?

[eric-wieser]

I'm honestly not sure. In my opinion, dereferencing an opaque struct only to take it's address again should be legal, no memory access actually occurs.

[lpsinger]

Here are the related GCC bug reports, which were indeed fixed in GCC 10:

• https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88827
• https://gcc.gnu.org/bugzilla/show_bug.cgi?id=36941

The bug reports refer to 6.5.3.2/3 of ISO/IEC 9899:2018, which describes a special rule for evaluation of the & operator when the operand is the result of the * operator.

So yes, this is standards-conforming C, and it was a bug in GCC to reject it, fixed in GCC 10 and later.

I was able to work around this in my project by adding a blank forward declaration of PyTypeObject (see lpsinger/ligo.skymap@b1bc1ab).