Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compilation fails with python 3.11 #10305

Closed
ConiKost opened this issue Jul 22, 2022 · 13 comments
Closed

Compilation fails with python 3.11 #10305

ConiKost opened this issue Jul 22, 2022 · 13 comments
Assignees
@haberman
Labels
python

Comments

@ConiKost
Copy link

What version of protobuf and what language are you using?
Version: 3.21.3
Language: Python

What operating system (Linux, Windows, ...) and version?
Gentoo Linux

What runtime / compiler are you using (e.g., python version or gcc version)

$ python --version
Python 3.11.0b4

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/12.1.1/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-12.1.1_p20220625/work/gcc-12-20220625/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/12.1.1 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/12.1.1/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/12.1.1 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/12.1.1/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/12.1.1/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/12.1.1/include/g++-v12 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/12.1.1/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --disable-libunwind-exceptions --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 12.1.1_p20220625 p8' --disable-esp --enable-libstdcxx-time --disable-libstdcxx-pch --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-fixed-point --enable-targets=all --enable-libgomp --disable-libssp --disable-libada --disable-cet --disable-systemtap --disable-valgrind-annotations --enable-vtable-verify --with-zstd --enable-lto --with-isl --disable-isl-version-check --enable-default-pie --enable-default-ssp --with-build-config=bootstrap-lto
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.1.1 20220625 (Gentoo 12.1.1_p20220625 p8)

What did you do?
Compile python project

What did you expect to see
Compilation works with python 3.11.

What did you see instead?
Compilation fails with python 3.11.

x86_64-pc-linux-gnu-g++ -shared -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -Wl,-O1 -Wl,--sort-common -Wl,-z,now -march=native -mfpmath=sse,387 -mtune=intel -O3 -frecord-gcc-switches -fomit-frame-pointer -malign-data=abi -mtls-dialect=gnu2 -pipe build/temp.linux-x86_64-cpython-311/google/protobuf/internal/api_implementation.o -L/usr/lib64 -o build/lib.linux-x86_64-cpython-311/google/protobuf/internal/_api_implementation.cpython-311-x86_64-linux-gnu.so
google/protobuf/pyext/descriptor.cc: In function ‘bool google::protobuf::python::_CalledFromGeneratedFile(int)’:
google/protobuf/pyext/descriptor.cc:103:18: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  103 |     frame = frame->f_back;
      |                  ^~
In file included from /usr/include/python3.11/Python.h:42,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
/usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:109:12: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  109 |   if (frame->f_code->co_filename == nullptr) {
      |            ^~
/usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
In file included from /usr/include/python3.11/Python.h:44:
google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
/usr/include/python3.11/object.h:775:59: note: in definition of macro ‘PyType_FastSubclass’
  775 | #define PyType_FastSubclass(type, flag) PyType_HasFeature(type, flag)
      |                                                           ^~~~
/usr/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_CAST’
  107 | #define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
      |                            ^~~~~~~~
/usr/include/python3.11/object.h:136:31: note: in expansion of macro ‘_PyObject_CAST’
  136 | #  define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob))
      |                               ^~~~~~~~~~~~~~
/usr/include/python3.11/unicodeobject.h:115:25: note: in expansion of macro ‘Py_TYPE’
  115 |     PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_UNICODE_SUBCLASS)
      |                         ^~~~~~~
google/protobuf/pyext/descriptor.cc:54:4: note: in expansion of macro ‘PyUnicode_Check’
   54 |   (PyUnicode_Check(ob)                                           \
      |    ^~~~~~~~~~~~~~~
google/protobuf/pyext/descriptor.cc:114:7: note: in expansion of macro ‘PyString_AsStringAndSize’
  114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |       ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
google/protobuf/pyext/descriptor.cc:56:40: note: in definition of macro ‘PyString_AsStringAndSize’
   56 |                PyUnicode_AsUTF8AndSize(ob, (sizep)))) == nullptr \
      |                                        ^~
/usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
google/protobuf/pyext/descriptor.cc:59:34: note: in definition of macro ‘PyString_AsStringAndSize’
   59 |        : PyBytes_AsStringAndSize(ob, (charpp), (sizep)))
      |                                  ^~
/usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:135:12: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  135 |   if (frame->f_globals != frame->f_locals) {
      |            ^~
/usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:135:32: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  135 |   if (frame->f_globals != frame->f_locals) {
      |                                ^~
/usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
@ConiKost ConiKost added the untriaged auto added to all issues by default when created. label Jul 22, 2022
@zhangskz zhangskz added python and removed untriaged auto added to all issues by default when created. labels Jul 22, 2022
@zhangskz
Copy link
Member

3.11 is still in pre-release so I think this isn't an issue for 21.3 (note, for python this is technically 4.21.3). We'll likely want to make sure we support 3.11 eventually when this is closer to release.

@mkillianey
Copy link

@zhangskz Can we re-open, now that 3.11 has been released?

@zhangskz
Copy link
Member

zhangskz commented Oct 25, 2022 •
edited
Loading

I suspect this is fixed already by #10403. But yes, please comment if this is still an issue and we can reopen it.

@xylar
Copy link

xylar commented Oct 26, 2022

Does not appear to be fixed, see:
conda-forge/protobuf-feedstock#167

@xylar
Copy link

xylar commented Oct 26, 2022 •
edited
Loading 

  google/protobuf/pyext/descriptor.cc: In function 'bool google::protobuf::python::_CalledFromGeneratedFile(int)':
  google/protobuf/pyext/descriptor.cc:103:18: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
    103 |     frame = frame->f_back;
        |                  ^~
  In file included from /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/Python.h:42,
                   from ./google/protobuf/pyext/descriptor.h:37,
                   from google/protobuf/pyext/descriptor.cc:33:
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
     22 | typedef struct _frame PyFrameObject;
        |                ^~~~~~
  google/protobuf/pyext/descriptor.cc:109:12: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
    109 |   if (frame->f_code->co_filename == nullptr) {
        |            ^~
  In file included from /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/Python.h:42,
                   from ./google/protobuf/pyext/descriptor.h:37,
                   from google/protobuf/pyext/descriptor.cc:33:
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
     22 | typedef struct _frame PyFrameObject;
        |                ^~~~~~
  In file included from /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/Python.h:44,
                   from ./google/protobuf/pyext/descriptor.h:37,
                   from google/protobuf/pyext/descriptor.cc:33:
  google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
    114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
        |                                     ^~
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/object.h:775:59: note: in definition of macro 'PyType_FastSubclass'
    775 | #define PyType_FastSubclass(type, flag) PyType_HasFeature(type, flag)
        |                                                           ^~~~
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/object.h:107:28: note: in expansion of macro '_Py_CAST'
    107 | #define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
        |                            ^~~~~~~~
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/object.h:136:31: note: in expansion of macro '_PyObject_CAST'
    136 | #  define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob))
        |                               ^~~~~~~~~~~~~~
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/unicodeobject.h:115:25: note: in expansion of macro 'Py_TYPE'
    115 |     PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_UNICODE_SUBCLASS)
        |                         ^~~~~~~
  google/protobuf/pyext/descriptor.cc:54:4: note: in expansion of macro 'PyUnicode_Check'
     54 |   (PyUnicode_Check(ob)                                           \
        |    ^~~~~~~~~~~~~~~
  google/protobuf/pyext/descriptor.cc:114:7: note: in expansion of macro 'PyString_AsStringAndSize'
    114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
        |       ^~~~~~~~~~~~~~~~~~~~~~~~
  In file included from /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/Python.h:42,
                   from ./google/protobuf/pyext/descriptor.h:37,
                   from google/protobuf/pyext/descriptor.cc:33:
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
     22 | typedef struct _frame PyFrameObject;
        |                ^~~~~~
  google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
    114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
        |                                     ^~
  google/protobuf/pyext/descriptor.cc:56:40: note: in definition of macro 'PyString_AsStringAndSize'
     56 |                PyUnicode_AsUTF8AndSize(ob, (sizep)))) == nullptr \
        |                                        ^~
  In file included from /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/Python.h:42,
                   from ./google/protobuf/pyext/descriptor.h:37,
                   from google/protobuf/pyext/descriptor.cc:33:
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
     22 | typedef struct _frame PyFrameObject;
        |                ^~~~~~
  google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
    114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
        |                                     ^~
  google/protobuf/pyext/descriptor.cc:59:34: note: in definition of macro 'PyString_AsStringAndSize'
     59 |        : PyBytes_AsStringAndSize(ob, (charpp), (sizep)))
        |                                  ^~
  In file included from /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/Python.h:42,
                   from ./google/protobuf/pyext/descriptor.h:37,
                   from google/protobuf/pyext/descriptor.cc:33:
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
     22 | typedef struct _frame PyFrameObject;
        |                ^~~~~~
  google/protobuf/pyext/descriptor.cc:135:12: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
    135 |   if (frame->f_globals != frame->f_locals) {
        |            ^~
  In file included from /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/Python.h:42,
                   from ./google/protobuf/pyext/descriptor.h:37,
                   from google/protobuf/pyext/descriptor.cc:33:
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
     22 | typedef struct _frame PyFrameObject;
        |                ^~~~~~
  google/protobuf/pyext/descriptor.cc:135:32: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
    135 |   if (frame->f_globals != frame->f_locals) {
        |                                ^~
  In file included from /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/Python.h:42,
                   from ./google/protobuf/pyext/descriptor.h:37,
                   from google/protobuf/pyext/descriptor.cc:33:
  /home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
     22 | typedef struct _frame PyFrameObject;
        |                ^~~~~~
  error: command '/home/conda/feedstock_root/build_artifacts/protobuf_1666803804985/_build_env/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
  error: subprocess-exited-with-error

@xylar
Copy link

xylar commented Oct 26, 2022

Apologies, I see that this fix simply hasn't been part of a release yet. When can we expect it? (It's holding up migration of at least 95 other packages to python 3.11 on conda-forge.)

@haberman
Copy link
Member

Can you clarify what exactly is being built here that causes this issue?

As of Protobuf 4.21.0, the code in https://github.com/protocolbuffers/protobuf/tree/main/python/google/protobuf/pyext is no longer part of our official Python release. It is not compiled or distributed in our pip packages at all. The extension has been completely rewritten (while still implementing the same API) and the new code is here: https://github.com/protocolbuffers/upb/tree/main/python

So in some sense the fixes submitted in #10403 will never be released, because we are no longer releasing this code.

Which brings us to the question of how you end up building this legacy code. What were you building that actually triggered this error?

@xylar
Copy link

xylar commented Oct 26, 2022

We are building with the code from this Git repo.

@xylar xylar mentioned this issue Oct 26, 2022
Merged
@xylar
Copy link

xylar commented Oct 26, 2022

I will see how it goes if we switch to building from PyPI instead.

@haberman
Copy link
Member

I am not sure that we have a source package in pip that will build the new code as you would expect. Our general assumption has been that if we build the binaries and release them on pip, other projects will not need to build them again.

What is your use case for building the extension module yourself?

@xylar
Copy link

xylar commented Oct 26, 2022

conda-forge is a completely different package manager. It cannot use PyPI binaries, we build our own. So we definitely do need to be building from source. If necessary source for building the extensions has been stripped out of the code distributed on PyPI, we won't be able to use that.

@haberman
Copy link
Member

It appears that our source package currently contains the old code (ie. the code from https://github.com/protocolbuffers/protobuf/tree/main/python/google/protobuf/pyext). This may even continue to be buildable via pip. I think you may need to pass --cpp_implementation: https://github.com/protocolbuffers/protobuf/blob/main/python/setup.py#L283. You could also build directly from Git sources.

We should update our source package to contain the new code at https://github.com/protocolbuffers/upb/tree/main/python, and an appropriate setup.py so that it is buildable. I can file a bug for this, but it won't be resolved fast enough to address your current issue.

@xylar
Copy link

xylar commented Oct 26, 2022

We do, indeed use --cpp_implementation. I will switch to getting source from PyPI but based on what you described, it is unlikely to solve the problem. I will therefore patch with #10403 for now. (However, that PR introduced bugs for PyPy that I am going to also make a PR here to try to fix.)

This was referenced Oct 26, 2022
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@haberman haberman

Labels
python
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@haberman @mkillianey @xylar @ConiKost @zhangskz