Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: incompatibility with CPython default branch (future py311) #3417

Closed
2 of 3 tasks
tacaswell opened this issue Oct 26, 2021 · 3 comments
Closed
2 of 3 tasks

[BUG]: incompatibility with CPython default branch (future py311) #3417

tacaswell opened this issue Oct 26, 2021 · 3 comments
Labels
triage New bug, unverified

Comments

@tacaswell
Copy link

Required prerequisites

Problem description

Trying to build scipy with the current CPython default branch results in failures that look like:

<snip>  
  creating build/temp.linux-x86_64-3.11/scipy/fft
  creating build/temp.linux-x86_64-3.11/scipy/fft/_pocketfft
  compile options: '-DPOCKETFFT_PTHREADS -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.11/numpy/distutils/include -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c'

 extra options: '-std=c++14 -pthread -fvisibility=hidden -msse -msse2 -msse3'
  g++: scipy/fft/_pocketfft/pypocketfft.cxx
  In file included from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/cast.h:16,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/attr.h:13,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                   from scipy/fft/_pocketfft/pypocketfft.cxx:15:
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/type_caster_base.h: In function ‘std::string pybind11::detail::error_string()’:
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/type_caster_base.h:473:38: error: ‘PyFrameObject’ {aka ‘struct _frame’} has no member named ‘f_code’
    473 |                 "  " + handle(frame->f_code->co_filename).cast<std::string>() +
        |                                      ^~~~~~
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/type_caster_base.h:473:75: error: expected primary-expression before ‘>’ token
    473 |                 "  " + handle(frame->f_code->co_filename).cast<std::string>() +
        |                                                                           ^
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/type_caster_base.h:473:77: error: expected primary-expression before ‘)’ token
    473 |                 "  " + handle(frame->f_code->co_filename).cast<std::string>() +
        |                                                                             ^
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/type_caster_base.h:475:31: error: ‘PyFrameObject’ {aka ‘struct _frame’} has no member named ‘f_code’
    475 |                 handle(frame->f_code->co_name).cast<std::string>() + "\n";
        |                               ^~~~~~
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/type_caster_base.h:475:64: error: expected primary-expression before ‘>’ token
    475 |                 handle(frame->f_code->co_name).cast<std::string>() + "\n";
        |                                                                ^
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/type_caster_base.h:475:66: error: expected primary-expression before ‘)’ token
    475 |                 handle(frame->f_code->co_name).cast<std::string>() + "\n";
        |                                                                  ^
  In file included from scipy/fft/_pocketfft/pypocketfft.cxx:15:
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h: In function ‘pybind11::function pybind11::detail::get_type_override(const void*, const pybind11::detail::type_info*, const char*)’:
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:2338:49: error: cannot convert ‘_interpreter_frame*’ to ‘PyFrameObject*’ {aka ‘_frame*’} in initialization
   2338 |     PyFrameObject *frame = PyThreadState_Get()->frame;
        |                            ~~~~~~~~~~~~~~~~~~~~~^~~~~
        |                                                 |
        |                                                 _interpreter_frame*
  In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/pystate.h:135,
                   from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:75,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/common.h:186,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pytypes.h:12,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/cast.h:13,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/attr.h:13,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                   from scipy/fft/_pocketfft/pypocketfft.cxx:15:
  /home/tcaswell/.pybuild/bleeding/include/python3.11/cpython/pystate.h:81:12: note: class type ‘_interpreter_frame’ is incomplete
     81 |     struct _interpreter_frame *frame;
        |            ^~~~~~~~~~~~~~~~~~
  In file included from scipy/fft/_pocketfft/pypocketfft.cxx:15:
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:2339:54: error: ‘PyFrameObject’ {aka ‘struct _frame’} has no member named ‘f_code’
   2339 |     if (frame != nullptr && (std::string) str(frame->f_code->co_name) == name
        |                                                      ^~~~~~
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:2340:19: error: ‘PyFrameObject’ {aka ‘struct _frame’} has no member named ‘f_code’
   2340 |         && frame->f_code->co_argcount > 0) {
        |                   ^~~~~~
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:2343:20: error: ‘PyFrameObject’ {aka ‘struct _frame’} has no member named ‘f_locals’
   2343 |             frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0));
        |                    ^~~~~~~~
  In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/tupleobject.h:39,
                   from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:57,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/common.h:186,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pytypes.h:12,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/cast.h:13,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/attr.h:13,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                   from scipy/fft/_pocketfft/pypocketfft.cxx:15:
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:2343:54: error: ‘PyFrameObject’ {aka ‘struct _frame’} has no member named ‘f_code’
   2343 |             frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0));
        |                                                      ^~~~~~
  /home/tcaswell/.pybuild/bleeding/include/python3.11/cpython/tupleobject.h:19:73: note: in definition of macro ‘_PyTuple_CAST’
     19 | #define _PyTuple_CAST(op) (assert(PyTuple_Check(op)), (PyTupleObject *)(op))
        |                                                                         ^~
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:2343:30: note: in expansion of macro ‘PyTuple_GET_ITEM’
   2343 |             frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0));
        |                              ^~~~~~~~~~~~~~~~
  error: Command "g++ -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPOCKETFFT_PTHREADS -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.11/numpy/distutils/include -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c scipy/fft/_pocketfft/pypocketfft.cxx -o build/temp.linux-x86_64-3.11/scipy/fft/_pocketfft/pypocketfft.o -MMD -MF build/temp.linux-x86_64-3.11/scipy/fft/_pocketfft/pypocketfft.o.d -std=c++14 -pthread -fvisibility=hidden -msse -msse2 -msse3" failed with exit status 1

  ########### EXT COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt.cache_flush[822] : write cache to path -> /home/tcaswell/source/other_source/scipy/build/temp.linux-x86_64-3.11/ccompiler_opt_cache_ext.py

  ########### CLIB COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt.cache_flush[822] : write cache to path -> /home/tcaswell/source/other_source/scipy/build/temp.linux-x86_64-3.11/ccompiler_opt_cache_clib.py
  Building wheel for scipy (pyproject.toml) ... error
  ERROR: Failed building wheel for scipy


Unfortunately I can only reproduce this with the current default branch of CPython. I suspect that this is due to upstream changes to the frame structure, but my (cursory) attempts to track down the changes did not lead anywhere, however I know cython had similar issues. My intuition as that this is fallout from the fast-python work.

Reproducible example code

No response

@tacaswell tacaswell added the triage New bug, unverified label Oct 26, 2021
@henryiii
Copy link
Collaborator

Those have been fixed in #3368, probably to be released in a day or so (and in master), though our tests still have an uninvestigated segfault on 3.11 dev builds.

@henryiii
Copy link
Collaborator

Oh, and your intuition is correct; an API for accessing this was added in 3.9, and now the newly "private" internals are finally changing for performance. We just moved to the new API for 3.9+.

@tacaswell
Copy link
Author

Great!

Sorry I did not find #3368, I searched issues for py311 but not PRs 🤦🏻 .

Both you and cython have reports of segfaults with the default CPython branch. My day-to-day environment is going to be fun for a while :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
triage New bug, unverified
Projects
None yet
Development

No branches or pull requests

3 participants