PyPy3 5.8.0 failures --

[DavidCallahan]

Issue description

test failures with PyPy 5.8.0

Reproducible example code

Using a recent "portable" PyPy distribution (https://github.com/squeaky-pl/portable-pypy)

Python 3.5.3 (a37ecfe5f142bc971a86d17305cc5d1d70abec64, Jun 08 2017, 19:43:54)
[PyPy 5.8.0-beta0 with GCC 6.3.0]

Running the tests fails with this error (which also happens in the 2.1 branch)

/home/dcallahan/projects/pybind11/include/pybind11/class_support.h: In function ‘PyObject* pybind11::detail::make_new_python_type(const pybind11::detail::type_record&)’:
/home/dcallahan/projects/pybind11/include/pybind11/class_support.h:555:16: error: ‘struct _heaptypeobject’ has no member named ‘ht_qualname’
     heap_type->ht_qualname = ht_qualname.release().ptr();

PyPy does not define this field. (from cpyext_opbject.h) (which is also true in their source)

typedef struct _heaptypeobject {
    PyTypeObject ht_type;
    PyNumberMethods as_number;
    PyMappingMethods as_mapping;
    PySequenceMethods as_sequence;
    PyBufferProcs as_buffer;
    PyObject *ht_name, *ht_slots;
} PyHeapTypeObject;

[jagerman]

Currently only pypy2 is supported. We'd accept a PR to make pypy3 work, but I suspect that such a PR would not be trivial.

[DavidCallahan]

Thanks for the response Jason. Should I leave the issue open or close it for now?

[jagerman]

I'm going to close it but with a comment that anyone who wants to undertake this is welcome to give it a shot (and should feel free to reopen).

I did try a few months ago, and adding support was non-trivial: some C API functions that we rely on appear to be missing from PyPy3 entirely (though they are present in PyPy2), so I suspect it would take a not insignificant amount of effort to write the initial support. (The original PyPy2 support (#521, #527) was itself not a trivial addition).

[standy66]

With a little modification on PyPy3 v5.10.1 version I was able to successfully compile and run all but the following 3 tests from the pybind11 test suite:

• test_multiple_inheritance fails with TypeError: object() takes no parameters on objects inherited from both python object and pybind11-objects
• test_qualname fails on assert m.NestBase.Nested.__qualname__ == "NestBase.Nested" with AssertionError: assert 'Nested' == 'NestBase.Nested'
• test_eval does not compile

This makes the library useful in almost any imaginable scenario under PyPy. I am working on integrating the changes into upstream repo. For further discussion please visit PyPy issue.

[jagerman]

That's great work. Is there a pybind PR associated with it incoming? (From what I remember, some of the ifdefs implicitly assume PYPY means Python 2 API and needed updating).

The test_multiple_inheritance issue is, I believe, an upstream PyPy issue that affects both PyPy 2 and 3: https://bitbucket.org/pypy/pypy/issues/2482. (As I don't actually use PyPy, getting it fixed hasn't been a huge priority for me). I believe the current test suite is broken under newer PyPy2 versions than we use in the travis-ci build for this very reason.

It's not ideal, but I think acceptable for now to just disable the Python-side multiple inheritance tests, with a note in the documentation that Python-side multiple inheritance isn't officially supported under PyPy. C++-side multiple inheritance should be fine. (This wouldn't be the only thing that isn't supported under PyPy).

[standy66]

@jagerman My pull request in PyPy 3.5 branch was merged today and the changes will appear in the next release (that's pypy3-v5.11 I think).

In my case, C++ code from pybind11 master branch didn't require any modifications to work with PyPy3, so I am not sure about ifdefs, but if someone familiar with the matter clarified, it would be great!

As I said, there are couple methods missing in PyPy's cpyext that are required for test_eval.cpp to compile (here is compilation log), presumably those are PyRun_FileEx and _Py_fopen_obj. Also, test_multiple_inheritance is not the only issue, there is a strange bug with qualname, where m.NestBase.Nested.__qualname__ does not equal to "NestBase.Nested", but instead equals to "Nested", although I am not sure why. Both of this issues are not severe and most likely will never a serious obstacle to using pybind11 with PyPy3. Anyways, here is pytest log (with test_eval disabled because of compilation error).

Also, I think this issue may be renamed to something like PyPy3 compatibility to track the progress on that.

[jagerman]

The __qualname__ issue is probably a result if the ifdefs: I think we're doing something in the Python 3 path that we shouldn't be under PyPy3. I just pushed the local work I'd done a few months ago to here: https://github.com/jagerman/pybind11/tree/pypy3

Feel free to steal that and update/change it as necessary (it may need some updating to apply cleanly to current master).

[perkinslr]

Using pypy3 6.0, I got similar results. I patched eval.h to get it to compile, it's not quite as simple as helping it realise pypy -!> py2, since there's a couple spots where pypy3 does its own thing (no way to convert PyFile to FILE*, and no _Py_fopen, so I used the raw fopen).

The module compiles, and the associated test passes. I don't know enough about the internal workings to guarantee there's not something subtly broken, possibly with the way unicode is handled, but it appears to work. I've included the 10 line diff here, or I can open a pull request if you prefer.

eval.diff

[bstaletic]

Is this thing resolved with #2146 ?

[YannickJadoul]

Let's say it is until we hear it isn't. We're at PyPy 7.3, by now, so 5.8 seems quite old.