[QUESTION] Proper way to handle std::span<T>-like types

[psalvaggio]

I have a C++11 backported std::span<T> type in my code that I am trying to bind into Python and was unsure about the proper way to do this, as there is not built-in conversion for span to use as an example.

I have had success using NumPy for span's of arithmetic types using the following:

template <typename T, size_t N>
pybind11::array_t<T> NumpyArray(const span<T, N>& s, pybind11::object& obj) {
  return pybind11::array_t<T>(s.size(), s.data(), obj);
}

however, when I try this with a struct type, I get the following error:

RuntimeError: NumPy type info missing for <MangledTypeName>

So, my question is, what is the proper way to handle binding such a span of objects? Is it using this NumPy approach and figuring out a way to register my type (which is already bound via py::class_) or is there another way to handle this?

I also tried using memoryview, but I was unable to figure out the format string, as the docs say to just use format_descriptor<T>::value, but I got an error about this being undefined.

Thanks,
Phil

[YannickJadoul]

@psalvaggio What would you expect the corresponding Python type to be? Indeed, memoryview and NumPy's ndarray match this, but for general types, I'm not sure Python has a "view of another range" type?

Do note that NumPy & pybind11 support custom struct-of-primitive-types dtypes: https://pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html#structured-types

[psalvaggio]

@YannickJadoul yes, I gave the NumPy structured types a try and it does work, although I do wind up losing the field names using that approach. I'd really like a NumPy array or something similar, but instead of getting numpy.void as the element type, I'd like them to be the bound type for my structs, so that I could use the field names. It sounds like that might not be possible?

[YannickJadoul]

@psalvaggio I don't understand. Normally, PYBIND11_NUMPY_DTYPE supports field names? Can you be more precise and show what you have?

[psalvaggio]

@YannickJadoul yes, PYBIND11_NUMPY_DTYPE takes in the field names. I think my interpretation of how to use them was wrong, however. Here's essentially what I am trying to do (taking out all of my app-specific stuff)...

So I have the following structs:

template <typename T> 
struct Vector3 {
  T x, y, z;
};

struct ElementType {
  double time;
  Vector3<double> vec;
};

I have a method which returns a span<ElementType>, which is a view to one of that object's internal buffers. I would like to bind this method. So, I was using the following binding code:

  auto subM = m.def_submodule("submodule");
  auto subsubM = subM.def_submodule("subsubmodule");

  py::class_<Vector3<double>>(subM, "Vector3d", "Vector of 3 doubles")
      .def_readwrite("x", &Vector3<double>::x)
      .def_readwrite("y", &Vector3<double>::y)
      .def_readwrite("z", &Vector3<double>::z);
  PYBIND11_NUMPY_DTYPE(Vector3<double>, x, y, z);

  py::class_<ElementType>(subsubM, "ElementType")
      .def_readwrite("time", &ElementType::time)
      .def_readwrite("vec", &ElementType::vec);
  PYBIND11_NUMPY_DTYPE(ElementType, time, vec);

then, when I'm binding my method:

myClass.def(
    "method",
    [](py::object& obj, size_t index) {
      auto& o = obj.cast<MyClass&>();
      auto mySpan = o.method(index);
      return pybind11::array_t<ElementType>(mySpan.size(), mySpan.data(), obj);
    });

An example I get out of this is:

>>> obj.method(0)
array([(0., (-120., 0., 100.)), (1., ( 120., 0., 100.))],
      dtype=[('time', '<f8'), ('vec', [('x', '<f8'), ('y', '<f8'), ('z', '<f8')])])

I was expecting to be able to get that first 0 via obj.method(0)[0].time, but it appears I have to use obj.method(0)[0]["time"] instead. A bit clunky, but it should work. I'd like to be able to remap the names (we use camelCase in C++ and snake_case in Python), but I can live without that.

[YannickJadoul]

Isn't that just how NumPy's structured arrays work? That's nothing pybind11 can do anything about?

What you could also do is return some kind of a list of py::cast(some_object_ptr, py::return_value_policy::reference), but it won't be the same kind of span, because it'll copy the Python objects to a Python list. So if you want that, you'll probably have to implement your own span-like type, and have __getitem__/__setitem__/... on it. It should be possible, though.

PYBIND11_NUMPY_DTYPE_EX exists (badly documented, but the tests demonstrate), if you want to rename fields.

[psalvaggio]

Yes, that appears to be an issue on NumPy's end rather than on pybind11's end. I'll look into PYBIND11_NUMPY_DTYPE_EX. Thanks for the help!

[YannickJadoul]

OK, great. Let's close this then.