Construct py::array without copy

[yangle]

Hi Wenzel,

I'm trying to construct a py::array from a bare pointer without copying data.

In #27 you mentioned that

... the memory is copied during the construction of array. The way to avoid this is by creating the array first and then writing the data directly to it ...

Could you please elaborate a bit on how to do this?

Also, it seems to me a simple alternative would be to optionally skip the PyArray_NewCopy call in the py::array constructor, something like

    template <typename Type> array(size_t size, const Type *ptr, bool copy = true) {
        ...
        if (copy && ptr && tmp)
            tmp = object(api.PyArray_NewCopy_(tmp.ptr(), -1 /* any order */), false);
        ...
    }

I tried this and it seems to work just fine. Am I missing something here?

Thanks!

[wjakob]

To make it truly safe (reference counting and all), you also need to inform PyArray about the object which owns the buffer. A way of directly doing this sort of stuff with pybind11 is in the works (#308) but may still take a bit.

[aldanor]

@wjakob @yangle This is on my next-to-do list for sure, but will take a bit of effort to do properly and will require a separate PR.

[LeslieGerman]

Hello @wjakob, can you please explain how the merged #308 resolved this?
I'v spent almost a day to figure out how make pybind11 array without copying the data, but still could not find the solution...
Can you please help?

[LeslieGerman]

Ok guys, after reading lots of SO articles, lots of pybind issues, PRs and discussions and after reading the code I figured it out:
I need to pass any valid pybind object as the base parameter to an array ctor, and then pybind won't copy...

HUH! 😄

Now my code looks like this:

void addInput(std::shared_ptr<const MyFancyMatrix> packet)
{
  // When a valid object is passed as 'base', it tells pybind not to take ownership of the data,
  // because 'base' will own it. In fact 'packet' will own it, but - psss! - , we don't tell it to
  // pybind... Alos note that ANY valid object is good for this purpose, so I chose "str"...
  py::str dummyDataOwner;
  py::array input{getNumpyDTypeFromElementType(packet->elementType()), packet->shape(),
                  packet->strides(), packet->data(), dummyDataOwner};

  assert(!input.owndata()); // if fires, s.g. is not OK with ownership (probably matrix data was copied)
  // ...
}

Awesome hacky undocumented stuff... 😁 🌟

[power1628]

it will be great if pybind11 could provide more documents on handling numpy arrays

[axsaucedo]

@LeslieGerman your approach worked perfectly - @power1628 I agree, documentation on this, even in the form of tests would be more reassuring as at least it's confirmed that this is a reasonable approach.

[icfaust]

void addInput(std::shared_ptr<const MyFancyMatrix> packet)
{
  // When a valid object is passed as 'base', it tells pybind not to take ownership of the data,
  // because 'base' will own it. In fact 'packet' will own it, but - psss! - , we don't tell it to
  // pybind... Alos note that ANY valid object is good for this purpose, so I chose "str"...
  py::str dummyDataOwner;
  py::array input{getNumpyDTypeFromElementType(packet->elementType()), packet->shape(),
                  packet->strides(), packet->data(), dummyDataOwner};

  assert(!input.owndata()); // if fires, s.g. is not OK with ownership (probably matrix data was copied)
  // ...
}

Wouldn't using py::none() instead of dummyDataOwner be more ideal? The base property of input would still point at dummyDataOwner, or am I mistaken?