You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, many of the methods of pyarrow Array/ChunkedArray/RecordBatch/Table crash when the object is backed by buffers pointing to non-CPU data (i.e. all methods that actually do something with the data and not just return info like the schema or the length).
We should ensure users don't see segfaults but get a proper error about the method only being support for data on the CPU. Probably that will require adding some "ensure cpu" helper function to most methods in the Python bindings.
This helper function will then have to inspect the device of the buffers and ensure all of them are on the CPU (not sure if there already exists a helper for this on the C++ side?)
This helper function will then have to inspect the device of the buffers and ensure all of them are on the CPU (not sure if there already exists a helper for this on the C++ side?)
In bridge.cc, we already have ValidateDeviceInfo which loops recursively over all buffers of the array and its children, but that is C Data interface specific (the error messages). Similar logic can be added to the main ArrayData code as well, which is already being done in #40807 (which is adding a device_type property on ArrayData)
Currently, many of the methods of pyarrow
Array
/ChunkedArray
/RecordBatch
/Table
crash when the object is backed by buffers pointing to non-CPU data (i.e. all methods that actually do something with the data and not just return info like the schema or the length).We should ensure users don't see segfaults but get a proper error about the method only being support for data on the CPU. Probably that will require adding some "ensure cpu" helper function to most methods in the Python bindings.
This helper function will then have to inspect the device of the buffers and ensure all of them are on the CPU (not sure if there already exists a helper for this on the C++ side?)
The text was updated successfully, but these errors were encountered: