Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] Ensure (Chunked)Array/RecordBatch/Table methods don't crash with non-CPU data #41665

Closed
4 tasks done
jorisvandenbossche opened this issue May 15, 2024 · 2 comments
Closed
4 tasks done

Comments

@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented May 15, 2024

Currently, many of the methods of pyarrow Array/ChunkedArray/RecordBatch/Table crash when the object is backed by buffers pointing to non-CPU data (i.e. all methods that actually do something with the data and not just return info like the schema or the length).

We should ensure users don't see segfaults but get a proper error about the method only being support for data on the CPU. Probably that will require adding some "ensure cpu" helper function to most methods in the Python bindings.
This helper function will then have to inspect the device of the buffers and ensure all of them are on the CPU (not sure if there already exists a helper for this on the C++ side?)

@jorisvandenbossche
Copy link
Member Author

This helper function will then have to inspect the device of the buffers and ensure all of them are on the CPU (not sure if there already exists a helper for this on the C++ side?)

In bridge.cc, we already have ValidateDeviceInfo which loops recursively over all buffers of the array and its children, but that is C Data interface specific (the error messages). Similar logic can be added to the main ArrayData code as well, which is already being done in #40807 (which is adding a device_type property on ArrayData)

@jorisvandenbossche
Copy link
Member Author

Closing this issue, with all subtasks being completed.

Another related issue, to do the same for compute functions, is still open -> #43541

@jorisvandenbossche jorisvandenbossche added this to the 18.0.0 milestone Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant