Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Arrow PyCapsule Interface #5070

Merged
merged 28 commits into from
Nov 15, 2023
Merged
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
b1c43a4
arrow ffi array copy
kylebarron Nov 13, 2023
c61d270
remove copy_ffi_array
kylebarron Nov 13, 2023
6521cba
docstring
kylebarron Nov 13, 2023
92b070a
wip: pycapsule support
kylebarron Nov 13, 2023
6460701
return
kylebarron Nov 13, 2023
dfdcfae
Update arrow/src/pyarrow.rs
kylebarron Nov 13, 2023
5a4f738
remove sync impl
kylebarron Nov 13, 2023
8a1a05e
Update arrow/src/pyarrow.rs
kylebarron Nov 13, 2023
e109c1a
Remove copy()
kylebarron Nov 13, 2023
3b95ebc
Merge branch 'kyle/arrow-ffi-copy' of github.com:kylebarron/arrow-rs …
kylebarron Nov 13, 2023
05ea67d
Need &mut FFI_ArrowArray for std::mem::replace
kylebarron Nov 13, 2023
e7ed58d
Use std::ptr::replace
kylebarron Nov 13, 2023
dc04b13
update comments
kylebarron Nov 13, 2023
86918fa
Minimize unsafe block
kylebarron Nov 13, 2023
0e273a3
revert pub release functions
kylebarron Nov 13, 2023
252e746
Add RecordBatch and Stream conversion
kylebarron Nov 14, 2023
60bee4a
fix returns
kylebarron Nov 14, 2023
46612ce
Fix return type
kylebarron Nov 14, 2023
becda12
Fix name
kylebarron Nov 14, 2023
2f7767b
fix ci
kylebarron Nov 14, 2023
1e7bcd3
Add tests
kylebarron Nov 15, 2023
ae909fb
Add table test
kylebarron Nov 15, 2023
6f01c91
skip if pre pyarrow 14
kylebarron Nov 15, 2023
f183057
bump python version in CI to use pyarrow 14
kylebarron Nov 15, 2023
107acef
Add record batch test
kylebarron Nov 15, 2023
6c44e01
Update arrow/src/pyarrow.rs
kylebarron Nov 15, 2023
6020247
run on pyarrow 13 and 14
kylebarron Nov 15, 2023
2e42926
Update .github/workflows/integration.yml
kylebarron Nov 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions arrow/src/pyarrow.rs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential refactor for later, but it would be nice to move the Capsule imports to their own functions that aren't named as if to by PyArrow-specific. Right now this module is named for PyArrow, but the import logic will apply to other Arrow implementations as well.

Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
//! [pyarrow.Table.to_reader()](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_reader)
//! and then importing the reader as a [ArrowArrayStreamReader].

use std::borrow::BorrowMut;
use std::convert::{From, TryFrom};
use std::ptr::{addr_of, addr_of_mut};
use std::sync::Arc;
Expand Down Expand Up @@ -262,9 +263,10 @@ impl FromPyArrow for ArrayData {
validate_pycapsule(array_capsule, "arrow_array")?;

let schema_ptr = unsafe { schema_capsule.reference::<FFI_ArrowSchema>() };
let array_ptr = unsafe { array_capsule.reference::<FFI_ArrowArray>() };
let array_ptr = array_capsule.pointer() as *mut FFI_ArrowArray;
let array_mut = unsafe { array_ptr.as_mut() };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW you could use https://doc.rust-lang.org/std/ptr/fn.replace.html as we're already playing with pointers here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in e7ed58d; is that what you meant?


let array = std::mem::replace(array_ptr, FFI_ArrowArray::empty());
let array = std::mem::replace(array_mut.unwrap(), FFI_ArrowArray::empty());
return ffi::from_ffi(array, schema_ptr).map_err(to_py_err);
}

Expand Down
Loading