Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ArrayData buffers are inconsistent accross implementations #207

Closed
alamb opened this issue Apr 26, 2021 · 1 comment
Closed

ArrayData buffers are inconsistent accross implementations #207

alamb opened this issue Apr 26, 2021 · 1 comment

Comments

@alamb
Copy link
Contributor

alamb commented Apr 26, 2021

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-12223

ArrayData implementations seems to share close structure fields accross languages, but their usage is not consistent accross implementation.

 

Example using ListArray's offsets buffer, in C++, Rust and JavaScript implementation:

 - C++: offset's buffer is the second buffer (validity bitmap is first buffer, and buffers are laid in a type-dependant way) https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/array_nested.cc#L189

 - Rust: offset's buffer is the first buffer (validity bitmap is not part of the collection, and buffers are laid in a type-dependant way) https://github.com/apache/arrow/blob/master/rust/arrow/src/array/array_list.rs#L235

 - JavaScript: offset's buffer is the first buffer (they have fixed position) [https://github.com/apache/arrow/blob/8e43f23dcc6a9e630516228f110c48b64d13cec6/js/src/data.ts#L125]

 

Note that we have the same inconsistency for validity and data buffers.

 

This is important in my project because I would like to transport buffers list accross technologies, and ArrayData seemed the easiest structure to transport.

@alamb alamb added the arrow Changes to the arrow crate label Apr 26, 2021
@jorgecarleitao
Copy link
Member

I will close this as won't fix since this can be accomplished via C data interface or IPC are the mechanisms to share pointers between implementations.

@jorgecarleitao jorgecarleitao removed the arrow Changes to the arrow crate label Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants