Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Array Destructuring APIs #3879

Closed
tustvold opened this issue Mar 17, 2023 · 3 comments · Fixed by #4263
Closed

Array Destructuring APIs #3879

tustvold opened this issue Mar 17, 2023 · 3 comments · Fixed by #4263
Assignees
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

As part of #1799 strongly typed ArrayData abstractions were added that allow constructing arrays from their constituent parts, and destructuring arrays back into their parts. Whilst it looks like we won't be moving ahead with a strongly typed ArrayData abstraction, these APIs are powerful and should be ported across to the corresponding Array implementation.

Describe the solution you'd like

Each of the various Array should have:

  • An unsafe new_unchecked that constructs it from raw ScalarBuffer, OffsetBuffer, etc...
  • A safe try_new method that performs validation and returns a Result
  • An into_parts method that returns the constituent parts that can be fed to new_unchecked
  • Methods should return &ScalarBuffer<T>, &BooleanBuffer, etc... instead of &[T], &[u8] etc...

Describe alternatives you've considered

Additional context

@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label Mar 17, 2023
@tustvold tustvold self-assigned this Mar 17, 2023
@jhorstmann
Copy link
Contributor

Would the into_parts method consume the array and return owned buffers? I think this could avoid some Arc clones for commons usecases.

tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
tustvold added a commit that referenced this issue Mar 21, 2023
* Use BooleanBuffer in BooleanArray (#3879)

* Review feedback
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
tustvold added a commit that referenced this issue Mar 22, 2023
* Return ScalarBuffer from PrimitiveArray::values (#3879)

* Fix docs

* Review feedback
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 22, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 22, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 22, 2023
tustvold added a commit that referenced this issue Mar 23, 2023
* Add PrimitiveArray::new (#3879)

* Review feedback

* Format
tustvold added a commit that referenced this issue Mar 23, 2023
* Add BooleanArray::new (#3879)

* Review feedback
spebern pushed a commit to spebern/arrow-rs that referenced this issue Mar 25, 2023
* Use BooleanBuffer in BooleanArray (apache#3879)

* Review feedback
spebern pushed a commit to spebern/arrow-rs that referenced this issue Mar 25, 2023
…#3896)

* Return ScalarBuffer from PrimitiveArray::values (apache#3879)

* Fix docs

* Review feedback
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 11, 2023
tustvold added a commit that referenced this issue Apr 11, 2023
* Add offsets accessors to variable length arrays (#3879)

* Review feedback
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 13, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 13, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 13, 2023
tustvold added a commit that referenced this issue Apr 16, 2023
* Add ListArray constructors (#3879)

* More cleanup

* Checked arithmetic

* Add try_new

* Add tests

* Clippy

* Update arrow-array/src/array/list_array.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 18, 2023
tustvold added a commit that referenced this issue Apr 18, 2023
* Add PrimitiveArray::try_new (#3879)

* Add tests

* Review feedback
tustvold added a commit that referenced this issue Apr 19, 2023
* Add ByteArray constructors (#3879)

* Clippy

* Make ListArray error message consistent

* Review feedback
tustvold added a commit that referenced this issue Apr 25, 2023
* Add StructArray Constructors (#3879)

* Fix doc

* Add try_new

* Update other constructors
@alamb
Copy link
Contributor

alamb commented May 2, 2023

I wonder if this issue is complete now

@tustvold
Copy link
Contributor Author

tustvold commented May 2, 2023

I think it is just some of the more esoteric arrays that need constructors, like FixedSizeBinary. I will try to find time to get this over the line soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants