-
Notifications
You must be signed in to change notification settings - Fork 819
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support writing FixedSizeList to Parquet #4214
Comments
I think this should be relatively straightforward, as far as anything involving dremel is straightforward. This series of posts may be helpful for whoever picks this one up - https://arrow.apache.org/blog/2022/10/05/arrow-parquet-encoding-part-1/ |
@tustvold I'd be happy to take over the implementation for this (assuming you're not already working on it). I believe I have a working proof of concept, I just need to integrate it into the codebase and add tests. |
Always happy to review PRs, FWIW I think the hard part is the read side |
|
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Have array data which are all with fixed size, when I tried to write the
RecordBatch
with FixedSizeList column, it failed and reported:Describe the solution you'd like
I have searched this error and found at:
arrow-rs/parquet/src/arrow/arrow_writer/levels.rs
Line 161 in 0190408
It looks like not hard to support
FixedSizeList
just likeList
(not sure), the only thing need to do is constructing theLevelInfoBuilder
forFixedSizeList
.Describe alternatives you've considered
No
Additional context
I'm happy to take it if it's simple as I described above
The text was updated successfully, but these errors were encountered: