Support to read/write parquet for FixedSizeList type #4226
Conversation
github-actions
bot
added
arrow
Signed-off-by: yah01 <yah2er0ne@outlook.com>
| Ok(Arc::new(result_array)) | ||
| let result_array: ArrayRef = match *list_data.data_type() { | ||
| ArrowType::FixedSizeList(_, _) => { | ||
| Arc::new(FixedSizeListArray::from(list_data)) |
There was a problem hiding this comment.
This is not correct, you are constructing a FixedSizeListArray from an ArrayData that contains an offsets buffer. In particular this code needs to verify that the values array is the right length. I'll see if I can't finish up #3879 so that you can make use of that
There was a problem hiding this comment.
This is actually significantly more complicated, as the null padding logic is different for fixed size types. I'll have a think about how this could be supported
There was a problem hiding this comment.
This is actually significantly more complicated, as the null padding logic is different for fixed size types. I'll have a think about how this could be supported
Thank you @tustvold, I know not much about arrow implementation, please let me know if it's too complex so that I can't handle
There was a problem hiding this comment.
Maybe you should padding some null data? As https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout
There was a problem hiding this comment.
I can add it to my list, its not as trivial as I had first thought 😅
Which issue does this PR close?
Closes #4214
Rationale for this change
As mentioned in the issue
What changes are included in this PR?
FixedSizeListFixedSizeListdata for ArrowWriterListArraytoFixedSizeListfor ArrowReaderAre there any user-facing changes?
No