Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug/Question: arrow'sFixedSizeList is not roundtrippable #2162

Closed
universalmind303 opened this issue Feb 3, 2024 · 1 comment
Closed

Bug/Question: arrow'sFixedSizeList is not roundtrippable #2162

universalmind303 opened this issue Feb 3, 2024 · 1 comment
Labels
question Further information is requested

Comments

@universalmind303
Copy link
Contributor

Bug

What happened:

Currently if you try to write an arrow FixedSizeList using delta, the original type is converted to a list. I'm wondering if this is a bug or intentional. Are there any suggested workarounds for working with arrow FixedSizeList without losing the fixed size?

What you expected to happen:
I'd expect being able to roundtrip arrow's FixedSizeList to/from delta without losing the original type.

How to reproduce it:

More details:
@universalmind303 universalmind303 added the bug Something isn't working label Feb 3, 2024
@ion-elgreco
Copy link
Collaborator

Intentional, the delta primitive types doesn't have a fixedsizedlist/array, only list

@ion-elgreco ion-elgreco added question Further information is requested and removed bug Something isn't working labels Feb 5, 2024
@universalmind303 universalmind303 closed this as not planned Won't fix, can't repro, duplicate, stale Feb 7, 2024
@universalmind303 universalmind303 mentioned this issue Feb 7, 2024
Merged
universalmind303 added a commit to GlareDB/glaredb that referenced this issue Feb 7, 2024
@universalmind303
fix: fixedsizelist in delta (#2607)
3ec75e6 
fixes the issue of delta converting fsl -> list when writing delta
tables.

This PR adds some metadata containing the original field if it was
downcast when writing to delta, then compares those fields. If they are
different, it applies a projection from the delta field to the original
field.

If we need further performance, we'll likely have to read and write
directly instead of using delta's arrow/datafusion logic as they have
[no intention](delta-io/delta-rs#2162) on
supporting FixedSizeList.
@universalmind303 universalmind303 mentioned this issue Mar 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ion-elgreco @universalmind303