Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not matching column names when creating a RecordBatch from MapArray #1257

Closed
balbok0 opened this issue Apr 3, 2023 · 0 comments · Fixed by #1258
Closed

Not matching column names when creating a RecordBatch from MapArray #1257

balbok0 opened this issue Apr 3, 2023 · 0 comments · Fixed by #1258
Labels
bug Something isn't working

Comments

@balbok0
Copy link
Contributor

balbok0 commented Apr 3, 2023 •
edited
Loading

Environment

Delta-rs version: 0.8.0

Binding: Rust

Environment:

Bug

What happened: When trying to create a RecordBatch with MapArray, I get the following error:

thread 'main' panicked at 'Failed to create RecordBatch: InvalidArgumentError("column types must match schema types, expected Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Binary, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) but found Map(Field { name: \"entries\", data_type: Struct([Field { name: \"keys\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\", data_type: Binary, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) at column index 0")', src/main.rs:66:90

What you expected to happen: Successful creation of RecordBatch without any errors.

How to reproduce it: Create a MapArray and RecordBatch with it (See rust explorer link above).

More details:
I believe these lines are at fault. https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_arrow.rs#L145-L161
Hopefully I will be able to test this hypothesis and make a PR in few hours.
@balbok0 balbok0 added the bug Something isn't working label Apr 3, 2023
@balbok0 balbok0 mentioned this issue Apr 3, 2023
Merged
wjones127 pushed a commit that referenced this issue Apr 9, 2023
@balbok0
Fix names and nullability when creating RecordBatch from MapArray (#1258
2f998ea 
)

# Description
When creating a RecordBatch with one of the columns being a MapArray,
there are issues with naming of sub-columns, as well as nullability of
the "elements/key_value" sub-column.

# Related Issue(s)

- closes #1257

# Documentation

See https://www.rustexplorer.com/b/o7bfm4 for a breaking example.
I initially thought that the issue is specifically for array with
0-length maps, but `nullable` mismatch happens even when all of the map
elements have values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix names and nullability when creating RecordBatch from MapArray balbok0/delta-rs
1 participant
@balbok0