Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DictionaryHandling does not recurse into Map fields #6644

Closed
nathanielc opened this issue Oct 29, 2024 · 2 comments · Fixed by #6645
Closed

DictionaryHandling does not recurse into Map fields #6644

nathanielc opened this issue Oct 29, 2024 · 2 comments · Fixed by #6645
Labels
arrow Changes to the arrow crate arrow-flight Changes to the arrow-flight crate bug

Comments

@nathanielc
Copy link
Contributor

nathanielc commented Oct 29, 2024 •
edited
Loading

Describe the bug

When using the FlightDataEncoder the DictionaryHandling logic does not recuse into fields of a Map type. As a result any dictionary within a Map is not hydrated and can therefore break encoding as multiple dictionaries are sent.

To Reproduce

  1. Using FlightDataEncoder encode two batches where there is a Map with a dictionary for either the key or value field.
  2. An error occurs as the error_on_replacement check inside the FlightIPCEncoder fails as a dictionary is replaced

Expected behavior

The FlightDataEncoder correctly hydrates the dictionaries to avoid the encoding error

Additional context

PR with a fix incoming.

The error string

Dictionary replacement detected when writing IPC file format. Arrow IPC files only support a single dictionary for a given field across all batches
@nathanielc nathanielc added the bug label Oct 29, 2024
nathanielc added a commit to nathanielc/arrow-rs that referenced this issue Oct 29, 2024
@nathanielc
fix: recurse into Map datatype when hydrating dictionaries
2b7e126 
When hydrating dictionaries for the FlightDataEncoder the Map data type
was assumed to not be nested. This change correctly recurses into the
Map field to hydrate any dictionaries within the map.

Fixes apache#6644
@nathanielc nathanielc mentioned this issue Oct 29, 2024
Merged
adbmal pushed a commit to adbmal/arrow-rs that referenced this issue Nov 2, 2024
@nathanielc @adbmal
fix: recurse into Map datatype when hydrating dictionaries (apache#6645)
70449e9 
When hydrating dictionaries for the FlightDataEncoder the Map data type
was assumed to not be nested. This change correctly recurses into the
Map field to hydrate any dictionaries within the map.

Fixes apache#6644
@alamb alamb added the arrow Changes to the arrow crate label Nov 16, 2024
@alamb
Copy link
Contributor

alamb commented Nov 16, 2024

label_issue.py automatically added labels {'arrow'} from #6645

@alamb alamb added the arrow-flight Changes to the arrow-flight crate label Nov 16, 2024
@alamb
Copy link
Contributor

alamb commented Nov 16, 2024

label_issue.py automatically added labels {'arrow-flight'} from #6645

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arrow Changes to the arrow crate arrow-flight Changes to the arrow-flight crate bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: recurse into Map datatype when hydrating dictionaries nathanielc/arrow-rs
2 participants
@alamb @nathanielc