Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Avoid materialization of indices in filter_record_batch for single arrays #234

Closed
Dandandan opened this issue Jul 29, 2021 · 0 comments · Fixed by #235
Closed

Avoid materialization of indices in filter_record_batch for single arrays #234

Dandandan opened this issue Jul 29, 2021 · 0 comments · Fixed by #235
Labels
enhancement An improvement to an existing feature

Comments

@Dandandan
Copy link
Collaborator

Dandandan commented Jul 29, 2021 •
edited
Loading

Some optimization possible in the kernel seems to avoid materializing (start, end) and pre-building the filter when there is only a single input array. This probably will be a time/memory usage win.

This is a common situation in DataFusion / SQL (select a from x where [...]).
@Dandandan Dandandan changed the title Avoid materialization of indices in filter for single arrays Avoid materialization of indices in filter_record_batch for single arrays Jul 29, 2021
@jorgecarleitao jorgecarleitao added the enhancement An improvement to an existing feature label Jul 29, 2021
@Dandandan Dandandan mentioned this issue Jul 29, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
enhancement An improvement to an existing feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Special-case filter single array record batch in filter_record_batch Dandandan/arrow2
2 participants
@Dandandan @jorgecarleitao