Add function for row alignment with page mask #1790
Labels
enhancement
Any new improvement worthy of a entry in the changelog
parquet
Changes to the parquet crate
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
For now row group filter in datafusion pass a closure to arrow-rs
https://github.com/apache/arrow-datafusion/blob/585bc3a629b92ea7a86ebfe8bf762dbef4155710/datafusion/core/src/physical_plan/file_format/parquet.rs#L559-L562
So for page filter in datafusion, define filter_predicate
datafusion will send a
mask
(&[bool]) to arrow-rs,then use mask call
compute_row_ranges
to constructRowRanges
: row ranges in a row-group (one col) if col is sorted vec size will be 1.For multi filter combine:
if there are two filters use
and
connect,useRowRanges::intersection
to get the final rowRange; two filters useor
connect,useRowRanges::union
to get the final rowRange.Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: