-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
Now we don't support pushdown nested datatype pushdown to parquet:
fn check_single_column(&mut self, column_name: &str) -> Option<TreeNodeRecursion> {
if let Ok(idx) = self.table_schema.index_of(column_name) {
self.required_columns.insert(idx);
if DataType::is_nested(self.table_schema.field(idx).data_type()) {
self.non_primitive_columns = true;
return Some(TreeNodeRecursion::Jump);
}
} else {
// If the column does not exist in the (un-projected) table schema then
// it must be a projected column.
self.projected_columns = true;
return Some(TreeNodeRecursion::Jump);
}
None
}But i think, some nest datatypes will be useful if we support pushdown, for example, the list column a, here is the case:
array_has_all(a, ['c'])We can pushdown to parquet, so arrow-rs side can filter it? But i am not sure if arrow-rs support it.
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request