Skip to content

Commit 6870cc1

Browse files
authored
Add PhysicalExpr optimizer and cast unwrapping (#16530)
* Add PhysicalExpr optimizer and cast unwrapping * address pr feedback * Update datafusion/pruning/src/pruning_predicate.rs * more lit(Xi64)
1 parent 705ea42 commit 6870cc1

File tree

11 files changed

+2101
-818
lines changed

11 files changed

+2101
-818
lines changed

Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

datafusion/datasource-parquet/src/opener.rs

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ use arrow::datatypes::{FieldRef, SchemaRef, TimeUnit};
3333
use arrow::error::ArrowError;
3434
use datafusion_common::{exec_err, DataFusionError, Result};
3535
use datafusion_datasource::PartitionedFile;
36+
use datafusion_physical_expr::simplifier::PhysicalExprSimplifier;
3637
use datafusion_physical_expr::PhysicalExprSchemaRewriter;
3738
use datafusion_physical_expr_common::physical_expr::{
3839
is_dynamic_physical_expr, PhysicalExpr,
@@ -246,7 +247,16 @@ impl FileOpener for ParquetOpener {
246247
)
247248
.rewrite(p)
248249
.map_err(ArrowError::from)
250+
.map(|p| {
251+
// After rewriting to the file schema, further simplifications may be possible.
252+
// For example, if `'a' = col_that_is_missing` becomes `'a' = NULL` that can then be simplified to `FALSE`
253+
// and we can avoid doing any more work on the file (bloom filters, loading the page index, etc.).
254+
PhysicalExprSimplifier::new(&physical_file_schema)
255+
.simplify(p)
256+
.map_err(ArrowError::from)
257+
})
249258
})
259+
.transpose()?
250260
.transpose()?;
251261

252262
// Build predicates for this specific file

0 commit comments

Comments
 (0)