-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
@jayzhan211 removed the UnwrapCastInComparison
optimizer (and combined it with the simplifier) In
Doing so seemed to make a non trivial difference for planning speed:
I think the improvement came from reducing the number of Optimizer passes (and this rewrites/copies of the plan and all expressions) that happened
Here were my mesurements about speed
Describe the solution you'd like
I would like to make planning faster by potentially combining other passes from this list:
datafusion/datafusion/optimizer/src/optimizer.rs
Lines 243 to 272 in 43ecd9b
let rules: Vec<Arc<dyn OptimizerRule + Sync + Send>> = vec![ | |
Arc::new(EliminateNestedUnion::new()), | |
Arc::new(SimplifyExpressions::new()), | |
Arc::new(UnwrapCastInComparison::new()), | |
Arc::new(ReplaceDistinctWithAggregate::new()), | |
Arc::new(EliminateJoin::new()), | |
Arc::new(DecorrelatePredicateSubquery::new()), | |
Arc::new(ScalarSubqueryToJoin::new()), | |
Arc::new(ExtractEquijoinPredicate::new()), | |
Arc::new(EliminateDuplicatedExpr::new()), | |
Arc::new(EliminateFilter::new()), | |
Arc::new(EliminateCrossJoin::new()), | |
Arc::new(CommonSubexprEliminate::new()), | |
Arc::new(EliminateLimit::new()), | |
Arc::new(PropagateEmptyRelation::new()), | |
// Must be after PropagateEmptyRelation | |
Arc::new(EliminateOneUnion::new()), | |
Arc::new(FilterNullJoinKeys::default()), | |
Arc::new(EliminateOuterJoin::new()), | |
// Filters can't be pushed down past Limits, we should do PushDownFilter after PushDownLimit | |
Arc::new(PushDownLimit::new()), | |
Arc::new(PushDownFilter::new()), | |
Arc::new(SingleDistinctToGroupBy::new()), | |
// The previous optimizations added expressions and projections, | |
// that might benefit from the following rules | |
Arc::new(SimplifyExpressions::new()), | |
Arc::new(UnwrapCastInComparison::new()), | |
Arc::new(CommonSubexprEliminate::new()), | |
Arc::new(EliminateGroupByConstant::new()), | |
Arc::new(OptimizeProjections::new()), |
Describe alternatives you've considered
Some potential candidates to try consolidating:
EliminateNestedUnion
+EliminateOneUnion
EliminateJoin
andEliminateJoin
You can run the planning benchmarks like
cargo bench --bench sql_planner
Additional context
No response
jayzhan211
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request