[SPARK-24953] [SQL] Prune a branch in CaseWhen if previously seen
#21904
Conversation
|
Test build #93705 has finished for PR 21904 at commit
|
| CaseWhen( h :+ t.head, None) | ||
|
|
||
| case e @ CaseWhen(branches, _) => | ||
| val newBranches = branches.foldLeft(List[(Expression, Expression)]()) { |
There was a problem hiding this comment.
what about using ArrayBuffer? So we don't have to recreate the List at every iteration...
| newBranches | ||
| } else if (newBranches.nonEmpty && newBranches.last._2.semanticEquals(branch._2)) { | ||
| // If the outputs of two adjacent branches are the same, two branches can be combined. | ||
| newBranches.take(newBranches.length - 1) |
| } else if (newBranches.nonEmpty && newBranches.last._2.semanticEquals(branch._2)) { | ||
| // If the outputs of two adjacent branches are the same, two branches can be combined. | ||
| newBranches.take(newBranches.length - 1) | ||
| .:+((Or(newBranches.last._1, branch._1), newBranches.last._2)) |
There was a problem hiding this comment.
For example, the following case can be benefitted from this rule,
CaseWhen((UnresolvedAttribute("a"), Literal(1)) ::
(Not(UnresolvedAttribute("a")), Literal(1)) ::
(LessThan(Rand(1), Literal(0.5)), Literal(3)) ::
(NonFoldableLiteral(true), Literal(4)) ::
(NonFoldableLiteral(false), Literal(5)) ::
Nil,
None),
Literal(1))| // TODO: In fact, if a condition is a sub-condition of the previous one, | ||
| // TODO: it can be pruned. This is less strict and can be implemented | ||
| // TODO: by decomposing seen conditions. | ||
| newBranches |
There was a problem hiding this comment.
This seems good as the branch is useless. Removing it should simplify code and query plan.
| e.copy(branches = pruneSeenBranches(branches).get) | ||
|
|
||
| case e @ CaseWhen(branches, _) if combineAdjacentBranches(branches).nonEmpty => | ||
| e.copy(branches = combineAdjacentBranches(branches).get) |
There was a problem hiding this comment.
Maybe:
case e @ CaseWhen(branches, _) =>
val prunedBranches = pruneSeenBranches(branches).getOrElse(branches)
val newBranches = combineAdjacentBranches(prunedBranches).getOrElse(prunedBranches)
if (newBranches.length < branches.length) {
e.copy(branches = newBranches)
} else {
e
}There was a problem hiding this comment.
If pruneSeenBranches and combineAdjacentBranches both return input branches if no changes applied, this maybe more simplified:
case e @ CaseWhen(branches, _) =>
val prunedBranches = pruneSeenBranches(branches)
val newBranches = combineAdjacentBranches(prunedBranches)
if (newBranches.length < branches.length) {
e.copy(branches = newBranches)
} else {
e
}There was a problem hiding this comment.
This will work. But my only concern is that this will be caught all the CaseWhen cases; as a result, no more rule can be added after this.
|
Test build #93804 has finished for PR 21904 at commit
|
4 participants
What changes were proposed in this pull request?
If a condition in a branch is previously seen, this branch can be pruned.
If the outputs of two adjacent branches are the same, two branches can be combined.
How was this patch tested?
Tests added.