Skip to content

[SPARK-33315][SQL] Simplify CaseWhen with EqualTo #30222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 9 commits into from
Closed

[SPARK-33315][SQL] Simplify CaseWhen with EqualTo #30222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3a1cd10
simplify CaseWhen with EqualTo
wangyum Nov 2, 2020
593678c
Address comment
wangyum Nov 3, 2020
ee5e6dd
fix
wangyum Nov 5, 2020
5af6ab3
Add more test
wangyum Nov 5, 2020
b611659
simplify test
wangyum Nov 5, 2020
7d7eca3
Address comments
wangyum Nov 5, 2020
5a90bfc
Add deterministic check
wangyum Nov 6, 2020
0099d2a
Merge remote-tracking branch 'upstream/master' into SPARK-33315
wangyum Dec 11, 2020
312c613
Another case
wangyum Dec 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,16 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
} else {
e.copy(branches = branches.take(i).map(branch => (branch._1, elseValue)))
}

case e @ EqualTo(c @ CaseWhen(branches, elseValue), right)
if c.deterministic &&
Copy link
Contributor

@cloud-fan cloud-fan Nov 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More precisely, I think we only need to make sure the skipped branches are all deterministic.

val (picked, skipped) = branches.partition(_._2.equals(right))
if (skipped.forall(_._1.determinisitc)) {
  ...
} else {
  original
}

right.isInstanceOf[Literal] && branches.forall(_._2.isInstanceOf[Literal]) &&
elseValue.forall(_.isInstanceOf[Literal]) =>
if ((branches.map(_._2) ++ elseValue).forall(!_.equals(right))) {
Copy link
Contributor

@cloud-fan cloud-fan Dec 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use an EqualTo expression to compare literals? and how about the null semantic?

FalseLiteral
Copy link
Contributor

@cloud-fan cloud-fan Dec 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update the JIRA/PR title, as it's a different optimization now.

Copy link
Member Author

@wangyum wangyum Dec 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/apache/spark/pull/30790/files

} else {
e
}
}
}
}
Expand Down
34 changes: 34 additions & 0 deletions ...yst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,4 +199,38 @@ class SimplifyConditionalSuite extends PlanTest with ExpressionEvalHelper with P
If(Factorial(5) > 100L, b, nullLiteral).eval(EmptyRow))
}
}

test("SPARK-33315: simplify CaseWhen with EqualTo") {
val a = EqualTo(UnresolvedAttribute("a"), Literal(100))
val b = UnresolvedAttribute("b")
val c = EqualTo(UnresolvedAttribute("c"), Literal(true))
val caseWhen = CaseWhen(Seq((a, Literal(1)), (c, Literal(2))), Some(Literal(3)))

assertEquivalent(EqualTo(caseWhen, Literal(4)), FalseLiteral)
assertEquivalent(EqualTo(caseWhen, Literal(3)), EqualTo(caseWhen, Literal(3)))
assertEquivalent(EqualTo(caseWhen, Literal("4")), FalseLiteral)
assertEquivalent(EqualTo(caseWhen, Literal("3")), EqualTo(caseWhen, Literal(3)))
assertEquivalent(
EqualTo(CaseWhen(Seq((a, Literal("1")), (c, Literal("2"))), None), Literal("4")),
FalseLiteral)

assertEquivalent(
And(EqualTo(caseWhen, Literal(5)), EqualTo(caseWhen, Literal(6))),
FalseLiteral)

assertEquivalent(
EqualTo(CaseWhen(Seq(normalBranch, (a, Literal(1)), (c, Literal(1))), None), Literal(-1)),
FalseLiteral)

// Do not simplify if it contains non foldable expressions.
assertEquivalent(EqualTo(caseWhen, NonFoldableLiteral(true)),
EqualTo(caseWhen, NonFoldableLiteral(true)))
val nonFoldable = CaseWhen(Seq(normalBranch, (a, b)), None)
assertEquivalent(EqualTo(nonFoldable, Literal(1)), EqualTo(nonFoldable, Literal(1)))

// Do not simplify if it contains non-deterministic expressions.
val nonDeterministic = CaseWhen(Seq((LessThan(Rand(1), Literal(0.5)), Literal(1))), Some(b))
assert(!nonDeterministic.deterministic)
assertEquivalent(EqualTo(nonDeterministic, Literal(-1)), EqualTo(nonDeterministic, Literal(-1)))
}
}