[SPARK-22301][SQL] Add rule to Optimizer for In with not-nullable value and empty list #19523
Conversation
|
|
||
| override def children: Seq[Expression] = value +: list | ||
| lazy val inSetConvertible = list.forall(_.isInstanceOf[Literal]) | ||
| lazy val isListEmpty = list.isEmpty |
There was a problem hiding this comment.
I am using it to be consistent with the current implementation (see the line above)
There was a problem hiding this comment.
Call list.isEmpty is, in comparison, fast and constant time. It doesn't save much of anything to cache it, and the overhead of a lazy val
|
|
||
| case In(_: AttributeReference, list: Seq[Expression]) if list.isEmpty => Literal.FalseLiteral | ||
| // We rely on the optimizations in org.apache.spark.sql.catalyst.optimizer.OptimizeIn | ||
| // to be sure that the list cannot be empty |
There was a problem hiding this comment.
IMHO this comment is not accurate, since in optimizer we only deal with the case attribute is not nullable.
There was a problem hiding this comment.
We can remove this line after we merge this PR #19522
|
ok to test |
|
@mgaido91 Could you update the PR title? |
|
Test build #82923 has finished for PR 19523 at commit
|
|
retest this please |
|
Test build #82928 has finished for PR 19523 at commit
|
mgaido91
changed the title
|
Test build #83011 has finished for PR 19523 at commit
|
|
LGTM |
|
Thanks! Merged to master. |
5 participants
What changes were proposed in this pull request?
For performance reason, we should resolve in operation on an empty list as false in the optimizations phase, ad discussed in #19522.
How was this patch tested?
Added UT
cc @gatorsmile