[SPARK-49063][SQL] Fix Between with ScalarSubqueries #47581
Conversation
This reverts commit d54453e.
mihailomilosevic2001
changed the title
| .createWithDefault(true) | ||
|
|
||
| val OPTIMIZE_BETWEEN = | ||
| buildConf("spark.sql.optimizeBetween") |
There was a problem hiding this comment.
It's not only an optimization, but also a correctness fix if the BETWEEN input is nondeterministic. I think a legacy config is better here. How about spark.sql.legacy.duplicateBetweenInput? By default it's false, and people can set it to true to restore to the legacy behavior.
What does this PR fix? |
|
@HyukjinKwon This ticket's PR https://issues.apache.org/jira/browse/SPARK-46366 introduced an optimization for Between expression. Unfortunately it lead to ScalarSubqueries being firstly resolved before duplicated. This lead to failure in RewriteCorrelatedScalarSubqueries as resolved Subqueries will not need another alias resolution, so we would end up left joining the same query twice. Left join is unresolved if both sides have the same alias. |
|
thanks, merging to master! |
3 participants
What changes were proposed in this pull request?
Fix for between with ScalarSubqueries.
Why are the changes needed?
There is a regression introduced from a previous PR #44299. This needs to be addressed as between operator was completely broken with resolved ScalarSubqueries.
Does this PR introduce any user-facing change?
No, the bug is not release yet.
How was this patch tested?
Tests added to golden file.
Was this patch authored or co-authored using generative AI tooling?
No.