[SPARK-40800][SQL] Always inline expressions in OptimizeOneRowRelationSubquery #38260
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
allisonwang-db
force-pushed
the
spark-40800-inline-expr-subquery
branch
from
3f0b1a3 to
2e641c9
Compare
Contributor
Author
|
cc @cloud-fan |
cloud-fan
approved these changes
Oct 20, 2022
Contributor
|
The test failure seems unrelated, can you retrigger? |
Contributor
|
@allisonwang-db You can rebase the code. I pin the Java version to 8u345 at #38311 for workaround and GA can pass without waiting for #38317 |
allisonwang-db
force-pushed
the
spark-40800-inline-expr-subquery
branch
from
2e641c9 to
9a55c95
Compare
allisonwang-db
force-pushed
the
spark-40800-inline-expr-subquery
branch
from
9a55c95 to
8122e23
Compare
Contributor
|
thanks, merging to master! |
SandishKumarHN
pushed a commit
to SandishKumarHN/spark
that referenced
this pull request
Dec 12, 2022
…nSubquery
### What changes were proposed in this pull request?
This PR modifies the optimizer rule `OptimizeOneRowRelationSubquery` to always collapse projects and inline non-volatile expressions.
### Why are the changes needed?
SPARK-39699 made `CollpaseProjects` more conservative. This has impacted correlated subqueries that Spark used to be able to support. For example, Spark used to be able to execute this correlated subquery:
```sql
SELECT (
SELECT array_sort(a, (i, j) -> rank[i] - rank[j]) AS sorted
FROM (SELECT MAP('a', 1, 'b', 2) rank)
) FROM t1
```
But after SPARK-39699, it will throw an exception `Unexpected operator Join Inner` because the projects inside the subquery can no longer be collapsed. We should always inline expressions if possible to support a broader range of correlated subqueries and avoid adding expensive domain joins.
### Does this PR introduce _any_ user-facing change?
Yes. It will allow Spark to execute more types of correlated subqueries.
### How was this patch tested?
Unit test.
Closes apache#38260 from allisonwang-db/spark-40800-inline-expr-subquery.
Authored-by: allisonwang-db <allison.wang@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR modifies the optimizer rule
OptimizeOneRowRelationSubqueryto always collapse projects and inline non-volatile expressions.Why are the changes needed?
SPARK-39699 made
CollpaseProjectsmore conservative. This has impacted correlated subqueries that Spark used to be able to support. For example, Spark used to be able to execute this correlated subquery:But after SPARK-39699, it will throw an exception
Unexpected operator Join Innerbecause the projects inside the subquery can no longer be collapsed. We should always inline expressions if possible to support a broader range of correlated subqueries and avoid adding expensive domain joins.Does this PR introduce any user-facing change?
Yes. It will allow Spark to execute more types of correlated subqueries.
How was this patch tested?
Unit test.