-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-17251][SQL] Improve OuterReference to be NamedExpression
#16015
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #69178 has finished for PR 16015 at commit
|
|
The only one test failure is irrelevant to this PR. |
|
Retest this please. |
|
Test build #69181 has started for PR 16015 at commit |
|
Unknown Jenkins failure. |
|
Retest this please. |
|
Test build #69182 has finished for PR 16015 at commit
|
| val m = intercept[AnalysisException] { | ||
| SimpleAnalyzer.ResolveSubquery(expr) | ||
| }.getMessage | ||
| assert(m.contains("Correlated predicates are not supported outside of WHERE/HAVING clauses")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor: Maybe we should update this error. Something like this: Expressions referencing the outer query are not supported outside of WHERE/HAVING clauses
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for review, @hvanhovell . Sure.
|
LGTM pending jenkins. |
|
Thank you, @hvanhovell ! |
|
Test build #69192 has finished for PR 16015 at commit
|
## What changes were proposed in this pull request?
Currently, `OuterReference` is not `NamedExpression`. So, it raises 'ClassCastException` when it used in projection lists of IN correlated subqueries. This PR aims to support that by making `OuterReference` as `NamedExpression` to show correct error messages.
```scala
scala> sql("CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES 1, 2 AS t1(a)")
scala> sql("CREATE TEMPORARY VIEW t2 AS SELECT * FROM VALUES 1 AS t2(b)")
scala> sql("SELECT a FROM t1 WHERE a IN (SELECT a FROM t2)").show
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.OuterReference cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression
```
## How was this patch tested?
Pass the Jenkins test with new test cases.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #16015 from dongjoon-hyun/SPARK-17251-2.
(cherry picked from commit 9c03c56)
Signed-off-by: Herman van Hovell <hvanhovell@databricks.com>
## What changes were proposed in this pull request?
Currently, `OuterReference` is not `NamedExpression`. So, it raises 'ClassCastException` when it used in projection lists of IN correlated subqueries. This PR aims to support that by making `OuterReference` as `NamedExpression` to show correct error messages.
```scala
scala> sql("CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES 1, 2 AS t1(a)")
scala> sql("CREATE TEMPORARY VIEW t2 AS SELECT * FROM VALUES 1 AS t2(b)")
scala> sql("SELECT a FROM t1 WHERE a IN (SELECT a FROM t2)").show
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.OuterReference cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression
```
## How was this patch tested?
Pass the Jenkins test with new test cases.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #16015 from dongjoon-hyun/SPARK-17251-2.
(cherry picked from commit 9c03c56)
Signed-off-by: Herman van Hovell <hvanhovell@databricks.com>
|
Merging to master/2.1/2.0. Thanks! |
|
Thank you for merging, @hvanhovell ! |
## What changes were proposed in this pull request?
Currently, `OuterReference` is not `NamedExpression`. So, it raises 'ClassCastException` when it used in projection lists of IN correlated subqueries. This PR aims to support that by making `OuterReference` as `NamedExpression` to show correct error messages.
```scala
scala> sql("CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES 1, 2 AS t1(a)")
scala> sql("CREATE TEMPORARY VIEW t2 AS SELECT * FROM VALUES 1 AS t2(b)")
scala> sql("SELECT a FROM t1 WHERE a IN (SELECT a FROM t2)").show
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.OuterReference cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression
```
## How was this patch tested?
Pass the Jenkins test with new test cases.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes apache#16015 from dongjoon-hyun/SPARK-17251-2.
## What changes were proposed in this pull request?
Currently, `OuterReference` is not `NamedExpression`. So, it raises 'ClassCastException` when it used in projection lists of IN correlated subqueries. This PR aims to support that by making `OuterReference` as `NamedExpression` to show correct error messages.
```scala
scala> sql("CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES 1, 2 AS t1(a)")
scala> sql("CREATE TEMPORARY VIEW t2 AS SELECT * FROM VALUES 1 AS t2(b)")
scala> sql("SELECT a FROM t1 WHERE a IN (SELECT a FROM t2)").show
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.OuterReference cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression
```
## How was this patch tested?
Pass the Jenkins test with new test cases.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes apache#16015 from dongjoon-hyun/SPARK-17251-2.
What changes were proposed in this pull request?
Currently,
OuterReferenceis notNamedExpression. So, it raises 'ClassCastExceptionwhen it used in projection lists of IN correlated subqueries. This PR aims to support that by makingOuterReferenceasNamedExpression` to show correct error messages.How was this patch tested?
Pass the Jenkins test with new test cases.