[SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join #16026

hvanhovell · 2016-11-28T00:30:38Z

What changes were proposed in this pull request?

We currently push down join conditions of a Left Anti join to both sides of the join. This is similar to Inner, Left Semi and Existence (a specialized left semi) join. The problem is that this changes the semantics of the join; a left anti join filters out rows that matches the join condition.

This PR fixes this by only pushing down conditions to the left hand side of the join. This is similar to the behavior of left outer join.

How was this patch tested?

Added tests to FilterPushdownSuite.scala and created a SQLQueryTestSuite file for left anti joins with a regression test.

…ide of the join.

hvanhovell · 2016-11-28T00:35:02Z

cc @dongjoon-hyun could you take a look at this?

SparkQA · 2016-11-28T02:47:42Z

Test build #69211 has finished for PR 16026 at commit 4d6b183.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-11-28T03:16:37Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala

+    comparePlans(optimized, analysis.EliminateSubqueryAliases(correctAnswer))
+  }
+
+  test("joins: only push down to the right of a left anti join") {


How about joins: only push down join conditions to the right of a left anti join?

Do we need a JIRA issue number here?

gatorsmile · 2016-11-28T03:19:13Z

The current PR title is a little bit confusing. Could you please update the PR title from push-down predicates to push-down join conditions?

dongjoon-hyun · 2016-11-28T03:39:52Z

Thank you for pinging me, @hvanhovell . I'll take a look, too.

gatorsmile · 2016-11-28T04:00:19Z

LGTM except a minor comment.

dongjoon-hyun · 2016-11-28T04:00:50Z

LGTM, @hvanhovell .

SparkQA · 2016-11-28T15:09:27Z

Test build #69242 has finished for PR 16026 at commit 47bbdff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-11-28T15:10:27Z

I am merging this to master/2.1. Thanks for the reviews!

… of a LEFT ANTI join ## What changes were proposed in this pull request? We currently push down join conditions of a Left Anti join to both sides of the join. This is similar to Inner, Left Semi and Existence (a specialized left semi) join. The problem is that this changes the semantics of the join; a left anti join filters out rows that matches the join condition. This PR fixes this by only pushing down conditions to the left hand side of the join. This is similar to the behavior of left outer join. ## How was this patch tested? Added tests to `FilterPushdownSuite.scala` and created a SQLQueryTestSuite file for left anti joins with a regression test. Author: Herman van Hovell <hvanhovell@databricks.com> Closes #16026 from hvanhovell/SPARK-18597. (cherry picked from commit 38e2982) Signed-off-by: Herman van Hovell <hvanhovell@databricks.com>

… of a LEFT ANTI join ## What changes were proposed in this pull request? We currently push down join conditions of a Left Anti join to both sides of the join. This is similar to Inner, Left Semi and Existence (a specialized left semi) join. The problem is that this changes the semantics of the join; a left anti join filters out rows that matches the join condition. This PR fixes this by only pushing down conditions to the left hand side of the join. This is similar to the behavior of left outer join. ## How was this patch tested? Added tests to `FilterPushdownSuite.scala` and created a SQLQueryTestSuite file for left anti joins with a regression test. Author: Herman van Hovell <hvanhovell@databricks.com> Closes apache#16026 from hvanhovell/SPARK-18597. (cherry picked from commit 38e2982) Signed-off-by: Herman van Hovell <hvanhovell@databricks.com> # Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

nsyca · 2016-11-28T17:08:22Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala


      joinType match {
-        case _: InnerLike | LeftExistence(_) =>
+        case _: InnerLike |  LeftSemi | ExistenceJoin(_) =>


The semantics of ExistenceJoin says we need to preserve all the rows from the left table through the join operation as if it is a regular LeftOuter join. The ExistenceJoin augments the LeftOuter operation with a new column called exists, set to true when the join condition in the ON clause is true and false otherwise. The filter of any rows will happen in the Filter operation above the ExistenceJoin.

Example:

A(c1, c2):
{ (1, 1), (1, 2) }

B(c1):
{ (NULL) }

// can be any value as it is irrelevant in this example

select A.* from A where exists (select 1 from B where A.c1 = A.c2) or A.c2=2

In this example, the correct result is all the rows from A. If the pattern ExistenceJoin at line 935 in Optimizer.scala added by the PR of this JIRA is indeed active, the code will push down the predicate A.c1 = A.c2 to be a Filter on relation A, which will filter the row (1,2) from A.

@nsyca Have you tried the above example using the latest master?

I should have included the full comment from the JIRA. The keyword is "this is not currently exposed". Here is the first part of my comment:

"ExistenceJoin should be treated the same as LeftOuter and LeftAnti, not InnerLike and LeftSemi. This is not currently exposed because the rewrite of [NOT] EXISTS OR ... to ExistenceJoin happens in rule RewritePredicateSubquery, which is in a separate rule set and placed after the rule PushPredicateThroughJoin. During the transformation in the rule PushPredicateThroughJoin, an ExistenceJoin never exists."

…of a Left Anti join [BRANCH-2.0] ## What changes were proposed in this pull request? We currently push down join conditions of a Left Anti join to both sides of the join. This is similar to Inner, Left Semi and Existence (a specialized left semi) join. The problem is that this changes the semantics of the join; a left anti join filters out rows that matches the join condition. This PR fixes this by only pushing down conditions to the right hand side of the join. This is similar to the behavior of left outer join. This PR is a backport of #16026 ## How was this patch tested? Added tests to `FilterPushdownSuite.scala` and created a SQLQueryTestSuite file for left anti joins with a regression test. Author: Herman van Hovell <hvanhovell@databricks.com> Closes #16039 from hvanhovell/SPARK-18597-branch-2.0.

… of a LEFT ANTI join ## What changes were proposed in this pull request? We currently push down join conditions of a Left Anti join to both sides of the join. This is similar to Inner, Left Semi and Existence (a specialized left semi) join. The problem is that this changes the semantics of the join; a left anti join filters out rows that matches the join condition. This PR fixes this by only pushing down conditions to the left hand side of the join. This is similar to the behavior of left outer join. ## How was this patch tested? Added tests to `FilterPushdownSuite.scala` and created a SQLQueryTestSuite file for left anti joins with a regression test. Author: Herman van Hovell <hvanhovell@databricks.com> Closes apache#16026 from hvanhovell/SPARK-18597.

hvanhovell added 5 commits November 20, 2016 13:42

Fix 2.10 compilation

7696cf2

Merge remote-tracking branch 'apache-github/master'

d51b635

Merge remote-tracking branch 'apache-github/master'

7ccb680

Merge remote-tracking branch 'apache-github/master'

e386fed

Do not change left anti join semantics by pushing down to the right s…

4d6b183

…ide of the join.

gatorsmile reviewed Nov 28, 2016

View reviewed changes

hvanhovell changed the title ~~[SPARK-18597][SQL] Do push-down predicates to right side for a LEFT ANTI join~~ [SPARK-18597][SQL] Do not push-down join conditions to the right side for a LEFT ANTI join Nov 28, 2016

hvanhovell changed the title ~~[SPARK-18597][SQL] Do not push-down join conditions to the right side for a LEFT ANTI join~~ [SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join Nov 28, 2016

CR: Modify test name

47bbdff

asfgit closed this in 38e2982 Nov 28, 2016

hvanhovell mentioned this pull request Nov 28, 2016

[SPARK-18597][SQL] Do not push-down join conditions to the left side of a Left Anti join [BRANCH-2.0] #16039

Closed

nsyca reviewed Nov 28, 2016

View reviewed changes

[SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join #16026

[SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join #16026

Uh oh!

Conversation

hvanhovell commented Nov 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

hvanhovell commented Nov 28, 2016

Uh oh!

SparkQA commented Nov 28, 2016

Uh oh!

gatorsmile Nov 28, 2016

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 28, 2016

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Nov 28, 2016

Uh oh!

dongjoon-hyun commented Nov 28, 2016

Uh oh!

gatorsmile commented Nov 28, 2016

Uh oh!

dongjoon-hyun commented Nov 28, 2016

Uh oh!

SparkQA commented Nov 28, 2016

Uh oh!

hvanhovell commented Nov 28, 2016

Uh oh!

nsyca Nov 28, 2016

Choose a reason for hiding this comment

Uh oh!

gatorsmile Nov 28, 2016

Choose a reason for hiding this comment

Uh oh!

nsyca Nov 28, 2016

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hvanhovell commented Nov 28, 2016 •

edited

Loading