[SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict caused by InferFiltersFromConstraints #19149

gatorsmile · 2017-09-06T18:11:49Z

What changes were proposed in this pull request?

The optimizer rule InferFiltersFromConstraints could trigger our batch Operator Optimizations exceeds the max iteration limit (i.e., 100) so that the final plan might not be properly optimized. The rule InferFiltersFromConstraints could conflict with the other Filter/Join predicate reduction rules. Thus, we need to separate InferFiltersFromConstraints from the other rules.

This PR is to separate InferFiltersFromConstraints from the main batch Operator Optimizations .

How was this patch tested?

The existing test cases.

SparkQA · 2017-09-06T19:38:12Z

Test build #81469 has finished for PR 19149 at commit 9c57c8b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class NettyMemoryMetrics implements MetricSet

SparkQA · 2017-09-06T19:41:23Z

Test build #81470 has finished for PR 19149 at commit 41785dd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-06T21:43:49Z

Test build #81475 has finished for PR 19149 at commit 0be3b67.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-06T21:56:21Z

Test build #81476 has finished for PR 19149 at commit 6fc7140.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-07T00:47:25Z

Test build #81483 has finished for PR 19149 at commit e5501e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-07T07:04:47Z

Test build #81495 has finished for PR 19149 at commit 1a22533.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-09-07T07:07:08Z

retest this please

SparkQA · 2017-09-07T09:47:37Z

Test build #81502 has finished for PR 19149 at commit 1a22533.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-09-08T00:59:46Z

Hi, @gatorsmile .
According to the PR description, it's about PruneFilters. Do we need a test case because SPARK-21652 is about ConstantPropagation, not PruneFilters. Also, PR title needs to be updated like description.

dongjoon-hyun · 2017-09-08T01:00:45Z

Except that, Isolation of InferFiltersFromConstraints looks good to me.

yhuai · 2017-09-29T17:08:05Z

Can we add a test?

adrian-ionescu · 2017-09-29T17:29:05Z

I think the idea is good, but maybe the fix can be refined by identifying which are the rules that conflict with InferFilterFromConstraints and only rerunning those ones? Seems like a waste right now to run rules like CombineUnions or RewriteCorrelatedScalarSubquery twice (before and after), when they're obviously not impacted by InferFilterFromConstraints.

adrian-ionescu · 2017-09-30T17:26:48Z

At least part of the issue was solved by #19201.
Unless we have a repro case, let's not pursue this.

gatorsmile · 2017-10-03T16:24:35Z

Actually, the root issue is not resolve by #19201. Will add a unit test case later.

SparkQA · 2017-12-17T08:05:02Z

Test build #85015 has finished for PR 19149 at commit 9b6fe36.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-12-17T08:07:02Z

retest this please

SparkQA · 2017-12-17T11:03:12Z

Test build #85018 has finished for PR 19149 at commit 9b6fe36.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-12-17T17:08:05Z

cc @cloud-fan @sameeragarwal @jiangxb1987 @gengliangwang

cloud-fan · 2017-12-18T10:17:11Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

+ PushDownPredicate,
+ LimitPushDown,
+ ColumnPruning,
+ InferFiltersFromConstraints,


we still have it in the big batch?

Yeah, it is filtered out in the first and the third batch.

just curious, why not remove it from the big batch?

gatorsmile · 2017-12-19T06:43:11Z

retest this please

SparkQA · 2017-12-19T08:05:01Z

Test build #85095 has finished for PR 19149 at commit 9b6fe36.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-12-19T08:20:39Z

retest this please

cloud-fan · 2017-12-19T13:54:51Z

retest this please

jiangxb1987

LGTM

SparkQA · 2017-12-19T16:52:57Z

Test build #85108 has finished for PR 19149 at commit 9b6fe36.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-12-19T17:06:05Z

Thanks! Merged to master.

## What changes were proposed in this pull request? Previously, PR apache#19201 fix the problem of non-converging constraints. After that PR apache#19149 improve the loop and constraints is inferred only once. So the problem of non-converging constraints is gone. However, the case below will fail. ``` spark.range(5).write.saveAsTable("t") val t = spark.read.table("t") val left = t.withColumn("xid", $"id" + lit(1)).as("x") val right = t.withColumnRenamed("id", "xid").as("y") val df = left.join(right, "xid").filter("id = 3").toDF() checkAnswer(df, Row(4, 3)) ``` Because `aliasMap` replace all the aliased child. See the test case in PR for details. This PR is to fix this bug by removing useless code for preventing non-converging constraints. It can be also fixed with apache#20270, but this is much simpler and clean up the code. ## How was this patch tested? Unit test Author: Wang Gengliang <ltnwgl@gmail.com> Closes apache#20278 from gengliangwang/FixConstraintSimple.

## What changes were proposed in this pull request? Previously, PR #19201 fix the problem of non-converging constraints. After that PR #19149 improve the loop and constraints is inferred only once. So the problem of non-converging constraints is gone. However, the case below will fail. ``` spark.range(5).write.saveAsTable("t") val t = spark.read.table("t") val left = t.withColumn("xid", $"id" + lit(1)).as("x") val right = t.withColumnRenamed("id", "xid").as("y") val df = left.join(right, "xid").filter("id = 3").toDF() checkAnswer(df, Row(4, 3)) ``` Because `aliasMap` replace all the aliased child. See the test case in PR for details. This PR is to fix this bug by removing useless code for preventing non-converging constraints. It can be also fixed with #20270, but this is much simpler and clean up the code. ## How was this patch tested? Unit test Author: Wang Gengliang <ltnwgl@gmail.com> Closes #20278 from gengliangwang/FixConstraintSimple. (cherry picked from commit 8598a98) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

gatorsmile force-pushed the inferFilterRule branch 2 times, most recently from 0be3b67 to 6fc7140 Compare September 6, 2017 20:08

gatorsmile changed the title ~~[SPARK-21652][SQL][FOLLOW-UP] Fix rule confliction between InferFiltersFromConstraints and ConstantPropagation~~ [SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict between InferFiltersFromConstraints and ConstantPropagation Sep 6, 2017

fix.

9b6fe36

gatorsmile changed the title ~~[SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict between InferFiltersFromConstraints and ConstantPropagation~~ [SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict caused by InferFiltersFromConstraints Dec 17, 2017

gatorsmile force-pushed the inferFilterRule branch from 1a22533 to 9b6fe36 Compare December 17, 2017 07:12

cloud-fan reviewed Dec 18, 2017

View reviewed changes

jiangxb1987 approved these changes Dec 19, 2017

View reviewed changes

asfgit closed this in ef10f45 Dec 19, 2017

This was referenced Jan 15, 2018

[SPARK-23079][SQL] Fix query constraints propagation with aliases #20270

Closed

[SPARK-23079][SQL]Fix query constraints propagation with aliases #20278

Closed

maropu mentioned this pull request Jul 21, 2020

[SPARK-30876][SQL] Optimizer fails to infer constraints within join #29170

Closed

tanelk mentioned this pull request Sep 4, 2020

[SPARK-32801][SQL] Make InferFiltersFromConstraints take into account EqualNullSafe #29650

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict caused by InferFiltersFromConstraints #19149

[SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict caused by InferFiltersFromConstraints #19149

gatorsmile commented Sep 6, 2017 •

edited

Loading

SparkQA commented Sep 6, 2017

SparkQA commented Sep 6, 2017

SparkQA commented Sep 6, 2017

SparkQA commented Sep 6, 2017

SparkQA commented Sep 7, 2017

SparkQA commented Sep 7, 2017

gatorsmile commented Sep 7, 2017

SparkQA commented Sep 7, 2017

dongjoon-hyun commented Sep 8, 2017 •

edited

Loading

dongjoon-hyun commented Sep 8, 2017

yhuai commented Sep 29, 2017

adrian-ionescu commented Sep 29, 2017

adrian-ionescu commented Sep 30, 2017

gatorsmile commented Oct 3, 2017 •

edited

Loading

SparkQA commented Dec 17, 2017

gatorsmile commented Dec 17, 2017

SparkQA commented Dec 17, 2017

gatorsmile commented Dec 17, 2017

cloud-fan Dec 18, 2017

gatorsmile Dec 19, 2017 •

edited

Loading

gengliangwang Dec 19, 2017

gatorsmile commented Dec 19, 2017

SparkQA commented Dec 19, 2017

gatorsmile commented Dec 19, 2017

cloud-fan commented Dec 19, 2017

jiangxb1987 left a comment

SparkQA commented Dec 19, 2017

gatorsmile commented Dec 19, 2017

[SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict caused by InferFiltersFromConstraints #19149

[SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict caused by InferFiltersFromConstraints #19149

Conversation

gatorsmile commented Sep 6, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Sep 6, 2017

SparkQA commented Sep 6, 2017

SparkQA commented Sep 6, 2017

SparkQA commented Sep 6, 2017

SparkQA commented Sep 7, 2017

SparkQA commented Sep 7, 2017

gatorsmile commented Sep 7, 2017

SparkQA commented Sep 7, 2017

dongjoon-hyun commented Sep 8, 2017 • edited Loading

dongjoon-hyun commented Sep 8, 2017

yhuai commented Sep 29, 2017

adrian-ionescu commented Sep 29, 2017

adrian-ionescu commented Sep 30, 2017

gatorsmile commented Oct 3, 2017 • edited Loading

SparkQA commented Dec 17, 2017

gatorsmile commented Dec 17, 2017

SparkQA commented Dec 17, 2017

gatorsmile commented Dec 17, 2017

cloud-fan Dec 18, 2017

Choose a reason for hiding this comment

gatorsmile Dec 19, 2017 • edited Loading

Choose a reason for hiding this comment

gengliangwang Dec 19, 2017

Choose a reason for hiding this comment

gatorsmile commented Dec 19, 2017

SparkQA commented Dec 19, 2017

gatorsmile commented Dec 19, 2017

cloud-fan commented Dec 19, 2017

jiangxb1987 left a comment

Choose a reason for hiding this comment

SparkQA commented Dec 19, 2017

gatorsmile commented Dec 19, 2017

gatorsmile commented Sep 6, 2017 •

edited

Loading

dongjoon-hyun commented Sep 8, 2017 •

edited

Loading

gatorsmile commented Oct 3, 2017 •

edited

Loading

gatorsmile Dec 19, 2017 •

edited

Loading