Skip to content

[SPARK-11973] [SQL] push filter through aggregation with alias and literals #9959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
davies wants to merge 4 commits into from
Closed

[SPARK-11973] [SQL] push filter through aggregation with alias and literals #9959

davies wants to merge 4 commits into from

Conversation

@davies
Copy link
Contributor

@davies davies commented Nov 25, 2015

Currently, filter can't be pushed through aggregation with alias or literals, this patch fix that.

After this patch, the time of TPC-DS query 4 go down to 13 seconds from 141 seconds (10x improvements).

cc @nongli @yhuai

@rxin
Copy link
Contributor

rxin commented Nov 25, 2015

cc @cloud-fan too

nongli
nongli reviewed Nov 25, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Outdated
Copy link
Contributor

@nongli nongli Nov 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is confusing to read. Not clear why this is calling a utility in PushPredicateThroughProject. Move this into a better named place (similar to PredicateHelper)

@SparkQA
Copy link

SparkQA commented Nov 25, 2015

Test build #46668 has finished for PR 9959 at commit 2fb7a1c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 25, 2015

Test build #46701 has finished for PR 9959 at commit 951fe7a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 25, 2015

Test build #46699 has finished for PR 9959 at commit 0f5d7ba.

  • This patch fails from timeout after a configured wait of 250m.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed Nov 26, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Outdated
Copy link
Contributor

@cloud-fan cloud-fan Nov 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not move this to the existing trait PredicateHelper?

@SparkQA
Copy link

SparkQA commented Nov 26, 2015

Test build #2116 has finished for PR 9959 at commit 951fe7a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed Nov 26, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
Copy link
Contributor

@cloud-fan cloud-fan Nov 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a simplification: expr.find(_.isInstanceOf[AggregateExpression]).isDefined, but it's minor, may not worth another roundtrip.

@cloud-fan
Copy link
Contributor

cloud-fan commented Nov 26, 2015

LGTM

@SparkQA
Copy link

SparkQA commented Nov 26, 2015

Test build #46749 has finished for PR 9959 at commit 37b3088.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Nov 26, 2015
@davies
[SPARK-11973] [SQL] push filter through aggregation with alias and li…
24e62b3 
…terals

Currently, filter can't be pushed through aggregation with alias or literals, this patch fix that.

After this patch, the time of TPC-DS query 4 go down to 13 seconds from 141 seconds (10x improvements).

cc nongli  yhuai

Author: Davies Liu <davies@databricks.com>

Closes #9959 from davies/push_filter2.

(cherry picked from commit 27d69a0)
Signed-off-by: Davies Liu <davies.liu@gmail.com>
@davies
Copy link
Contributor Author

davies commented Nov 26, 2015

Merging into master and 1.6 branch.

@asfgit asfgit closed this in 27d69a0 Nov 26, 2015
@rxin rxin mentioned this pull request Nov 26, 2015
Closed
asfgit pushed a commit that referenced this pull request Nov 27, 2015
@rxin
[SPARK-11973][SQL] Improve optimizer code readability.
de28e4d 
This is a followup for #9959.

I added more documentation and rewrote some monadic code into simpler ifs.

Author: Reynold Xin <rxin@databricks.com>

Closes #9995 from rxin/SPARK-11973.
asfgit pushed a commit that referenced this pull request Nov 27, 2015
@rxin
[SPARK-11973][SQL] Improve optimizer code readability.
f18de5a 
This is a followup for #9959.

I added more documentation and rewrote some monadic code into simpler ifs.

Author: Reynold Xin <rxin@databricks.com>

Closes #9995 from rxin/SPARK-11973.

(cherry picked from commit de28e4d)
Signed-off-by: Reynold Xin <rxin@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@davies @rxin @SparkQA @cloud-fan @nongli @watermen