Skip to content

[SPARK-13763] [SQL] Remove Project when its Child's Output is Nil #11599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gatorsmile wants to merge 3 commits into from
Closed

[SPARK-13763] [SQL] Remove Project when its Child's Output is Nil #11599

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0fa21ac
remove Project with an empty projectList
gatorsmile Mar 9, 2016
a31b1b5
added two cases.
gatorsmile Mar 9, 2016
68decd1
reorder it.
gatorsmile Mar 9, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,12 +380,12 @@ object ColumnPruning extends Rule[LogicalPlan] {
p
}

// Can't prune the columns on LeafNode
case p @ Project(_, l: LeafNode) => p

// Eliminate no-op Projects
case p @ Project(projectList, child) if sameOutput(child.output, p.output) => child

// Can't prune the columns on LeafNode
case p @ Project(_, l: LeafNode) => p

// for all other logical plans that inherits the output from it's children
case p @ Project(_, child) =>
val required = child.references ++ p.references
Expand Down
16 changes: 16 additions & 0 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ColumnPruningSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,22 @@ class ColumnPruningSuite extends PlanTest {
comparePlans(Optimize.execute(query), expected)
}

test("Eliminate the Project with an empty projectList") {
val input = OneRowRelation
val expected = Project(Literal(1).as("1") :: Nil, input).analyze

val query1 =
Project(Literal(1).as("1") :: Nil, Project(Literal(1).as("1") :: Nil, input)).analyze
comparePlans(Optimize.execute(query1), expected)

val query2 =
Project(Literal(1).as("1") :: Nil, Project(Nil, input)).analyze
comparePlans(Optimize.execute(query2), expected)

// to make sure the top Project will not be removed.
comparePlans(Optimize.execute(expected), expected)
}

test("column pruning for group") {
val testRelation = LocalRelation('a.int, 'b.int, 'c.int)
val originalQuery =
Expand Down