[SPARK-10534][SQL] ORDER BY clause allows only columns that are present in the select projection list

[dilipbiswal]

Find out the missing attributes by recursively looking
at the sort order expression and rest of the code
takes care of projecting them out.

Added description from @cloud-fan

I wanna explain a bit more about this bug.

When we resolve sort ordering, we will use a special method, which only resolves UnresolvedAttributes and UnresolvedExtractValue. However, for something like Floor('a), even the 'a is resolved, the floor expression may still being unresolved as data type mismatch(for example, 'a is string type and Floor need double type), thus can't pass this filter, and we can't push down this missing attribute 'a

[dilipbiswal]

@cloud-fan
Can you please take a look. Thanks ..

[cloud-fan]

the test case can be simplified to:

    val a = testRelation2.output(0)
    val c = testRelation2.output(2)

    val plan = testRelation2.select('c).orderBy(Floor('a).asc)
    val expected = testRelation2.select(c, a).orderBy(Floor(a.cast(DoubleType)).asc).select(c)

    checkAnalysis(plan, expected)

[dilipbiswal]

Thanks a LOT. I will make the change.

[cloud-fan]

I wanna explain a bit more about this bug.

When we resolve sort ordering, we will use a special method, which only resolves UnresolvedAttributes and UnresolvedExtractValue. However, for something like Floor('a), even the 'a is resolved, the floor expression may still being unresolved as data type mismatch(for example, 'a is string type and Floor need double type), thus can't pass this filter, and we can't push down this missing attribute 'a.

[cloud-fan]

ok to test.

[cloud-fan]

how about
val requiredAttributes = AttributeSet(newOrdering).filter(_.resolved)

[dilipbiswal]

Thanks. Much simpler :-). Will make the change and test..

[dilipbiswal]

@cloud-fan
Hi Wenchen, i have fixed the code based on your comments.

[SparkQA]

Test build #43776 has finished for PR 9123 at commit ac78af1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43778 has finished for PR 9123 at commit 6f97fb7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

remove these 2 extra blank lines please.

[cloud-fan]

use 'c and 'a instead of c and a. The plan will get analyzed in the method checkAnalysis, so we should give unresolved attributes, to simulate normal query plans. (FYI, in spark SQL DSL, the symbol 'a will be automatically turned into UnresolvedAttribute)

[cloud-fan]

hi @dilipbiswal , can you update the PR title to make it a complete sentence(i.e. no ellipsis)? and can you also put my explaination of this bug in PR description? so that if someone look into the commit log in the future, they can easily understand what's going on here.

[SparkQA]

Test build #43798 has finished for PR 9123 at commit 4f162ed.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

LGTM pending test.

[cloud-fan]

retest this please.

[SparkQA]

Test build #43796 has finished for PR 9123 at commit 9ed26d1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

cc @marmbrus

[SparkQA]

Test build #43802 has finished for PR 9123 at commit 4f162ed.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

ping @yhuai

[yhuai]

Thank you! Merging to master and 1.5 branch.