[SPARK-12542] [SQL] support except/intersect in HiveQl

[davies]

Parse the SQL query with except/intersect in FROM clause for HivQL.

[davies]

@hvanhovell This patch can't be compiled, failed with:

[error] ~spark/sql/hive/target/scala-2.10/src_managed/main/org/apache/spark/sql/parser/SparkSqlParser_IdentifiersParser.java:11056:  ERROR: too long
[error]     static final String[] DFA5_transitionS = {
[error]                           ^

Have you hit this before?

[hvanhovell]

@davies I haven't seen this one before. We can take a look at the generated source...

[hvanhovell]

@davies I just tried this. If you add this rule the identifier parser grows beyond 64K rows which the JVM can't handle. The short term solution is to split the identifier parser (see @viirya's PR).

In the long term we might need to check if some rules aren't blowing things out of proportion.

[davies]

@hvanhovell Thanks, I will wait for @viirya's PR go first.

[rxin]

do we support just union, i.e. alias for union distinct?

[yhuai]

Hive 1.2's parser should support union.

[rxin]

Actually this test case shouldn't be here? It should just go into the normal sql test case now, and make sure we also test union.

[SparkQA]

Test build #48882 has finished for PR 10622 at commit 4ae0ccf.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48891 has finished for PR 10622 at commit a83d4a2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class CatalystQlSuite extends PlanTest

[rxin]

LGTM. Would be great for @hvanhovell to take a quick look too.

[davies]

The following query does not work:

(select 1 as a) union (select 2 as a)

I had not figure out a way to fix this.

PS: It seems that HiveParse is not a good stand point. lots of corner cases do not work.

[rxin]

Let's open a jira ticket to track that and see if some open source contributors can pick it up.

[davies]

https://issues.apache.org/jira/browse/SPARK-12687

[hvanhovell]

LGTM

[rxin]

Merging this in. Thanks!