[SPARK-14507] [SQL] EXTERNAL keyword in a CTAS statement is not allowed

[yhuai]

What changes were proposed in this pull request?

This PR makes the parser to throw an exception if a hive style CTAS command has an external keyword. The main reason of this change is to make the syntax consistent with CREATE TABLE USING AS SELECT syntax (it does not allow EXTERNAL keyword).

Another options is to make EXTERNAL keyword optional.

How was this patch tested?

HiveDDLCommandSuite

NOTE: This PR is based on #13386, which needs to be merged first.

[SparkQA]

Test build #59595 has finished for PR 13395 at commit 27332df.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

test this please

[SparkQA]

Test build #59596 has finished for PR 13395 at commit 27332df.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tejasapatil]

nit: CREATE TABLE ... AS SELECT to be consistent with other places.

[andrewor14]

I think it's OK to just leave this; CTAS is a pretty common term

[gatorsmile]

Just realized you are working on all the restrictions of CTAS. Based on the official document of Hive, CTAS has three restrictions:

• The target table cannot be a partitioned table.
• The target table cannot be an external table.
• The target table cannot be a list bucketing table.

I have two test cases you can put in DDLCommandSuite:

  test("CTAS - PARTITIONED BY is not supported") {
    assertUnsupported(
      sql = "CREATE TABLE my_tab(a double, b double) PARTITIONED BY (a STRING) " +
        "AS SELECT key, value FROM mytable1")
  }

  test("CTAS - Bucketing is not supported") {
    assertUnsupported(
      sql = "CREATE TABLE my_tab(a double, b double) " +
        "CLUSTERED BY (a) SORTED BY (b) INTO 5 BUCKETS AS SELECT key, value FROM mytable1")
  }

[yhuai]

@gatorsmile Thanks. For partitioned by, it is in #13386. For clustered by, seems we do have test case in HiveDDLCommandSuite.

[gatorsmile]

uh, I see. Thanks!

[yhuai]

chatted with @andrewor14 . Since https://github.com/apache/spark/pull/13386/files will fix the location handling when convertCTAS is true, probably it is not really needed to ban EXTERNAL keyword (also, hive's syntax support external for a regular create table command).