From da6410545222ace99c733dacc473dd9781ee6f39 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Sun, 25 Dec 2016 21:06:12 -0800 Subject: [PATCH 1/2] [SPARK-18941][SQL][DOC] Add a new behavior document on `CREATE/DROP TABLE` with `LOCATION` --- docs/sql-programming-guide.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index 0f6e3446559b..7a2c8b56efb3 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -1413,6 +1413,13 @@ options. - Dataset and DataFrame API `explode` has been deprecated, alternatively, use `functions.explode()` with `select` or `flatMap` - Dataset and DataFrame API `registerTempTable` has been deprecated and replaced by `createOrReplaceTempView` + - Changes to `CREATE TABLE ... LOCATION` behavior. + - From Spark 2.0, `CREATE TABLE ... LOCATION` is equivalent to `CREATE EXTERNAL TABLE ... LOCATION` + in order to prevent accidental dropping the existing data in the user-provided locations. + Please see [SPARK-15276](https://issues.apache.org/jira/browse/SPARK-15276) for details. + - As a result, `DROP TABLE` statements on those tables will not remove the data. + Note that this is different than the Hive behavior. + ## Upgrading From Spark SQL 1.5 to 1.6 - From Spark 1.6, by default the Thrift server runs in multi-session mode. Which means each JDBC/ODBC From e60ec434bbe322b30a3f183193473170232a89b5 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Sat, 7 Jan 2017 09:19:28 -0800 Subject: [PATCH 2/2] Revise docs according to the comments. --- docs/sql-programming-guide.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index 7a2c8b56efb3..f4c89e58fa43 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -1413,12 +1413,13 @@ options. - Dataset and DataFrame API `explode` has been deprecated, alternatively, use `functions.explode()` with `select` or `flatMap` - Dataset and DataFrame API `registerTempTable` has been deprecated and replaced by `createOrReplaceTempView` - - Changes to `CREATE TABLE ... LOCATION` behavior. + - Changes to `CREATE TABLE ... LOCATION` behavior for Hive tables. - From Spark 2.0, `CREATE TABLE ... LOCATION` is equivalent to `CREATE EXTERNAL TABLE ... LOCATION` in order to prevent accidental dropping the existing data in the user-provided locations. - Please see [SPARK-15276](https://issues.apache.org/jira/browse/SPARK-15276) for details. + That means, a Hive table created in Spark SQL with the user-specified location is always a Hive external table. + Dropping external tables will not remove the data. Users are not allowed to specify the location for Hive managed tables. + Note that this is different from the Hive behavior. - As a result, `DROP TABLE` statements on those tables will not remove the data. - Note that this is different than the Hive behavior. ## Upgrading From Spark SQL 1.5 to 1.6