[SPARK-14592][SQL] Native support for CREATE TABLE LIKE DDL command #12362

viirya · 2016-04-13T15:33:22Z

What changes were proposed in this pull request?

JIRA: https://issues.apache.org/jira/browse/SPARK-14592

This patch adds native support for DDL command CREATE TABLE LIKE.

The SQL syntax is like:

CREATE TABLE table_name LIKE existing_table
CREATE TABLE IF NOT EXISTS table_name LIKE existing_table

How was this patch tested?

HiveDDLCommandSuite. HiveQuerySuite already tests CREATE TABLE LIKE.

SparkQA · 2016-04-13T16:51:16Z

Test build #55722 has finished for PR 12362 at commit a23559d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-04-13T19:12:39Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

        rowFormat?  createFileFormat? locationSpec?
        (TBLPROPERTIES tablePropertyList)?
        (AS? query)?                                                   #createTable
+    | createTableHeader LIKE source=tableIdentifier                    #createTableLike


Curious why you do not implement the full Hive spec? It seems usefull to be able to change the non-structural properties.

actually, what does it mean to provide table properties or row format in a CREATE TABLE LIKE statement? Trying to merge the provided properties with the existing ones seems complicated and full of corner cases. It might be simpler to just handle CREATE TABLE x LIKE y instead of accepting more things that we don't know how to handle.

e.g. Hive only documents the simple case of CREATE TABLE x LIKE y without any other flags or properties
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableLike

(For that reason I don't think we should use createTableHeader here since it doesn't make sense to say CREATE TEMPORARY TABLE x LIKE ..., and we just throw an exception downstream anyway.

Yeah, just being curious here (not saying that we should support this). The grammar (of course) was much more elaborate in its definition: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g#L887-L892

yea, I am referring the Hive manual that only lists one syntax CREATE TABLE x LIKE y.

Not sure if we want to support CREATE TABLE IF NOT EXIST x LIKE y? It is not on the Hive manual, but HiveQuerySuite has tests using this syntax...

I just found that in HiveCompatibilitySuite, one test create_like_tbl_props uses the syntax:

CREATE TABLE test_table LIKE src TBLPROPERTIES('key'='value')

So do we want to support it? Or just disable this test? @andrewor14 @hvanhovell

I would say disable it. I don't think it actually merges the table properties (and it would be super confusing if it did)

andrewor14 · 2016-04-13T20:49:19Z

@viirya we just merged a patch that implements CREATE TABLE. It would be good if you can rebase. Some of the changes in that patch may be relevant here.

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveSqlParser.scala sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala

viirya · 2016-04-14T00:58:35Z

@andrewor14 @hvanhovell Thanks. Your comments are addressed. One remaining question is do we want to support the syntax like:

CREATE TABLE test_table LIKE src TBLPROPERTIES('key'='value')

Because in HiveCompatibilitySuite, one test create_like_tbl_props uses the syntax. If we don't want to support it, we need to disable the test then.

andrewor14 · 2016-04-14T01:55:20Z

I would say disable it. I don't think it actually merges the table properties (and it would be super confusing if it did)

SparkQA · 2016-04-14T02:02:19Z

Test build #55771 has finished for PR 12362 at commit 0bcafcd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

S

SparkQA · 2016-04-14T04:10:44Z

Test build #55783 has finished for PR 12362 at commit ec589b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-04-14T08:12:13Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+        s"Source table in CREATE TABLE LIKE cannot be temporary: '$sourceTable'")
+    }
+
+    val tableToCreate = catalog.getTableMetadata(sourceTable).copy(


instead of doing a copy and change some variables that we need to update, how about we create a new CatalogTable and put the stuff that we need to retain from the source table? Then it's more clear that what we retained for the CREATE TABLE LIKE command.

Thanks. But I do this to address previous comment. The main concern is that we may add more fields into CatalogTable. If so, we will need to revisit here to add these fields.

Actually I am neutral on this.

My concern is that we may add more fields into CatalogTable which we don't want to retain, like lastAccessTime, I'm not sure which way is safer too, cc @andrewor14

Your concern is correct. But I think most fields in source table should be retained. Only few exceptions like lastAccessTime.

@dilipbiswal and I also hit the same issue when doing the Describe Table. At the end, we decided to avoid adding more fields into CatalogTable. Will follow the discussion in this thread. Thanks!

I think things like lastAccessTime are exceptions. If we don't do copy here then we might forget to copy a new field that we add to CatalogTable.

cloud-fan · 2016-04-14T08:16:51Z

LGTM except one comment.

andrewor14 · 2016-04-14T18:06:59Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveSqlParser.scala

+   */
+  override def visitCreateTableLike(ctx: CreateTableLikeContext): LogicalPlan = withOrigin(ctx) {
+    val targetTable = visitTableIdentifier(ctx.target)
+    val sourceTable = visitTableIdentifier(ctx.source)


much better!

andrewor14 · 2016-04-14T18:07:06Z

LGTM2

andrewor14 · 2016-04-14T18:08:31Z

Merged into master, thanks for your work!

viirya added 4 commits April 13, 2016 10:45

init import.

2a0a474

Make it work.

6525695

Reset storage location.

82a3ee8

Modify comment.

a23559d

hvanhovell reviewed Apr 13, 2016
View reviewed changes

viirya added 3 commits April 14, 2016 00:09

Fix it.

def9039

Fix scala style.

0bcafcd

Disable create_like_tbl_props.

ec589b8

S

cloud-fan reviewed Apr 14, 2016
View reviewed changes

andrewor14 reviewed Apr 14, 2016
View reviewed changes

asfgit closed this in 28efdd3 Apr 14, 2016

viirya deleted the create-table-like branch December 27, 2023 18:33

[SPARK-14592][SQL] Native support for CREATE TABLE LIKE DDL command #12362

[SPARK-14592][SQL] Native support for CREATE TABLE LIKE DDL command #12362

Uh oh!

Conversation

viirya commented Apr 13, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Apr 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Apr 13, 2016

Uh oh!

viirya commented Apr 14, 2016

Uh oh!

andrewor14 commented Apr 14, 2016

Uh oh!

SparkQA commented Apr 14, 2016

Uh oh!

SparkQA commented Apr 14, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Apr 14, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Apr 14, 2016

Uh oh!

andrewor14 commented Apr 14, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants