[SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2 #25771

imback82 · 2019-09-12T04:52:04Z

What changes were proposed in this pull request?

This PR exposes USE CATALOG/USE SQL commands as described in this SPIP

It also exposes currentCatalog in CatalogManager.

Finally, it changes SHOW NAMESPACES and SHOW TABLES to use the current catalog if no catalog is specified (instead of default catalog).

Why are the changes needed?

There is currently no mechanism to change current catalog/namespace thru SQL commands.

Does this PR introduce any user-facing change?

Yes, you can perform the following:

// Sets the current catalog to 'testcat'
spark.sql("USE CATALOG testcat")

// Sets the current catalog to 'testcat' and current namespace to 'ns1.ns2'.
spark.sql("USE ns1.ns2 IN testcat")

// Now, the following will use 'testcat' as the current catalog and 'ns1.ns2' as the current namespace.
spark.sql("SHOW NAMESPACES")

How was this patch tested?

Added new unit tests.

imback82 · 2019-09-12T04:52:40Z

cc: @cloud-fan @rdblue

imback82 · 2019-09-12T04:54:01Z

@cloud-fan / @rdblue This will probably be refactored after #25747 is merged, but I wanted to send this out to get some feedback on the usage of current catalog.

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala

SparkQA · 2019-09-12T05:23:48Z

Test build #110501 has finished for PR 25771 at commit 0d5246d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2019-09-12T06:15:32Z

sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala


+  test("UseCatalog: use catalog with v2 catalog") {
+    val catalogManager = spark.sessionState.catalogManager
+    assert(catalogManager.currentCatalog.name() == "session")


This needs to be updated to "spark_session" right?

sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/CatalogManager.scala

imback82

I will update this when #25747 is merged.

SparkQA · 2019-09-17T01:34:09Z

Test build #110708 has finished for PR 25771 at commit 12685e0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public abstract class DelegatingCatalogExtension implements BaseSessionCatalog
class CatalogManager(conf: SQLConf, defaultSessionCatalog: BaseSessionCatalog) extends Logging

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala

SparkQA · 2019-09-17T17:08:36Z

Test build #110797 has started for PR 25771 at commit ad08096.

shaneknapp · 2019-09-17T17:41:21Z

test this please

SparkQA · 2019-09-17T19:06:50Z

Test build #110806 has finished for PR 25771 at commit ad08096.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public abstract class DelegatingCatalogExtension implements CatalogExtension

SparkQA · 2019-09-17T19:44:51Z

Test build #110822 has finished for PR 25771 at commit 1fb5751.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

docs/sql-keywords.md

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/BaseSessionCatalog.java

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala

sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala

cloud-fan · 2019-09-27T08:14:16Z

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala

+    assert(catalogManager.currentCatalog.name() == "session")
+    assert(catalogManager.currentNamespace === Array("default"))
+
+    // The following implicitly creates namespaces.


this reminds that we should have a CREATE NAMESPACE command. We can do it in followup.

Yes, I intend to do it right after this.

I created a JIRA to track this along with SHOW CURRENT CATALOG and SHOW CURRENT NAMESPACE as you suggested.

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala

SparkQA · 2019-09-27T10:01:11Z

Test build #111474 has finished for PR 25771 at commit 1579f97.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-28T00:40:02Z

Test build #111498 has finished for PR 25771 at commit b7c1e79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-28T07:05:01Z

Test build #111519 has finished for PR 25771 at commit 2534baf.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class SimpleTestOptimizer extends Optimizer(

imback82 · 2019-09-30T02:46:03Z

@cloud-fan this is ready for another round of review. Thanks!

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

cloud-fan · 2019-09-30T06:44:50Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala

+      namespace: Option[Seq[String]]): Unit = {
+    namespace.map { ns =>
+      val nsArray = ns.toArray
+      if (!catalog.asNamespaceCatalog.namespaceExists(nsArray)) {


This means that we can't set current namespace for a v2 catalog that doesn't implement SupportsNamespace.

This is a little counter-intuitive as TableCatalog.loadTable does accept a namespace parameter. Maybe we shouldn't do the namespace check when setting current namespace.

I removed this check. For V1, database check will be performed in SessionCatalog.setCurrentDatabase (so the behavior remains the same) and for V2, there will be no checks performed.

cloud-fan

LGTM, only a few minor comments

SparkQA · 2019-09-30T15:17:18Z

Test build #111609 has started for PR 25771 at commit 3f2e1c1.

cloud-fan · 2019-10-02T03:11:59Z

I think it's ready to go. @imback82 can you fix the conflict? thanks!

imback82 · 2019-10-02T04:43:10Z

I just resolved the conflicts. Thanks @cloud-fan!

SparkQA · 2019-10-02T07:05:02Z

Test build #111670 has finished for PR 25771 at commit 88872ea.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

…red.

SparkQA · 2019-10-02T12:22:01Z

Test build #111681 has finished for PR 25771 at commit 8485619.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-10-02T13:55:32Z

thanks, merging to master!

imback82 added 8 commits September 6, 2019 14:46

initial checkin

d956f10

add rules

826c148

Merge branch 'master' into use_namespace + more feature work

eda2d9d

more changes

3faebcf

adding more unit tests

b1b59fc

minor cleanups

f8b3058

Merge branch 'master' into use_namespace

081fdb3

update docs/sql-keywords.md

0d5246d

imback82 commented Sep 12, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala Outdated Show resolved Hide resolved

imback82 commented Sep 12, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala Show resolved Hide resolved

imback82 commented Sep 12, 2019

View reviewed changes

dongjoon-hyun added the SQL label Sep 12, 2019

cloud-fan reviewed Sep 12, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/CatalogManager.scala Outdated Show resolved Hide resolved

imback82 added 2 commits September 16, 2019 16:55

Merge branch 'master' into use_namespace

680b16d

Address PR comments (introduce BaseSessionCatalog)

12685e0

imback82 commented Sep 17, 2019

View reviewed changes

Fix tests

ad08096

imback82 commented Sep 17, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala Outdated Show resolved Hide resolved

Fix java style check failures.

1fb5751

cloud-fan reviewed Sep 18, 2019

View reviewed changes

docs/sql-keywords.md Outdated Show resolved Hide resolved

cloud-fan reviewed Sep 18, 2019

View reviewed changes

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 Outdated Show resolved Hide resolved

cloud-fan reviewed Sep 18, 2019

View reviewed changes

sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/BaseSessionCatalog.java Outdated Show resolved Hide resolved

cloud-fan reviewed Sep 27, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Sep 27, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Sep 27, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala Show resolved Hide resolved

Fix sql tests

b7c1e79

imback82 added 3 commits September 27, 2019 20:49

address PR comments

0bceddb

Simplify USE parsing.

b33060b

Retain existing formatting

2534baf

cloud-fan reviewed Sep 30, 2019

View reviewed changes

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Sep 30, 2019

View reviewed changes

cloud-fan approved these changes Sep 30, 2019

View reviewed changes

Address PR comments

3f2e1c1

imback82 added 2 commits October 1, 2019 21:34

Merge branch 'master' into use_namespace

57f05fd

formatting

88872ea

Reverting back GlobalTempView test since namespace check is not requi…

8485619

…red.

cloud-fan closed this in f2ead4d Oct 2, 2019

[SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2 #25771

[SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2 #25771

Uh oh!

Conversation

imback82 commented Sep 12, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

imback82 commented Sep 12, 2019

Uh oh!

imback82 commented Sep 12, 2019

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

imback82 Sep 12, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

imback82 left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

shaneknapp commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloud-fan Sep 27, 2019

Choose a reason for hiding this comment

Uh oh!

imback82 Sep 28, 2019

Choose a reason for hiding this comment

Uh oh!

imback82 Sep 28, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparkQA commented Sep 27, 2019

Uh oh!

SparkQA commented Sep 28, 2019

Uh oh!

SparkQA commented Sep 28, 2019

Uh oh!

imback82 commented Sep 30, 2019

Uh oh!

Uh oh!

cloud-fan Sep 30, 2019

Choose a reason for hiding this comment

Uh oh!

imback82 Sep 30, 2019

Choose a reason for hiding this comment

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 30, 2019

Uh oh!

cloud-fan commented Oct 2, 2019

Uh oh!

imback82 commented Oct 2, 2019

Uh oh!

SparkQA commented Oct 2, 2019

Uh oh!

SparkQA commented Oct 2, 2019

Uh oh!

cloud-fan commented Oct 2, 2019