[SPARK-16452][SQL] Support basic INFORMATION_SCHEMA #14116

dongjoon-hyun · 2016-07-09T13:06:37Z

What changes were proposed in this pull request?

This PR introduces INFORMATION_SCHEMA, a database consisting views which provide information about all of the tables, views, columns in a database.

scala> spark.catalog
scala> sql("create table t(a int, b double)")
scala> sql("create view v as select 1")
scala> sql("select * from information_schema.databases").show()
+------------+------------------+
|CATALOG_NAME|       SCHEMA_NAME|
+------------+------------------+
|     default|           default|
|     default|information_schema|
+------------+------------------+

scala> sql("select * from information_schema.schemata").show()
+------------+------------------+
|CATALOG_NAME|       SCHEMA_NAME|
+------------+------------------+
|     default|           default|
|     default|information_schema|
+------------+------------------+

scala> sql("select * from information_schema.tables").show()
+-------------+------------+----------+----------+
|TABLE_CATALOG|TABLE_SCHEMA|TABLE_NAME|TABLE_TYPE|
+-------------+------------+----------+----------+
|      default|     default|         t|     TABLE|
|      default|     default|         v|      VIEW|
+-------------+------------+----------+----------+

scala> sql("select * from information_schema.views").show()
+-------------+------------+----------+---------------+
|TABLE_CATALOG|TABLE_SCHEMA|TABLE_NAME|VIEW_DEFINITION|
+-------------+------------+----------+---------------+
|      default|     default|         v|           VIEW|
+-------------+------------+----------+---------------+


scala> sql("select * from information_schema.columns").show()
+-------------+------------+----------+-----------+----------------+-----------+---------+
|TABLE_CATALOG|TABLE_SCHEMA|TABLE_NAME|COLUMN_NAME|ORDINAL_POSITION|IS_NULLABLE|DATA_TYPE|
+-------------+------------+----------+-----------+----------------+-----------+---------+
|      default|     default|         t|          a|               0|       true|      int|
|      default|     default|         t|          b|               1|       true|   double|
|      default|     default|         v|          1|               0|       true|      int|
+-------------+------------+----------+-----------+----------------+-----------+---------+

scala> sql("select * from information_schema.session_variables").show(false)
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
|VARIABLE_NAME                  |VARIABLE_VALUE                                                                                                                               |
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
|hive.metastore.warehouse.dir   |file:/Users/dongjoon/SPARK-16452-SCHEMA/spark-warehouse                                                                                      |
|spark.app.id                   |local-1468412934354                                                                                                                          |
|spark.app.name                 |Spark shell                                                                                                                                  |
|spark.driver.host              |192.168.0.12                                                                                                                                 |
|spark.driver.memory            |6G                                                                                                                                           |
|spark.driver.port              |56198                                                                                                                                        |
|spark.executor.id              |driver                                                                                                                                       |
|spark.home                     |/Users/dongjoon/SPARK-16452-SCHEMA                                                                                                           |
|spark.jars                     |                                                                                                                                             |
|spark.master                   |local[*]                                                                                                                                     |
|spark.repl.class.outputDir     |/private/var/folders/dc/1pz9m69x14q_gw8t7m143t1c0000gn/T/spark-fd7a86e9-651f-4718-be82-6b84b30a97ac/repl-d4e25a57-c5f1-40d9-809b-1276c55f59c6|
|spark.repl.class.uri           |spark://192.168.0.12:56198/classes                                                                                                           |
|spark.sql.catalogImplementation|hive                                                                                                                                         |
|spark.submit.deployMode        |client                                                                                                                                       |
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+

How was this patch tested?

Pass the Jenkins tests including a new testsuite.

dongjoon-hyun · 2016-07-09T13:09:22Z

Hi, @rxin .
I made a PR for your directional advice.

SparkQA · 2016-07-09T14:34:58Z

Test build #62023 has finished for PR 14116 at commit cbdd641.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DatabasesSource extends SchemaRelationProvider
- case class DatabasesRelation(@transient sparkSession: SparkSession)
- class TablesSource extends SchemaRelationProvider
- case class TablesRelation(@transient sparkSession: SparkSession)
- class ViewsSource extends SchemaRelationProvider
- case class ViewsRelation(@transient sparkSession: SparkSession)
- class ColumnsSource extends SchemaRelationProvider
- case class ColumnsRelation(@transient sparkSession: SparkSession)
- class SessionVariablesSource extends SchemaRelationProvider
- case class SessionVariablesRelation(@transient sparkSession: SparkSession)

dongjoon-hyun · 2016-07-09T18:45:55Z

Wow, there was only one error. I fixed it a few minute ago.

- alter table: rename *** FAILED *** (8 milliseconds)
[info]   ArrayBuffer(`dbx`.`tab2`, `columns`, `databases`, `schemata`, `session_variables`, `tables`, `views`) did not equal List(`dbx`.`tab2`) (DDLSuite.scala:466)

listTable returns temporary tables for all databases. This suite should be fixed generally, not specifically for this PR.

dongjoon-hyun · 2016-07-09T19:17:50Z

cc @hvanhovell , too.

SparkQA · 2016-07-09T20:16:23Z

Test build #62029 has finished for PR 14116 at commit d9d9344.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-07-09T20:27:00Z

The failure seems to be irrelevant to this PR.

FAIL [31.391s]: test_predictions (pyspark.mllib.tests.StreamingLogisticRegressionWithSGDTests)
Test predicted values on a toy model.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/mllib/tests.py", line 1426, in test_predictions
    self._eventually(condition, catch_assertions=True)
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/mllib/tests.py", line 135, in _eventually
    raise lastValue
AssertionError: 11 != 20

Locally, it passes like the following.

$ python python/run-tests.py --python-executables python2.7 --modules pyspark-mllib
...
Finished test(python2.7): pyspark.mllib.tests (199s)
Tests passed in 199 seconds

dongjoon-hyun · 2016-07-09T20:27:09Z

Retest this please.

rxin · 2016-07-09T21:05:37Z

...ore/src/test/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchemaSuite.scala

mhm the indentation is really weird - i don't think we need to indent each line with one more level ...

rxin · 2016-07-09T21:13:58Z

It doesn't look like we are getting any benefits from column pruning - perhaps we should just do predicate pushdown? The code would be simpler.

rxin · 2016-07-09T21:14:39Z

sql/core/src/main/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchema.scala

and setupTable or registerTable

I used registerTable.

rxin · 2016-07-09T21:17:35Z

This looks pretty good. Can you add more comments explaining what each class/method does, and how the whole thing works?

dongjoon-hyun · 2016-07-09T21:36:41Z

Sure. I'll update the PR and proceed in this way.
Thank you, @rxin .

SparkQA · 2016-07-09T21:53:45Z

Test build #62030 has finished for PR 14116 at commit d9d9344.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-09T21:59:29Z

Test build #62031 has finished for PR 14116 at commit d9d9344.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-10T01:29:58Z

Test build #62038 has finished for PR 14116 at commit b89039d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DatabasesRelationProvider extends SchemaRelationProvider
- class TablesRelationProvider extends SchemaRelationProvider
- class ViewsRelationProvider extends SchemaRelationProvider
- class ColumnsRelationProvider extends SchemaRelationProvider
- class SessionVariablesRelationProvider extends SchemaRelationProvider

SparkQA · 2016-07-10T03:58:29Z

Test build #62040 has finished for PR 14116 at commit 8cb4956.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-10T04:13:05Z

Test build #62041 has finished for PR 14116 at commit a55da04.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-10T05:55:18Z

Test build #62043 has finished for PR 14116 at commit c770315.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-07-10T19:10:31Z

SparkR failures are due to my TODO item.

dongjoon-hyun · 2016-07-10T22:46:10Z

Locally, the last commit passes the R test, too. Now, I think I've finished my first implementation.
While waiting for #14114 and #14115 , I'll move on other new issues.

SparkQA · 2016-07-10T23:59:32Z

Test build #62064 has finished for PR 14116 at commit a645410.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-11T00:36:18Z

Test build #62065 has finished for PR 14116 at commit e6e96eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-07-11T01:51:23Z

Now, all tests are passed.

lw-lin · 2016-07-11T06:49:27Z

sql/core/src/main/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchema.scala

val maybe?

Oh, thank you for review! Right.

gatorsmile · 2016-07-18T05:15:45Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

Can you explain which cases will enter this processing? Is that possible we could hit backtick-quoted INFORMATION_SCHEMA_DATABASE here?

Backtick-quoted one will not reach here.

scala> sql("create table `aaa.bbb`(a int)") org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: [aaa.bbb]: is not a valid table name;

gatorsmile · 2016-07-18T05:44:43Z

        val catalog = spark.sessionState.catalog
        catalog.setCurrentDatabase(SessionCatalog.INFORMATION_SCHEMA_DATABASE)
        sql("CREATE TABLE my_tab (age INT, name STRING)")

We can set the current database to INFORMATION_SCHEMA_DATABASE, but we got an error when we are trying to create a table in this database.

Database 'information_schema' not found;
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'information_schema' not found;

gatorsmile · 2016-07-18T05:52:42Z

sql/core/src/main/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchema.scala

I am lost. A database INFORMATION_SCHEMA_DATABASE is created through a SQL command, but the function databaseExists always returns true for the database for INFORMATION_SCHEMA_DATABASE.

See the code:
https://github.com/dongjoon-hyun/spark/blob/b04f70127760dde5700a63e7dad4100cf407b863/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L177-L184

gatorsmile · 2016-07-18T06:28:03Z

Finished my first pass. The major concern is handling of INFORMATION_SCHEMA is not clean to me. It looks hacky. Many holes are caused by it. More test cases are needed.

dongjoon-hyun · 2016-07-18T06:59:28Z

Thank you so much, @gatorsmile . And, sorry for late response. Definitely, I have many things to do. Now, it's my turn. Let's see how much I can handle them. :)

SparkQA · 2016-07-22T08:57:34Z

Test build #62721 has finished for PR 14116 at commit eddaec6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-26T19:08:07Z

Test build #62897 has finished for PR 14116 at commit 72bc2dd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-30T00:18:53Z

Test build #63023 has finished for PR 14116 at commit 0fc02f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-08-08T07:58:44Z

Rebased to the master because CatalogColumn was removed from master.

SparkQA · 2016-08-08T09:59:49Z

Test build #63351 has finished for PR 14116 at commit 9ad03d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-12T00:20:55Z

Test build #63641 has finished for PR 14116 at commit dc5d1dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-14T21:44:14Z

Test build #63757 has finished for PR 14116 at commit e9302e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-19T06:31:22Z

Test build #64041 has finished for PR 14116 at commit bd85aa5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-24T17:28:37Z

Test build #64358 has finished for PR 14116 at commit 9bb92bf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-27T11:50:30Z

Test build #64530 has finished for PR 14116 at commit 5704e83.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-29T19:24:21Z

Test build #64577 has finished for PR 14116 at commit 7543069.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-02T22:59:00Z

Test build #64876 has finished for PR 14116 at commit d7bfc7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-07T06:43:55Z

Test build #65019 has finished for PR 14116 at commit d107721.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-09T08:38:39Z

Test build #65141 has finished for PR 14116 at commit a8c30c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-12T10:46:38Z

Test build #65250 has finished for PR 14116 at commit c531025.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-15T09:52:04Z

Test build #65431 has finished for PR 14116 at commit 2296c0e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-09-20T16:00:36Z

Rebased to resolve conflicts.

SparkQA · 2016-09-20T17:26:03Z

Test build #65664 has finished for PR 14116 at commit e832f5b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-10-19T10:41:03Z

The issue might be tackled later after Catalog Federation. Now, I close this PR since it's too stale. Thank you all for spending time at this PR.

rxin reviewed Jul 9, 2016
View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support~~ [SPARK-16452][SQL] Basic INFORMATION_SCHEMA support Jul 10, 2016

dongjoon-hyun changed the title ~~[SPARK-16452][SQL] Basic INFORMATION_SCHEMA support~~ [SPARK-16452][SQL] Support basic INFORMATION_SCHEMA Jul 10, 2016

lw-lin reviewed Jul 11, 2016
View reviewed changes

gatorsmile reviewed Jul 18, 2016
View reviewed changes

[SPARK-16452][SQL] Basic INFORMATION_SCHEMA support

e832f5b

dongjoon-hyun closed this Oct 19, 2016

dongjoon-hyun deleted the SPARK-16452 branch January 7, 2019 07:02

[SPARK-16452][SQL] Support basic INFORMATION_SCHEMA #14116

[SPARK-16452][SQL] Support basic INFORMATION_SCHEMA #14116

Uh oh!

Conversation

dongjoon-hyun commented Jul 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Jul 9, 2016

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

dongjoon-hyun commented Jul 9, 2016

Uh oh!

dongjoon-hyun commented Jul 9, 2016

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

dongjoon-hyun commented Jul 9, 2016

Uh oh!

dongjoon-hyun commented Jul 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rxin commented Jul 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rxin commented Jul 9, 2016

Uh oh!

dongjoon-hyun commented Jul 9, 2016

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

SparkQA commented Jul 10, 2016

Uh oh!

SparkQA commented Jul 10, 2016

Uh oh!

SparkQA commented Jul 10, 2016

Uh oh!

SparkQA commented Jul 10, 2016

Uh oh!

dongjoon-hyun commented Jul 10, 2016

Uh oh!

dongjoon-hyun commented Jul 10, 2016

Uh oh!

SparkQA commented Jul 10, 2016

Uh oh!

SparkQA commented Jul 11, 2016

Uh oh!

dongjoon-hyun commented Jul 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jul 18, 2016

Uh oh!

gatorsmile Jul 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jul 18, 2016

Uh oh!

dongjoon-hyun commented Jul 18, 2016

Uh oh!

SparkQA commented Jul 22, 2016

dongjoon-hyun commented Jul 9, 2016 •

edited

Loading

gatorsmile Jul 18, 2016 •

edited

Loading