[SPARK-17899][SQL] add a debug mode to keep raw table properties in HiveExternalCatalog #15458
Conversation
|
cc @yhuai |
|
LGTM pending Jenkins. |
| checkAnswer( | ||
| newSession.sql(s"SET ${SCHEMA_STRING_LENGTH_THRESHOLD.key}"), | ||
| Row(SCHEMA_STRING_LENGTH_THRESHOLD.key, "2000")) | ||
| test("static SQL conf comes from SparkConf") { |
There was a problem hiding this comment.
this change is not related to this PR.
SparkSession.builder()....getOrCreate() may not set the static SQL conf if there is an active or global default session. To make this test more robust, I explicitly create a new SparkSession.
This is the same approach I used in the new test, so I include this unrelated change in this PR.
|
Test build #66871 has finished for PR 15458 at commit
|
|
Cool beans. Merging into master 2.0. |
|
JK, actually it doesn't merge in 2.0. Probably OK to just leave it in master. |
|
If we turn this flag on, I am just afraid that some DDL statements might not work well. I assume this flag is really used for the our Spark SQL developers. |
|
Yes that's why it's |
|
Just FYI. #15478 shows at least three DDL commands do not work properly when |
|
I see. Then maybe we should add a comment above the config to note that several commands don't work (e.g. ALTER TABLE) if this is turned on, even if it's only internal. |
| val DEBUG_MODE = buildConf("spark.sql.debug") | ||
| .internal() | ||
| .booleanConf | ||
| .createWithDefault(false) |
There was a problem hiding this comment.
Let's add doc for this flag.
There was a problem hiding this comment.
Sure, I can do a quick PR to fix it.
…iveExternalCatalog ## What changes were proposed in this pull request? Currently `HiveExternalCatalog` will filter out the Spark SQL internal table properties, e.g. `spark.sql.sources.provider`, `spark.sql.sources.schema`, etc. This is reasonable for external users as they don't want to see these internal properties in `DESC TABLE`. However, as a Spark developer, sometimes we do wanna see the raw table properties. This PR adds a new internal SQL conf, `spark.sql.debug`, to enable debug mode and keep these raw table properties. This config can also be used in similar places where we wanna retain debug information in the future. ## How was this patch tested? new test in MetastoreDataSourcesSuite Author: Wenchen Fan <wenchen@databricks.com> Closes apache#15458 from cloud-fan/debug.
6 participants
What changes were proposed in this pull request?
Currently
HiveExternalCatalogwill filter out the Spark SQL internal table properties, e.g.spark.sql.sources.provider,spark.sql.sources.schema, etc. This is reasonable for external users as they don't want to see these internal properties inDESC TABLE.However, as a Spark developer, sometimes we do wanna see the raw table properties. This PR adds a new internal SQL conf,
spark.sql.debug, to enable debug mode and keep these raw table properties.This config can also be used in similar places where we wanna retain debug information in the future.
How was this patch tested?
new test in MetastoreDataSourcesSuite