[SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadata from SQLConf and docs #19129

dongjoon-hyun · 2017-09-05T07:40:38Z

What changes were proposed in this pull request?

Since SPARK-15639, spark.sql.parquet.cacheMetadata and PARQUET_CACHE_METADATA is not used. This PR removes from SQLConf and docs.

How was this patch tested?

Pass the existing Jenkins.

…f and docs

maropu · 2017-09-05T10:10:16Z

I roughly checked other options around parquet and I probably found parquetOutputCommitterClass in SQLConf also is not used now? If yes, it seems we have no jira entry for the option?

SparkQA · 2017-09-05T10:21:57Z

Test build #81401 has finished for PR 19129 at commit 3b305d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-09-05T10:56:34Z

Thank you, @maropu !
spark.sql.parquet.output.committer.class seems to be used ParquetIOSuite.scala.

maropu · 2017-09-05T10:58:49Z

oh, yea. I got you. Thanks!

dongjoon-hyun · 2017-09-05T15:29:52Z

Thank you for your review and approval, @HyukjinKwon !

gatorsmile · 2017-09-05T18:20:58Z

Could you check the change history and find when we removed the usage of this SQLConf? It sounds like we did not have a test case coverage for this in the past. We did not realize it when removing the usage. We also need to update the migration notes.

dongjoon-hyun · 2017-09-05T18:30:54Z

Sure, I'll.

HyukjinKwon · 2017-09-05T22:48:40Z

The last code looks removed in 678b96e and this option looks introduced in 9eb74c7.

dongjoon-hyun · 2017-09-05T23:16:42Z

Wow! Thank you, @HyukjinKwon !

gatorsmile · 2017-09-07T16:52:21Z

Please document it in the migration guides. Thanks!

dongjoon-hyun · 2017-09-07T16:59:15Z

Sure, @gatorsmile .
BTW, I searched more and updated the PR description.

It's SPARK-15639

dongjoon-hyun · 2017-09-07T17:08:22Z

It's marked as 2.0.1 and 2.1.0 with the following commit logs.

branch-2.0$ git log --oneline | grep SPARK-15639
977fbbfcae [SPARK-15639] [SPARK-16321] [SQL] Push down filter at RowGroups level for parquet reader
91dffcabde Revert "[SPARK-15639][SQL] Try to push down filter at RowGroups level for parquet reader"
7d6bd11964 [SPARK-15639][SQL] Try to push down filter at RowGroups level for parquet reader

Which section is proper?

Upgrading From Spark SQL 1.6 to 2.0
Upgrading From Spark SQL 2.0 to 2.1

I think it's Upgrading From Spark SQL 1.6 to 2.0, effectively.

dongjoon-hyun · 2017-09-07T17:15:08Z

Or, should I make Upgrading From Spark SQL 2.2 to 2.3?

gatorsmile · 2017-09-07T17:19:52Z

SQL 1.6 to 2.0 sounds good to me.

dongjoon-hyun · 2017-09-07T17:25:27Z

Thank you!

dongjoon-hyun · 2017-09-07T17:41:46Z

The PR title resolved two issues under title [SPARK-15639][SPARK-16321][SQL] Push down filter at RowGroups level for parquet reader I'll add like the following. Is it enough?

 - From Spark 2.0.1, `spark.sql.parquet.cacheMetadata` is no longer used. See
   [SPARK-16321](https://issues.apache.org/jira/browse/SPARK-16321) and
   [SPARK-15639](https://issues.apache.org/jira/browse/SPARK-15639) for details.

gatorsmile · 2017-09-07T18:15:45Z

docs/sql-programming-guide.md


+ - From Spark 2.0.1, `spark.sql.parquet.cacheMetadata` is no longer used. See
+   [SPARK-16321](https://issues.apache.org/jira/browse/SPARK-16321) and
+   [SPARK-15639](https://issues.apache.org/jira/browse/SPARK-15639) for details.


These two jiras are wrong.

#13701 is [SPARK-15639][SPARK-16321][SQL] Push down filter at RowGroups level for parquet reader.

It's removed here.

https://github.com/apache/spark/pull/13701/files#diff-ee26d4c4be21e92e92a02e9f16dbc285L625

There is no caller for initializeLocalJobFunc . Thus, initializeLocalJobFunc is a dead code.

Oh, then, it's another transitive search.

It sounds like https://issues.apache.org/jira/browse/SPARK-13664 is the one that removes the usage of this conf.

I will update like this.

- `spark.sql.parquet.cacheMetadata` is no longer used. See [SPARK-13664](https://issues.apache.org/jira/browse/SPARK-13664) for details.

Hi, I'm new to spark. I wonder how to disable metadata caching after deleting this conf. I created an external table, and the parquet files in specified location are updated daily, So I want to disable metadata caching rather than executing 'refresh table xxx'.

Hi, @zzl1787 . This is Apache Spark 2.3. In Apache Spark 2.3, the metadata cache is not controlled by this parameter.

@dongjoon-hyun Ok, got this, and thank you. Finally I find the parameter to control this.
spark.sql.filesourceTableRelationCacheSize = 0
This will disable the metadata cache.

SparkQA · 2017-09-07T20:38:16Z

Test build #81523 has finished for PR 19129 at commit 40ed9ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-07T21:17:12Z

Test build #81525 has finished for PR 19129 at commit 8e3d8fe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-09-07T23:27:37Z

Thanks! Merged to master.

dongjoon-hyun · 2017-09-07T23:55:11Z

Thank you for review, @gatorsmile , @HyukjinKwon , @maropu .
In this issue, I've learned how to track the unused stuff correctly. Thank you again.

[SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadata from SQLCon…

3b305d0

…f and docs

HyukjinKwon approved these changes Sep 5, 2017

View reviewed changes

Add to migraion guide

40ed9ff

gatorsmile reviewed Sep 7, 2017

View reviewed changes

Fix JIRA pointer.

8e3d8fe

asfgit closed this in e00f1a1 Sep 7, 2017

dongjoon-hyun deleted the SPARK-13656 branch September 7, 2017 23:55

[SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadata from SQLConf and docs #19129

[SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadata from SQLConf and docs #19129

Uh oh!

Conversation

dongjoon-hyun commented Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

maropu commented Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Sep 5, 2017

Uh oh!

dongjoon-hyun commented Sep 5, 2017

Uh oh!

maropu commented Sep 5, 2017

Uh oh!

dongjoon-hyun commented Sep 5, 2017

Uh oh!

gatorsmile commented Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Sep 5, 2017

Uh oh!

HyukjinKwon commented Sep 5, 2017

Uh oh!

dongjoon-hyun commented Sep 5, 2017

Uh oh!

gatorsmile commented Sep 7, 2017

Uh oh!

dongjoon-hyun commented Sep 7, 2017

Uh oh!

dongjoon-hyun commented Sep 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Sep 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Sep 7, 2017

Uh oh!

dongjoon-hyun commented Sep 7, 2017

Uh oh!

dongjoon-hyun commented Sep 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Sep 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 7, 2017

Uh oh!

SparkQA commented Sep 7, 2017

Uh oh!

gatorsmile commented Sep 7, 2017

Uh oh!

dongjoon-hyun commented Sep 7, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

dongjoon-hyun commented Sep 5, 2017 •

edited

Loading

maropu commented Sep 5, 2017 •

edited

Loading

gatorsmile commented Sep 5, 2017 •

edited

Loading

dongjoon-hyun commented Sep 7, 2017 •

edited

Loading

dongjoon-hyun commented Sep 7, 2017 •

edited

Loading

dongjoon-hyun commented Sep 7, 2017 •

edited

Loading

dongjoon-hyun Sep 7, 2017 •

edited

Loading