Skip to content

[SPARK-7447][SQL] Don't re-merge Parquet schema when the relation is deserialized #6012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
viirya wants to merge 3 commits into from
Closed

[SPARK-7447][SQL] Don't re-merge Parquet schema when the relation is deserialized #6012

viirya wants to merge 3 commits into from

Conversation

@viirya
Copy link
Member

@viirya viirya commented May 8, 2015

JIRA: https://issues.apache.org/jira/browse/SPARK-7447

MetadataCache in ParquetRelation2 is annotated as @transient. When ParquetRelation2 is deserialized, we ask MetadataCache to refresh and perform schema merging again. It is time-consuming especially for very many parquet files.

With the new FSBasedParquetRelation, although MetadataCache is not @transient now, MetadataCache.refresh() still performs schema merging again when the relation is deserialized.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32246 has finished for PR 6012 at commit b0fc09b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@scwf
Copy link
Contributor

scwf commented May 9, 2015

good catch

@marmbrus
Copy link
Contributor

marmbrus commented May 9, 2015

I think this codepath is going to be deleted, but we should make sure that #5526 does not suffer from the same issue.

/cc @liancheng

@viirya
Copy link
Member Author

viirya commented May 10, 2015

@marmbrus After checking #5526, I think this part of codes is still be there to use. So maybe this pr is still needed.

@marmbrus
Copy link
Contributor

marmbrus commented May 10, 2015

Yes, but look at the linked PR that replaces the parquet support with this
new API.
On May 9, 2015 7:18 PM, "Liang-Chi Hsieh" notifications@github.com wrote:

@marmbrus https://github.com/marmbrus After checking #5526
#5526, I think this part of codes
is still be there to use. So maybe this pr is still needed.

—
Reply to this email directly or view it on GitHub
#6012 (comment).

@viirya
Copy link
Member Author

viirya commented May 10, 2015

OK. I see. You meant liancheng#6.

@marmbrus
Copy link
Contributor

marmbrus commented May 10, 2015

Yep!
On May 9, 2015 7:46 PM, "Liang-Chi Hsieh" notifications@github.com wrote:

OK. I see. You meant liancheng#6
liancheng#6.

—
Reply to this email directly or view it on GitHub
#6012 (comment).

@viirya
Merge remote-tracking branch 'upstream/master' into without_remerge_s…
6ac7d93 
…chema

Conflicts:
	sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala
@viirya
Copy link
Member Author

viirya commented May 14, 2015

@marmbrus #6090 was merged. But this problem is still there. I updated the codes. Please take a look. Thanks.

@marmbrus
Copy link
Contributor

marmbrus commented May 14, 2015

LGTM

/cc @liancheng

@SparkQA
Copy link

SparkQA commented May 14, 2015

Test build #32691 has finished for PR 6012 at commit 6ac7d93.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@liancheng
Copy link
Contributor

liancheng commented May 17, 2015

@viirya Sorry for the late reply, I missed this PR while working on #6090. This fix LGTM, could you please rebase this PR? Notice that fsBasedParquet.scala was renamed back to newParquet.scala since the old ParquetRelation2 implementation was finally removed.

@viirya
Merge remote-tracking branch 'upstream/master' into without_remerge_s…
2663957 
…chema

Conflicts:
	sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala
@viirya
Copy link
Member Author

viirya commented May 17, 2015

@liancheng updated. Please take a look.

@SparkQA
Copy link

SparkQA commented May 17, 2015

Test build #32927 has finished for PR 6012 at commit 2663957.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@liancheng
Copy link
Contributor

liancheng commented May 17, 2015

Thanks for fixing this! Merging to master and branch-1.4.

asfgit pushed a commit that referenced this pull request May 17, 2015
@viirya @liancheng
[SPARK-7447] [SQL] Don't re-merge Parquet schema when the relation is…
898be62 
… deserialized

JIRA: https://issues.apache.org/jira/browse/SPARK-7447

`MetadataCache` in `ParquetRelation2` is annotated as `transient`. When `ParquetRelation2` is deserialized, we ask `MetadataCache` to refresh and perform schema merging again. It is time-consuming especially for very many parquet files.

With the new `FSBasedParquetRelation`, although `MetadataCache` is not `transient` now, `MetadataCache.refresh()` still performs schema merging again when the relation is deserialized.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes #6012 from viirya/without_remerge_schema and squashes the following commits:

2663957 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into without_remerge_schema
6ac7d93 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into without_remerge_schema
b0fc09b [Liang-Chi Hsieh] Don't generate and merge parquetSchema multiple times.

(cherry picked from commit 3399055)
Signed-off-by: Cheng Lian <lian@databricks.com>
@asfgit asfgit closed this in 3399055 May 17, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@viirya @jeanlyn
[SPARK-7447] [SQL] Don't re-merge Parquet schema when the relation is…
81ea7f4 
… deserialized

JIRA: https://issues.apache.org/jira/browse/SPARK-7447

`MetadataCache` in `ParquetRelation2` is annotated as `transient`. When `ParquetRelation2` is deserialized, we ask `MetadataCache` to refresh and perform schema merging again. It is time-consuming especially for very many parquet files.

With the new `FSBasedParquetRelation`, although `MetadataCache` is not `transient` now, `MetadataCache.refresh()` still performs schema merging again when the relation is deserialized.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes apache#6012 from viirya/without_remerge_schema and squashes the following commits:

2663957 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into without_remerge_schema
6ac7d93 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into without_remerge_schema
b0fc09b [Liang-Chi Hsieh] Don't generate and merge parquetSchema multiple times.
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@viirya @jeanlyn
[SPARK-7447] [SQL] Don't re-merge Parquet schema when the relation is…
1dac0fe 
… deserialized

JIRA: https://issues.apache.org/jira/browse/SPARK-7447

`MetadataCache` in `ParquetRelation2` is annotated as `transient`. When `ParquetRelation2` is deserialized, we ask `MetadataCache` to refresh and perform schema merging again. It is time-consuming especially for very many parquet files.

With the new `FSBasedParquetRelation`, although `MetadataCache` is not `transient` now, `MetadataCache.refresh()` still performs schema merging again when the relation is deserialized.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes apache#6012 from viirya/without_remerge_schema and squashes the following commits:

2663957 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into without_remerge_schema
6ac7d93 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into without_remerge_schema
b0fc09b [Liang-Chi Hsieh] Don't generate and merge parquetSchema multiple times.
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@viirya @nemccarthy
[SPARK-7447] [SQL] Don't re-merge Parquet schema when the relation is…
7346586 
… deserialized

JIRA: https://issues.apache.org/jira/browse/SPARK-7447

`MetadataCache` in `ParquetRelation2` is annotated as `transient`. When `ParquetRelation2` is deserialized, we ask `MetadataCache` to refresh and perform schema merging again. It is time-consuming especially for very many parquet files.

With the new `FSBasedParquetRelation`, although `MetadataCache` is not `transient` now, `MetadataCache.refresh()` still performs schema merging again when the relation is deserialized.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes apache#6012 from viirya/without_remerge_schema and squashes the following commits:

2663957 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into without_remerge_schema
6ac7d93 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into without_remerge_schema
b0fc09b [Liang-Chi Hsieh] Don't generate and merge parquetSchema multiple times.
@liancheng liancheng mentioned this pull request Jul 22, 2015
Closed
@viirya viirya deleted the without_remerge_schema branch December 27, 2023 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@viirya @SparkQA @scwf @marmbrus @liancheng