Skip to content

[SPARK-8501] [SQL] Avoids reading schema from empty ORC files (backport to 1.4) #7200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
liancheng wants to merge 2 commits into from
Closed

[SPARK-8501] [SQL] Avoids reading schema from empty ORC files (backport to 1.4) #7200

liancheng wants to merge 2 commits into from

Conversation

@liancheng
Copy link
Contributor

@liancheng liancheng commented Jul 2, 2015

This PR backports #7199 to branch-1.4

@liancheng
Copy link
Contributor Author

liancheng commented Jul 2, 2015

cc @yhuai @zhzhan

@SparkQA
Copy link

SparkQA commented Jul 2, 2015

Test build #36439 has finished for PR 7200 at commit 0fa25af.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zhzhan
Copy link
Contributor

zhzhan commented Jul 2, 2015

@liancheng Because in spark, we will not create the orc file if the record is empty. It is only happens with the ORC file created by hive, right?

Forget it. Your test case answers the question.

zhzhan
zhzhan reviewed Jul 2, 2015
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileOperator.scala Outdated
Copy link
Contributor

@zhzhan zhzhan Jul 2, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In what situation, will the third case happen? If not exist, can we collapse the 2nd and 3rd case?

@zhzhan
Copy link
Contributor

zhzhan commented Jul 2, 2015

some minor comments. Overall, LGTM

@liancheng
Copy link
Contributor Author

liancheng commented Jul 3, 2015

@zhzhan Thanks for the review! Updated.

Actually while writing Javadoc of OrcFileOperator.getFileReader, I feel the semantics of this method is kinda weird and need some refactoring. But let's leave this to a follow-up PR since 1.4.1 RC2 is being cut soon and I'd like to include this one in 1.4.1.

@liancheng liancheng force-pushed the spark-8501-for-1.4 branch from 9538bff to 725e9e3 Compare July 3, 2015 01:38
@SparkQA
Copy link

SparkQA commented Jul 3, 2015

Test build #36455 has finished for PR 7200 at commit 9538bff.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

liancheng
liancheng reviewed Jul 3, 2015
View reviewed changes
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala
Copy link
Contributor Author

@liancheng liancheng Jul 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The numbers were increased to 100 to workaround SPARK-8501. Now it's fixed, so revert them back.

@SparkQA
Copy link

SparkQA commented Jul 3, 2015

Test build #36458 has finished for PR 7200 at commit 725e9e3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@liancheng
Copy link
Contributor Author

liancheng commented Jul 3, 2015

Merging to branch-1.4.

@liancheng liancheng mentioned this pull request Jul 3, 2015
Closed
asfgit pushed a commit that referenced this pull request Jul 3, 2015
@liancheng
[SPARK-8501] [SQL] Avoids reading schema from empty ORC files (backpo…
de08024 
…rt to 1.4)

This PR backports #7199 to branch-1.4

Author: Cheng Lian <lian@databricks.com>

Closes #7200 from liancheng/spark-8501-for-1.4 and squashes the following commits:

725e9e3 [Cheng Lian] Addresses comments
0fa25af [Cheng Lian] Avoids reading schema from empty ORC files
@sarutak
Copy link
Member

sarutak commented Jul 3, 2015

This PR has already merged right?
It's funny that this PR is still open.
@liancheng Mind manually closing?

@liancheng
Copy link
Contributor Author

liancheng commented Jul 4, 2015

@sarutak Thanks for reminding, closing it :)

@liancheng liancheng closed this Jul 4, 2015
@liancheng liancheng deleted the spark-8501-for-1.4 branch July 4, 2015 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liancheng @SparkQA @zhzhan @sarutak