Skip to content

Conversation

@wangyum
Copy link
Member

@wangyum wangyum commented Aug 30, 2018

What changes were proposed in this pull request?

This pr fix FileFormatWriter's dataSchema should respect the schema of Hive. Otherwise there will be two issues.

  1. Throwing an exception(This can be reproduce by added test case):
java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:347)
	at scala.None$.get(Option.scala:345)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$3$$anonfun$4.apply(FileFormatWriter.scala:87)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$3$$anonfun$4.apply(FileFormatWriter.scala:87)
  1. The schema of the Hive table is not the same as the schema of the parquet file.

How was this patch tested?

  • Unit tests for FileFormatWriter should respect the schema of Hive.
  • Manual tests for didn't break UI issues fixed by SPARK-22834:
    image

@SparkQA
Copy link

SparkQA commented Aug 30, 2018

Test build #95483 has finished for PR 22287 at commit b54953a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants