[SPARK-15804][SQL]Include metadata in the toStructType #13555
Closed
+16
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
get latest code from upstream
adding trim characters support
get latest code for pr12646
merge latest code
merge upstream/master
|
|
||
| test("SPARK-15804: write out the metadata to parquet file") { | ||
| val data = (1, "abc") ::(2, "helloabcde") :: Nil | ||
| val df = spark.createDataFrame(data).toDF("a", "b") |
Contributor
There was a problem hiding this comment.
Merge the lines 630 and 631 to Seq((1,"abc"),(2,"hello")).toDF("a", "b") instead
Contributor
Author
There was a problem hiding this comment.
sure, I will do that.
Member
|
Actually I already did this in #13371. |
Contributor
|
ok to test |
| val dfWithmeta = df.select('a, 'b.as("b", md)) | ||
|
|
||
| withTempPath { dir => | ||
| val path = s"${dir.getCanonicalPath}/data" |
Contributor
There was a problem hiding this comment.
I think dir.getCanonicalPath is good here.
Contributor
Author
There was a problem hiding this comment.
ok, I will make change
|
Test build #60200 has finished for PR 13555 at commit
|
|
Test build #60211 has finished for PR 13555 at commit
|
| val dfWithmeta = df.select('a, 'b.as("b", md)) | ||
|
|
||
| withTempPath { dir => | ||
| val path = s"${dir.getCanonicalPath}" |
Contributor
There was a problem hiding this comment.
nit: just dir.getCanonicalPath, no need to wrap it inside ""
Contributor
Author
There was a problem hiding this comment.
Done, Thanks very much.
Contributor
|
LGTM, pending jenkins |
|
Test build #60222 has finished for PR 13555 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Jun 9, 2016
## What changes were proposed in this pull request? The help function 'toStructType' in the AttributeSeq class doesn't include the metadata when it builds the StructField, so it causes this reported problem https://issues.apache.org/jira/browse/SPARK-15804?jql=project%20%3D%20SPARK when spark writes the the dataframe with the metadata to the parquet datasource. The code path is when spark writes the dataframe to the parquet datasource through the InsertIntoHadoopFsRelationCommand, spark will build the WriteRelation container, and it will call the help function 'toStructType' to create StructType which contains StructField, it should include the metadata there, otherwise, we will lost the user provide metadata. ## How was this patch tested? added test case in ParquetQuerySuite.scala (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Author: Kevin Yu <qyu@us.ibm.com> Closes #13555 from kevinyu98/spark-15804. (cherry picked from commit 99386fe) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Contributor
|
thanks, merging to master and 2.0! |
zjffdu
pushed a commit
to zjffdu/spark
that referenced
this pull request
Jun 10, 2016
## What changes were proposed in this pull request? The help function 'toStructType' in the AttributeSeq class doesn't include the metadata when it builds the StructField, so it causes this reported problem https://issues.apache.org/jira/browse/SPARK-15804?jql=project%20%3D%20SPARK when spark writes the the dataframe with the metadata to the parquet datasource. The code path is when spark writes the dataframe to the parquet datasource through the InsertIntoHadoopFsRelationCommand, spark will build the WriteRelation container, and it will call the help function 'toStructType' to create StructType which contains StructField, it should include the metadata there, otherwise, we will lost the user provide metadata. ## How was this patch tested? added test case in ParquetQuerySuite.scala (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Author: Kevin Yu <qyu@us.ibm.com> Closes apache#13555 from kevinyu98/spark-15804.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The help function 'toStructType' in the AttributeSeq class doesn't include the metadata when it builds the StructField, so it causes this reported problem https://issues.apache.org/jira/browse/SPARK-15804?jql=project%20%3D%20SPARK when spark writes the the dataframe with the metadata to the parquet datasource.
The code path is when spark writes the dataframe to the parquet datasource through the InsertIntoHadoopFsRelationCommand, spark will build the WriteRelation container, and it will call the help function 'toStructType' to create StructType which contains StructField, it should include the metadata there, otherwise, we will lost the user provide metadata.
How was this patch tested?
added test case in ParquetQuerySuite.scala
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)