[SPARK-9929][SQL] support metadata in withColumn #8159
Conversation
|
Test build #40732 has finished for PR 8159 at commit
|
|
The ML changes look good. (Thanks for adding this.) The new method looks fine, unless you want to reduce code duplication. |
|
Does the new function needs to be public? Can it be private[spark] ? |
|
ML does not need it to be public. |
|
OK @cloud-fan let's make this private[spark] for now. |
|
Test build #40841 has finished for PR 8159 at commit
|
|
The API change LGTM. |
There was a problem hiding this comment.
should we make Column.as(alias: String, metadata: Metadata) also private? As it expose the Metadata too.
There was a problem hiding this comment.
Don't think we can change that since it's been public since 1.3.
|
Test build #40843 has finished for PR 8159 at commit
|
|
LGTM |
|
Merging this in master (not branch-1.5). |
in MLlib sometimes we need to set metadata for the new column, thus we will alias the new column with metadata before call `withColumn` and in `withColumn` we alias this clolumn again. Here I overloaded `withColumn` to allow user set metadata, just like what we did for `Column.as`. Author: Wenchen Fan <cloud0fan@outlook.com> Closes apache#8159 from cloud-fan/withColumn.
4 participants
in MLlib sometimes we need to set metadata for the new column, thus we will alias the new column with metadata before call
withColumnand inwithColumnwe alias this clolumn again. Here I overloadedwithColumnto allow user set metadata, just like what we did forColumn.as.