[SPARK-51281][SQL][3.5] DataFrameWriterV2 should respect the path option #50179
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Unlike `DataFrameWriter.saveAsTable` where we explicitly get the "path" option and treat it as table location, `DataFrameWriterV2` doesn't do it and treats the "path" option as a normal option which doesn't have any real impact. This PR fixes it, and adds a legacy config to restore the old behavior. bug fix Yes, now `DataFrameWriterV2` can correctly write data to the specified path for file source tables. new test no Closes apache#50040 from cloud-fan/prop. Lead-authored-by: Wenchen Fan <cloud0fan@gmail.com> Co-authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit a3671e5) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you for making a PR again. Could you make CI happy?
Contributor
Author
|
It seems like a bug in sbt or sbtunidoc and it tries to generate API doc for generated java classes in the |
cloud-fan
commented
Mar 14, 2025
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala
Outdated
Show resolved
Hide resolved
dongjoon-hyun
approved these changes
Mar 15, 2025
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @cloud-fan .
Merged to branch-3.5.
dongjoon-hyun
pushed a commit
that referenced
this pull request
Mar 15, 2025
backport #50040 to 3.5 ### What changes were proposed in this pull request? Unlike `DataFrameWriter.saveAsTable` where we explicitly get the "path" option and treat it as table location, `DataFrameWriterV2` doesn't do it and treats the "path" option as a normal option which doesn't have any real impact. This PR fixes it, and adds a legacy config to restore the old behavior. ### Why are the changes needed? bug fix ### Does this PR introduce _any_ user-facing change? Yes, now `DataFrameWriterV2` can correctly write data to the specified path for file source tables. ### How was this patch tested? new test ### Was this patch authored or co-authored using generative AI tooling? no Closes #50179 from cloud-fan/backport. Authored-by: Wenchen Fan <cloud0fan@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
backport #50040 to 3.5
What changes were proposed in this pull request?
Unlike
DataFrameWriter.saveAsTablewhere we explicitly get the "path" option and treat it as table location,DataFrameWriterV2doesn't do it and treats the "path" option as a normal option which doesn't have any real impact.This PR fixes it, and adds a legacy config to restore the old behavior.
Why are the changes needed?
bug fix
Does this PR introduce any user-facing change?
Yes, now
DataFrameWriterV2can correctly write data to the specified path for file source tables.How was this patch tested?
new test
Was this patch authored or co-authored using generative AI tooling?
no