[SPARK-17442][SPARKR] Additional arguments in write.df are not passed to data source #15010
Conversation
| # writing csv file | ||
| csvPath2 <- tempfile(pattern = "csvtest2", fileext = ".csv") | ||
| write.df(df2, path = csvPath2, "csv", header = "true") | ||
| df3 <- read.df(csvPath2, "csv", header = "true") |
There was a problem hiding this comment.
We could do this - but I was thinking we could also check if R's read.csv is able to read back the file correctly with headers ?
There was a problem hiding this comment.
need full path to work
> read.csv(file = csvPath2)
Error in read.table(file = file, header = header, sep = sep, quote = quote, :
no lines available in input
> read.csv(file = paste0(csvPath2, "/", "part-r-00000-bf045be1-500f-4e77-8957-b6d256166ca7.csv"))
year make model comment blank
1 2012 Tesla S No comment Empty
2 1997 Ford E350 Go get one now they are going fast Empty
3 2015 Chevy Volt
4 NA Dummy Placeholder
There was a problem hiding this comment.
The path is interpreted as a directory by write.df. It then puts in a part-0000 or a sequence of such files inside the directory
There was a problem hiding this comment.
right, read.csv doesn't work with wildcard it seems.
testing a fix.
|
Test build #65077 has finished for PR 15010 at commit
|
|
Thanks - the new test looks good |
|
Test build #65080 has finished for PR 15010 at commit
|
|
LGTM - @falaki any more comments ? |
|
Hey just saw. This looks good to me too. |
|
Merging into master, branch-2.0 |
… to data source ## What changes were proposed in this pull request? additional options were not passed down in write.df. ## How was this patch tested? unit tests falaki shivaram Author: Felix Cheung <felixcheung_m@hotmail.com> Closes #15010 from felixcheung/testreadoptions. (cherry picked from commit f0d21b7) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
4 participants
What changes were proposed in this pull request?
additional options were not passed down in write.df.
How was this patch tested?
unit tests
@falaki @shivaram