-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-8844][SPARKR] head/collect is broken in SparkR. #7419
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #37361 has finished for PR 7419 at commit
|
R/pkg/R/deserialize.R
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So fundamentally it looks like this we want to create a vector with mode = colType if the column is empty. But does it matter if we get the colType wrong ? i.e. what happens if we just use the default mode (which I think is logical)
|
cc @davies |
|
@sun-rui Any update on this ? |
|
@shivaram , as you pointed out, I come with a simpler fix. I realized that simply creating an empty vector using vector() without mode is OK when there is no row in columns. Because vector() creates a zero-length vector of type "logical", and "logical" is of virutally lowest precedence in the R's type hierarchy, so it is supposed that "logical" type won't affect later coercion in operations on a data.frame of 0 row, for example, rbind(). |
|
Jenkins, retest this please |
|
Thanks @sun-rui -- this fix is simple and looks good. Couple of minor things
|
|
Test build #40969 has finished for PR 7419 at commit
|
|
@shivaram, fixed. |
|
LGTM |
|
Test build #40978 has finished for PR 7419 at commit
|
|
LGTM. Merging this |
This is a WIP patch for SPARK-8844 for collecting reviews. This bug is about reading an empty DataFrame. in readCol(), lapply(1:numRows, function(x) { does not take into consideration the case where numRows = 0. Will add unit test case. Author: Sun Rui <rui.sun@intel.com> Closes #7419 from sun-rui/SPARK-8844. (cherry picked from commit 5f9ce73) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
This is a WIP patch for SPARK-8844 for collecting reviews.
This bug is about reading an empty DataFrame. in readCol(),
lapply(1:numRows, function(x) {
does not take into consideration the case where numRows = 0.
Will add unit test case.
Author: Sun Rui <rui.sun@intel.com>
Closes apache#7419 from sun-rui/SPARK-8844.
This is a WIP patch for SPARK-8844 for collecting reviews.
This bug is about reading an empty DataFrame. in readCol(),
lapply(1:numRows, function(x) {
does not take into consideration the case where numRows = 0.
Will add unit test case.