Skip to content

[SPARK-7438][Spark Core]Fixed validation of relativeSD in countApproxDistinct #5974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vinodkc wants to merge 5 commits into from
Closed

[SPARK-7438][Spark Core]Fixed validation of relativeSD in countApproxDistinct #5974

vinodkc wants to merge 5 commits into from

Conversation

@vinodkc
Copy link
Contributor

@vinodkc vinodkc commented May 7, 2015

No description provided.

@AmplabJenkins
Copy link

AmplabJenkins commented May 7, 2015

Can one of the admins verify this patch?

srowen
srowen reviewed May 7, 2015
View reviewed changes
core/src/main/scala/org/apache/spark/rdd/RDD.scala
Copy link
Member

@srowen srowen May 7, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems OK, in that it enforces the contract in the docs, and then avoids an error in a case that does not seem to be an error. @rxin do you have any thoughts on this one?

@srowen
Copy link
Member

srowen commented May 7, 2015

ok to test

@AmplabJenkins
Copy link

AmplabJenkins commented May 7, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 7, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 7, 2015

Test build #32107 has started for PR 5974 at commit 122d378.

@srowen
Copy link
Member

srowen commented May 7, 2015

Ah, I see these checks mirror those in countApproxDistinctByKey. OK.
But they're slightly different than the essentially-equivalent check in Python:

if relativeSD > 0.37:
            raise ValueError("relativeSD should be smaller than 0.37")

I still favor it as it makes the Scala impls consistent. I wonder if Python should just check the same thing since the docs don't say it rejects relativeSD > 0.37

CC @davies

@SparkQA
Copy link

SparkQA commented May 7, 2015

Test build #32107 has finished for PR 5974 at commit 122d378.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 7, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 7, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32107/
Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32184 has started for PR 5974 at commit b1b00a3.

@vinodkc
Copy link
Contributor Author

vinodkc commented May 8, 2015

Removed relativeSD validation in python.Validation is done in scala

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32185 has started for PR 5974 at commit 8ddbfae.

@rxin
Copy link
Contributor

rxin commented May 8, 2015

LGTM.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32184 has finished for PR 5974 at commit b1b00a3.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32184/
Test FAILed.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32185 has finished for PR 5974 at commit 8ddbfae.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32185/
Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32201 has started for PR 5974 at commit 799976e.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32201 has finished for PR 5974 at commit 799976e.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32201/
Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32212 has started for PR 5974 at commit 3a3d59c.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32212 has finished for PR 5974 at commit 3a3d59c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 8, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32212/
Test PASSed.

@srowen
Copy link
Member

srowen commented May 8, 2015

This LGTM. Will leave it a day or two for comments, but the logic seems sound. this improves consistency, behavior vs docs, and avoids an avoidable error

asfgit pushed a commit that referenced this pull request May 9, 2015
@srowen
[SPARK-7438] [SPARK CORE] Fixed validation of relativeSD in countAppr…
b0460f4 
…oxDistinct

Author: Vinod K C <vinod.kc@huawei.com>

Closes #5974 from vinodkc/fix_countApproxDistinct_Validation and squashes the following commits:

3a3d59c [Vinod K C] Reverted removal of validation relativeSD<0.000017
799976e [Vinod K C] Removed testcase to assert IAE when relativeSD>3.7
8ddbfae [Vinod K C] Remove blank line
b1b00a3 [Vinod K C] Removed relativeSD validation from python API,RDD.scala will do validation
122d378 [Vinod K C] Fixed validation of relativeSD in  countApproxDistinct

(cherry picked from commit dda6d9f)
Signed-off-by: Sean Owen <sowen@cloudera.com>
@asfgit asfgit closed this in dda6d9f May 9, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@jeanlyn
[SPARK-7438] [SPARK CORE] Fixed validation of relativeSD in countAppr…
125add1 
…oxDistinct

Author: Vinod K C <vinod.kc@huawei.com>

Closes apache#5974 from vinodkc/fix_countApproxDistinct_Validation and squashes the following commits:

3a3d59c [Vinod K C] Reverted removal of validation relativeSD<0.000017
799976e [Vinod K C] Removed testcase to assert IAE when relativeSD>3.7
8ddbfae [Vinod K C] Remove blank line
b1b00a3 [Vinod K C] Removed relativeSD validation from python API,RDD.scala will do validation
122d378 [Vinod K C] Fixed validation of relativeSD in  countApproxDistinct
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@jeanlyn
[SPARK-7438] [SPARK CORE] Fixed validation of relativeSD in countAppr…
9347b06 
…oxDistinct

Author: Vinod K C <vinod.kc@huawei.com>

Closes apache#5974 from vinodkc/fix_countApproxDistinct_Validation and squashes the following commits:

3a3d59c [Vinod K C] Reverted removal of validation relativeSD<0.000017
799976e [Vinod K C] Removed testcase to assert IAE when relativeSD>3.7
8ddbfae [Vinod K C] Remove blank line
b1b00a3 [Vinod K C] Removed relativeSD validation from python API,RDD.scala will do validation
122d378 [Vinod K C] Fixed validation of relativeSD in  countApproxDistinct
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
[SPARK-7438] [SPARK CORE] Fixed validation of relativeSD in countAppr…
e0b38b7 
…oxDistinct

Author: Vinod K C <vinod.kc@huawei.com>

Closes apache#5974 from vinodkc/fix_countApproxDistinct_Validation and squashes the following commits:

3a3d59c [Vinod K C] Reverted removal of validation relativeSD<0.000017
799976e [Vinod K C] Removed testcase to assert IAE when relativeSD>3.7
8ddbfae [Vinod K C] Remove blank line
b1b00a3 [Vinod K C] Removed relativeSD validation from python API,RDD.scala will do validation
122d378 [Vinod K C] Fixed validation of relativeSD in  countApproxDistinct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@vinodkc @AmplabJenkins @srowen @SparkQA @rxin