Skip to content

[SPARK-3130][MLLIB] detect negative values in naive Bayes#2038

Closed
mengxr wants to merge 2 commits intoapache:masterfrom
mengxr:nb-neg
Closed

[SPARK-3130][MLLIB] detect negative values in naive Bayes#2038
mengxr wants to merge 2 commits intoapache:masterfrom
mengxr:nb-neg

Conversation

@mengxr
Copy link
Contributor

@mengxr mengxr commented Aug 19, 2014

because NB treats feature values as term frequencies. @jkbradley

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have started for PR 2038 at commit 65f892d.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have finished for PR 2038 at commit 65f892d.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

jkbradley
jkbradley reviewed Aug 19, 2014
View reviewed changes
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala Outdated
Copy link
Member

@jkbradley jkbradley Aug 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could shorten:
if (!values.forall(_ >= 0.0)) {

@jkbradley
Copy link
Member

jkbradley commented Aug 19, 2014

LGTM. The function requireNonnegativeValues() actually also identifies NaN values, so you could add a test in the Suite for that. (Same as the test("detect negative values") but with -1.0 replaced with 0.0/0.0)

jkbradley
jkbradley reviewed Aug 19, 2014
View reviewed changes
docs/mllib-naive-bayes.md
Copy link
Member

@jkbradley jkbradley Aug 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is really an unnormalized frequency? Should that be clarified by saying "unnormalized frequency" or "frequency or count" ?

Copy link
Contributor Author

@mengxr mengxr Aug 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be normalized, then user needs to adjust alpha.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have started for PR 2038 at commit 52c37c3.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have finished for PR 2038 at commit 52c37c3.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@mengxr
Copy link
Contributor Author

mengxr commented Aug 19, 2014

Jenkins, retest this please.

1 similar comment
@mengxr
Copy link
Contributor Author

mengxr commented Aug 19, 2014

Jenkins, retest this please.

@jkbradley
Copy link
Member

jkbradley commented Aug 19, 2014

LGTM

@mengxr
Copy link
Contributor Author

mengxr commented Aug 19, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have started for PR 2038 at commit 52c37c3.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have finished for PR 2038 at commit 52c37c3.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@mengxr
Copy link
Contributor Author

mengxr commented Aug 20, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Aug 20, 2014

QA tests have started for PR 2038 at commit 52c37c3.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 20, 2014

QA tests have finished for PR 2038 at commit 52c37c3.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Aug 20, 2014
@mengxr
[SPARK-3130][MLLIB] detect negative values in naive Bayes
148e45b 
because NB treats feature values as term frequencies. jkbradley

Author: Xiangrui Meng <meng@databricks.com>

Closes #2038 from mengxr/nb-neg and squashes the following commits:

52c37c3 [Xiangrui Meng] address comments
65f892d [Xiangrui Meng] detect negative values in nb

(cherry picked from commit 068b6fe)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
@asfgit asfgit closed this in 068b6fe Aug 20, 2014
@mengxr
Copy link
Contributor Author

mengxr commented Aug 20, 2014

Merged into master and branch-1.1.

xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@mengxr @conviva-zz
[SPARK-3130][MLLIB] detect negative values in naive Bayes
2e6d945 
because NB treats feature values as term frequencies. jkbradley

Author: Xiangrui Meng <meng@databricks.com>

Closes apache#2038 from mengxr/nb-neg and squashes the following commits:

52c37c3 [Xiangrui Meng] address comments
65f892d [Xiangrui Meng] detect negative values in nb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mengxr @SparkQA @jkbradley