[SPARK-15149][EXAMPLE][DOC] update kmeans example #12925

zhengruifeng · 2016-05-05T07:38:31Z

What changes were proposed in this pull request?

Python example for ml.kmeans already exists, but not included in user guide.
1,small changes like: example_on example_off
2,add it to user guide
3,update examples to directly read datafile

How was this patch tested?

manual tests
`./bin/spark-submit examples/src/main/python/ml/kmeans_example.py

SparkQA · 2016-05-05T07:50:33Z

Test build #57857 has finished for PR 12925 at commit 221ea4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-05T07:55:26Z

Test build #57859 has finished for PR 12925 at commit 5fa7f5f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-05-05T08:52:12Z

@dongjoon-hyun This one as well. Do you mind if I ask your thoughts on the component in the title? Making good examples for PRs will help all other contributers.

zhengruifeng · 2016-05-05T08:59:20Z

@HyukjinKwon ok. I will change them to [EXAMPLE]

MLnick · 2016-05-05T12:49:35Z

@zhengruifeng I prefer the style of bisecting_k_means_example.py ie working with data = spark.read.text("data/mllib/kmeans_data.txt"). Could we harmonize this with that one?

I will comment on #11844 too about harmonzing the Scala examples.

zhengruifeng · 2016-05-05T13:12:12Z

@MLnick Ok. I will update this examples to read the datafile

zhengruifeng · 2016-05-05T13:38:12Z

@MLnick updated. Thanks for your comments.

zhengruifeng · 2016-05-05T13:40:04Z

Oh, I need to update the JavaKMeansExample and KMeansExample

SparkQA · 2016-05-05T13:44:55Z

Test build #57885 has finished for PR 12925 at commit 059a739.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-05T14:05:23Z

Test build #57888 has finished for PR 12925 at commit f9ff25a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sethah · 2016-05-05T20:44:41Z

Ah, I had a PR ready for this but didn't see you had created a Jira for it. I can review.

sethah · 2016-05-05T20:46:08Z

examples/src/main/scala/org/apache/spark/examples/ml/KMeansExample.scala

Can you see my comments about this on #11844 and let me know?

I see it. I will make KMeans examples keep in line with BiKMeans ones

zhengruifeng · 2016-05-08T03:05:17Z

data/mllib/sample_kmeans_data.txt was created in BisectingKMeans examples

SparkQA · 2016-05-08T03:15:24Z

Test build #58084 has finished for PR 12925 at commit d91cbe9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-09T13:30:29Z

Test build #58145 has finished for PR 12925 at commit d5f02c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sethah · 2016-05-09T18:07:22Z

examples/src/main/python/ml/kmeans_example.py

-
-import numpy as np
+# $example on$
 from pyspark.ml.clustering import KMeans, KMeansModel


Don't need to import KMeansModel here.

sethah · 2016-05-09T18:07:39Z

LGTM other than one minor comment and pending #11844

holdenk · 2016-05-09T18:44:15Z

examples/src/main/python/ml/kmeans_example.py

 Run with:
-  bin/spark-submit examples/src/main/python/ml/kmeans_example.py <input> <k>
-
-This example requires NumPy (http://www.numpy.org/).


nit: So I believe this example still requires NumPy even though it isn't explicitly imported (see inside of def toArray called inside of clusterCenters which says it returns a NumPy array).

Thanks. I will revert this removal.

SparkQA · 2016-05-11T01:56:01Z

Test build #58307 has finished for PR 12925 at commit 5020773.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zhengruifeng · 2016-05-11T02:02:08Z

@MLnick Thanks. Updated

SparkQA · 2016-05-11T02:05:48Z

Test build #58309 has finished for PR 12925 at commit f2ff8d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2016-05-11T06:51:41Z

LGTM. I'll merge this once #11844 is merged.

MLnick · 2016-05-11T08:01:10Z

Merged to master and branch-2.0. Thanks!

## What changes were proposed in this pull request? Python example for ml.kmeans already exists, but not included in user guide. 1,small changes like: `example_on` `example_off` 2,add it to user guide 3,update examples to directly read datafile ## How was this patch tested? manual tests `./bin/spark-submit examples/src/main/python/ml/kmeans_example.py Author: Zheng RuiFeng <ruifengz@foxmail.com> Closes #12925 from zhengruifeng/km_pe. (cherry picked from commit 8beae59) Signed-off-by: Nick Pentreath <nickp@za.ibm.com>

zhengruifeng changed the title ~~[SPARK-15149][DOC] include python example for kmeans~~ [SPARK-15149][EXAMPLE] include python example for kmeans May 5, 2016

sethah reviewed May 5, 2016
View reviewed changes

zhengruifeng changed the title ~~[SPARK-15149][EXAMPLE] include python example for kmeans~~ [SPARK-15149][EXAMPLE][DOC] update kmeans example May 7, 2016

use datafile

d91cbe9

zhengruifeng force-pushed the km_pe branch from f9ff25a to d91cbe9 Compare May 8, 2016 03:03

fix one nit

d5f02c6

sethah reviewed May 9, 2016
View reviewed changes

holdenk reviewed May 9, 2016
View reviewed changes

zhengruifeng added 2 commits May 11, 2016 09:43

update comments

5020773

update py cmd

f2ff8d6

asfgit closed this in 8beae59 May 11, 2016

zhengruifeng deleted the km_pe branch May 11, 2016 08:05

[SPARK-15149][EXAMPLE][DOC] update kmeans example #12925

[SPARK-15149][EXAMPLE][DOC] update kmeans example #12925

Uh oh!

Conversation

zhengruifeng commented May 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

HyukjinKwon commented May 5, 2016

Uh oh!

zhengruifeng commented May 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MLnick commented May 5, 2016

Uh oh!

zhengruifeng commented May 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhengruifeng commented May 5, 2016

Uh oh!

zhengruifeng commented May 5, 2016

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

sethah commented May 5, 2016

Uh oh!

sethah May 5, 2016

Choose a reason for hiding this comment

Uh oh!

zhengruifeng May 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng commented May 8, 2016

Uh oh!

SparkQA commented May 8, 2016

Uh oh!

SparkQA commented May 9, 2016

Uh oh!

sethah May 9, 2016

Choose a reason for hiding this comment

Uh oh!

sethah commented May 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holdenk May 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng May 11, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 11, 2016

Uh oh!

zhengruifeng commented May 11, 2016

Uh oh!

SparkQA commented May 11, 2016

Uh oh!

MLnick commented May 11, 2016

Uh oh!

MLnick commented May 11, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zhengruifeng commented May 5, 2016 •

edited

Loading

zhengruifeng commented May 5, 2016 •

edited

Loading

zhengruifeng commented May 5, 2016 •

edited

Loading

zhengruifeng May 6, 2016 •

edited

Loading

sethah commented May 9, 2016 •

edited

Loading

holdenk May 9, 2016 •

edited

Loading