Skip to content

[SPARK-6177][MLlib]Add note in LDA example to remind possible coalesce #4899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hhbyyh wants to merge 4 commits into from
Closed

[SPARK-6177][MLlib]Add note in LDA example to remind possible coalesce #4899

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
26a564a
add coalesce to LDAExample
hhbyyh Mar 5, 2015
f7fd5d4
Merge remote-tracking branch 'upstream/master' into adjustPartition
hhbyyh Mar 9, 2015
9a2d7b6
move to comment
hhbyyh Mar 9, 2015
a499630
update comment
hhbyyh Mar 10, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,9 @@ object LDAExample {
stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {

// Get dataset of document texts
// One document per line in each text file.
// One document per line in each text file. If the input consists of many small files,
// this can result in a large number of small partitions, which can degrade performance.
// In this case, consider using coalesce() to create fewer, larger partitions."
val textRDD: RDD[String] = sc.textFile(paths.mkString(","))

// Split text into words
Expand Down