Skip to content

SPARK-3580: New public method for RDD's to have consistent way of obtain... #2447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
patmcdonough wants to merge 2 commits into from
Closed

SPARK-3580: New public method for RDD's to have consistent way of obtain... #2447

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
de966fd
SPARK-3389: New public method for RDD's to have consistent way of obt…
patmcdonough Sep 18, 2014
afc4e09
Fixing code style per PR comments
patmcdonough Sep 19, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions core/src/main/scala/org/apache/spark/rdd/RDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,21 @@ abstract class RDD[T: ClassTag](
}
}

/**
* Get the number of partitions in this RDD
*
* {{{
* scala> val rdd = sc.parallelize(1 to 4, 2)
* rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at <console>:12
*
* scala> rdd.getNumPartitions
* res1: Int = 2
* }}}
Copy link
Contributor

@markhamstra markhamstra Sep 18, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Including a (pretty obvious) spark-shell example in the scaladoc of a simple RDD method isn't really consistent with the rest of the RDD API documentation.

Copy link
Contributor

@pwendell pwendell Sep 18, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I can clean this up along with the other style issue on merge.

Copy link
Author

@patmcdonough patmcdonough Sep 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, although it's worth noting this was essentially ported directly from the python API (including the doc). Any doc changes should be consistent across both versions if possible.

*
* @return The number of partitions in this RDD
*/
def getNumPartitions: Int = partitions.size

/**
* Get the preferred locations of a partition (as hostnames), taking into account whether the
* RDD is checkpointed.
Expand Down
1 change: 1 addition & 0 deletions core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ class RDDSuite extends FunSuite with SharedSparkContext {

test("basic operations") {
val nums = sc.makeRDD(Array(1, 2, 3, 4), 2)
assert(nums.getNumPartitions === 2)
assert(nums.collect().toList === List(1, 2, 3, 4))
assert(nums.toLocalIterator.toList === List(1, 2, 3, 4))
val dups = sc.makeRDD(Array(1, 1, 2, 2, 3, 3, 4, 4), 2)
Expand Down