[SPARK-21410][CORE] Create less partitions for RangePartitioner if RDD.count() is less than partitions

[apapi]

What changes were proposed in this pull request?

Fix a bug in RangePartitioner:
In RangePartitioner(partitions: Int, rdd: RDD[]), RangePartitioner.numPartitions is wrong if the number of elements in RDD (rdd.count()) is less than number of partitions (partitions in constructor).

How was this patch tested?

test as described in SPARK-SPARK-21410

Please review http://spark.apache.org/contributing.html before opening a pull request.

[cloud-fan]

Ok to test

[jiangxb1987]

Please update the title to:

[SPARK-21410][CORE] Create less partitions for RangePartitioner if RDD.count() is less than `partitions`

[cloud-fan]

ok to test

[cloud-fan]

cc @srowen is it ok to do this? e.g. rdd.sortByKey(numPartitions = 10), do we have to return an RDD of 10 partitions even some of them are empty?

[SparkQA]

Test build #79656 has finished for PR 18631 at commit 7239734.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

That seems reasonable. An RDD can have empty partitions. The comment in the docs says that the number of partitions might be smaller than the requested number.

[jiangxb1987]

retest this please

[SparkQA]

Test build #79662 has finished for PR 18631 at commit 7239734.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

LGTM

[cloud-fan]

thanks, merging to master!