[SPARK-17521]Error when I use sparkContext.makeRDD(Seq())

[codlife]

What changes were proposed in this pull request?

when i use sc.makeRDD below

val data3 = sc.makeRDD(Seq())
println(data3.partitions.length)

I got an error:
Exception in thread "main" java.lang.IllegalArgumentException: Positive number of slices required

We can fix this bug just modify the last line ,do a check of seq.size

  def makeRDD[T: ClassTag](seq: Seq[(T, Seq[String])]): RDD[T] = withScope {
    assertNotStopped()
    val indexToPrefs = seq.zipWithIndex.map(t => (t._2, t._1._2)).toMap
    new ParallelCollectionRDD[T](this, seq.map(_._1), math.max(seq.size, defaultParallelism), indexToPrefs)
  }

How was this patch tested?

manual tests

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

[srowen]

I would say math.max(seq.size, 1). Really this method would normally just use the provided partition count (called "numSlices" in this old API) but this one doesn't have that parameter, which is more reason it's an odd man out. Still I think the most reasonable behavior is to use at least 1 partition.

[codlife]

To keep the same with sc.parallelize, I think the defalutParallelism is reasonable,
to let the code below be same, I think we should use defaultParallelism,

val rdd = sc.makeRDD(Seq())
val rdd = sc.parallelize(Seq())

but which one to use, you can make a decision.

[srowen]

The problem is that the default is OK because it's changeable, but here someone has no way to change it. I think it might be better to stay conservative.

Really this is such a corner case that it doesn't matter much. It only showed up for you because you specified no type on your Seq. If you had, it would have chosen the other overload which works fine.

[codlife]

ok ,thanks for your explain.

[srowen]

Jenkins test this please

[SparkQA]

Test build #65362 has finished for PR 15077 at commit f454668.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master/2.0