[SPARK-12340][SQL] Fix overstep the bounds of Int in SparkPlan.executeTake

[QiangCai]

If partsScanned = 1, numPartsToTry = Int.MaxValue and totalParts = 200, the result of math.min(partsScanned + numPartsToTry, totalParts) will be Int.MinValue (-2147483648).

Modifies partsScanned to partsScanned.toLong and chang result of math.min to Int.

https://issues.apache.org/jira/browse/SPARK-12340

[srowen]

Sounds good to me.

[SparkQA]

Test build #2217 has finished for PR 10310 at commit 37acd54.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class WrapOption(child: Expression) extends UnaryExpression\n

[QiangCai]

The number of partitions which has scanned is the size of seq p, not numPartsToTry.

So I use p.size to instead of numPartsToTry in line partsScanned += numPartsToTry

[srowen]

Yes I think that also makes sense, in the case where we hit the limit of totalParts.

[SparkQA]

Test build #2220 has finished for PR 10310 at commit 56f151a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Can you also change take in RDD?

[QiangCai]

I think it (take in RDD) is ok.

[srowen]

@QiangCai I think that technically var partsScanned = 0 in take is a problem since it's incremented by numPartsToTry and could overflow causing partsScanned < totalParts to always be true. It can be set to OL to make it a long. This is a very rare case where numPartsToTry is large and totalParts is close to the max integer, and n is very large.

[QiangCai]

@srowen I have modified all the code, and try to keep them to be the same.

At first, the var numPartsToTry and partsScanned have been set to Long.
Secondly, because var p should be Seq[Int], so i have added toInt for the until statement

[srowen]

@QiangCai thanks that looks good, but this needs a rebase now.

[QiangCai]

@srowen I maybe have made a mistake in git bash. Can I new another pull request to resolve this problem?

[srowen]

I think you just need to rebase from master and force-push the result, but do what you need to.

[QiangCai]

@srowen I have new another pull request. This pull request will be closed.
another pull request #10487