[SPARK-4488][PySpark] Add control over map-side aggregation

[uncleGen]

No description provided.

[SparkQA]

Test build #23611 has started for PR 3366 at commit 66561d4.

• This patch merges cleanly.

[SparkQA]

Test build #23611 has finished for PR 3366 at commit 66561d4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23611/
Test PASSed.

[uncleGen]

@davies Could you help reviewing this patch? Thank you!

[davies]

What's the cases that we should disable map side aggregation?

[JoshRosen]

@uncleGen Could you comment here to provide examples of when it's beneficial to disable map-side aggregation? If there is a legitimate case for disabling it, then we should add this option in Scala / Java as well. Otherwise, do you mind closing this pull request?

[davies]

@JoshRosen We already have this in Scala/Java.

[JoshRosen]

@JoshRosen We already have this in Scala/Java.

What about reduceByKey? I don't see a variant with a flag for disabling map-side combining: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala#L262. We definitely have the mapSideCombine option for combineByKey but not for reduceByKey.

I guess I kind of pattern-matched on the reduceByKey in my earlier comment; the combineByKey flag makes sense and we should definitely include that for feature-parity.

[JoshRosen]

Actually, let's re-open this one since part of it should still go in.