[SPARK-1412][SQL] Disable partial aggregation automatically when reduction factor is low - WIP

[rxin]

This avoids building up an expensive hash map if partial aggregation does not result in data size reduction.

Just a prototype. Kinda ugly, doesn't properly connect with the config system yet, and have no test.

[rxin]

@concretevitamin I find it hard to actually use config options in a physical operator. Any suggestions?

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[rxin]

@pwendell / @mateiz should we actually build this into Spark directly (i.e. in Aggregator)?

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15952/

[concretevitamin]

@rxin If we are simply trying to read the default values for the params, but not user-set ones (i.e. in the absence of a SQLContext in execute()), I think we could move the default param values to a companion object of SQLConf, and in the assessors of this class, either get the user-set values or else get the default values from the static object.

[mateiz]

It would be great to add this into Aggregator as well. Would that replace the implementation here? I.e. does Spark SQL go through Aggregator?

[rxin]

Spark SQL doesn't currently use the aggregator, but we would want to do that.

[aarondav]

Man, those are some high standards!

[rxin]

@mateiz I submitted a patch to core's Aggregator in #1191.

After implementing it in Aggregator, I realized it might be hard for Spark SQL to reuse Aggregator unless we change Aggregator to allocate less temporary objects (or write the aggregation code path in Spark SQL to output key value tuples).