[SPARK-11767] [SQL] limit the size of caced batch

[davies]

Currently the size of cached batch in only controlled by batchSize (default value is 10000), which does not work well with the size of serialized columns (for example, complex types). The memory used to build the batch is not accounted, it's easy to OOM (especially after unified memory management).

This PR introduce a hard limit as 4M for total columns (up to 50 columns of uncompressed primitive columns).

This also change the way to grow buffer, double it each time, then trim it once finished.

cc @liancheng

[SparkQA]

Test build #46062 has finished for PR 9760 at commit d57c180.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2069 has finished for PR 9760 at commit d57c180.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2070 has finished for PR 9760 at commit 55a905b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class BitSet(numBits: Int) extends Serializable\n * class StreamingListener(object):\n * case class JSONOptions(\n * abstract class Aggregator[-A, B, C] extends Serializable\n

[liancheng]

This changes LGTM, but I don't quite understand this line in the PR description:

... This PR introduce a hard limit as 4M for total columns (up to 50 columns of uncompressed primitive columns).

Where is the "50" from? Is it a hard limit defined somewhere else or an estimation of average use case?

[davies]

It's an estimation based on LongType/DoubleType, it means most of the tables with primitive types will not be affected.