[SPARK-9340] [SQL] Fixes converting unannotated Parquet lists

[liancheng]

This PR is inspired by #8063 authored by @dguy. Especially, testing Parquet files added here are all taken from that PR.

Committer who merges this PR should attribute it to Damian Guy <damian.guy@gmail.com>.

---

SPARK-6776 and SPARK-6777 followed parquet-avro to implement backwards-compatibility rules defined in parquet-format spec. However, both Spark SQL and parquet-avro neglected the following statement in parquet-format:

This does not affect repeated fields that are not annotated: A repeated field that is neither contained by a LIST- or MAP-annotated group nor annotated by LIST or MAP should be interpreted as a required list of required elements where the element type is the type of the field.

One of the consequences is that, Parquet files generated by parquet-protobuf containing unannotated repeated fields are not correctly converted to Catalyst arrays.

This PR fixes this issue by

1. Handling unannotated repeated fields in CatalystSchemaConverter.

2. Converting this kind of special repeated fields to Catalyst arrays in CatalystRowConverter.

   Two special converters, RepeatedPrimitiveConverter and RepeatedGroupConverter, are added. They delegate actual conversion work to a child elementConverter and accumulates elements in an ArrayBuffer.

   Two extra methods, start() and end(), are added to ParentContainerUpdater. So that they can be used to initialize new ArrayBuffers for unannotated repeated fields, and propagate converted array values to upstream.

[SparkQA]

Test build #40299 has finished for PR 8070 at commit a11b7c0.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #40300 has finished for PR 8070 at commit 1c68b55.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dguy]

LGTM - working as expected.

[rdblue]

Why isn't this a CatalystPrimitiveConverter with RepeatedConverter?

[liancheng]

It's because CatalystPrimitiveConverter is defined as:

private[parquet] class CatalystPrimitiveConverter(val updater: ParentContainerUpdater)
  extends PrimitiveConverter with HasParentContainerUpdater {
  ...
}

the val updater part has two meanings:

1. updater is made a constructor argument, and
2. def updater in HasParentContainerUpdater is overriden since updater is a read-only val.

The 2nd fact prevents subclasses of CatalystPrimitiveConverter to override the updater field.

[SparkQA]

Test build #40304 has finished for PR 8070 at commit 450b606.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rdblue]

This looks good to me overall.

[liancheng]

@dguy @rdblue Thanks for the review! I just rebased this PR to resolve conflicts introduced by #8056. Will merge this pending Jenkins.

[SparkQA]

Test build #40355 has finished for PR 8070 at commit ace6df7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Merged to master and branch-1.5.