[SPARK-13070][SQL] Better error message when Parquet schema merging fails

[liancheng]

Now we also report path and schema of the file in trouble.

[rxin]

how about we just use a foreach on footers and then combine all of these in a single pass? Seems simpler.

[rxin]

i.e. something like

var mergedSchema = StructType(Nil)
footers.foreach { file =>
  val schema = ParquetRelation.readSchemaFromFooter(footer, converter)
  try {
    mergedSchema = mergedSchema.merge(schema)
  } catch {
    ...
  }
}
mergedSchema

[SparkQA]

Test build #50331 has finished for PR 10972 at commit bfe4987.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

Can we get the schema from first footer and then go through this loop for remaining footers? Because you merge the first schema with an empty schema, I think the all fields in merged schema will be optional in their metadata. So the pushing down of filters will not normally work.

[liancheng]

Yeah you'll right, filter push-down can be affected due to #9940 (which I just merged today). Thanks for pointing this out!

[SparkQA]

Test build #50327 has finished for PR 10972 at commit 1ce61a4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

can we catch something tighter here? This is too broad.

[rxin]

also i'd prefer to make this more concise, e.g.

                  throw new SparkException(
                    "Failed merging schema of file ${footer.getFile}: ${schema.treeString}", cause)

I don't know why these functions are super long.

[rxin]

@viirya do you want to submit a pull request to address your issue? @liancheng is busy with something right now.

i.e. create a pull request that contains this one and set the 1st

[SparkQA]

Test build #50354 has finished for PR 10972 at commit 2e5eddb.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

@viirya It would be great if you can help since you are pretty familiar with this part of code :)

[liancheng]

I'm closing this one.

[viirya]

@rxin @liancheng yes, I would love to do it. Thanks.