Update to Parquet 1.6.0 Final

[oliviertoupin]

Tested with our workload with success.

Bring last version of parquet-mr to spark.

Might fix => SPARK-4258

[AmplabJenkins]

Can one of the admins verify this patch?

[JoshRosen]

We just bumped to 1.7 in master (#6597); can you check whether that PR resolved SPARK-4258?

[kostya-sh]

Parquet 1.6.0 certainly fixes SPARK-4258, we've been using a custom build of Spark with parquet 1.6.0rc7 for a while now. We had to apply similar fix to the one from this pull request. See https://issues.apache.org/jira/browse/SPARK-6566

If parquet 1.7 is not coming to Spark 1.4.x it would be really good to at least update it to parquet 1.6.0

[rxin]

hey @kostya-sh - in the past parquet minor bumps have introduced bugs, and as a result we are very conservative w.r.t. upgrading Parquet. However, I think you can always specify a higher version of the Parquet jar in your user program, and then set the classloader to prefer user jars first. Alternatively, you can also just bump the Parquet version in your custom build of Spark. Does that make sense?

[kostya-sh]

Unfortunately a small change to Spark source code is required when updating from 1.6.0rc3 to 1.6.0, so it is not possible to just update parquet version in the user program. We bumped Parquet version in our custom build of Spark (and fixed the compilation issue I mentioned) instead.

[oliviertoupin]

Closing because fixed by #6597