[SPARK-14387][SPARK-19459][SQL] Enable Hive-1.x ORC compatibility with spark.sql.hive.convertMetastoreOrc

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR includes #14471 to enable spark.sql.hive.convertMetastoreOrc for SPARK-19459 (ORC tables cannot be read when they contain char/varchar columns). All credits should go to @rajeshbalamohan .

For SPARK-19459, the padding is handled by Hive-side HiveCharWritable via HiveBaseChar.java on read. So, ORCFileFormat can handle this unlikely from ParquetFileFormat. For Parquet, please see SPARK-21997.

How was this patch tested?

Pass the newly added test cases.

[gatorsmile]

I think this is not a fix. What is the root cause?

[SparkQA]

Test build #81789 has finished for PR 19235 at commit 57332c3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Won't this cause Hive's parquet code to be used by default instead of Spark's? I'm not sure that's what we want.

[dongjoon-hyun]

Thank you for review, @gatorsmile and @vanzin . Yes, this is not a way where Apache Spark is heading. It's slow.

Here, I wanted to raise this issue of convertMetastoreXXX things since convertMetastoreXXX seems not to be tested completly for both ORC/Parquet. Especially, as you see here, 14 failures means the test cases depend implicitly on that option. I think we need to test for both true and false for all those cases. Otherwise, we need to add withSQLConf(true) explicitly.

[dongjoon-hyun]

I'm not sure where I hit this before from some another JIRA issue. But, I updated 'WHERE' clause examples. Maybe, there was discussion on this difference.

[vanzin]

I wanted to raise this issue of convertMetastoreXXX things since convertMetastoreXXX seems not to be tested completly

That's fine, but that's not what your change is proposing. If you look at the test failures, at least the one I looked at is because the wrong exception is being thrown, not because the feature is broken. That doesn't excuse the fact that it's not tested, but maybe the situation is not that dire.

But as Xiao has mentioned, you're not fixing the problem you describe in the change summary; you haven't even root caused the problem. You just found out that using Hive classes to read the parquet data works around the problem, but that's not an acceptable fix for the problem - it's just a workaround that those affected by the issue can use.

[dongjoon-hyun]

I agree with you. Okay. I'll refocus this.

[dongjoon-hyun]

@gatorsmile and @vanzin .

I'm comparing with ORC now. Previously, ORC fails with another reason. I'll make another PR for that. I found that #14471 is enough for ORC.

In case of ORC, ORC itself handles truncations on write. The padding is handled by Hive side HiveCharWritable via HiveBaseChar.java on read. In case of Parquet, I guess Parquet is the same, but there is no such a padding logic like HiveCharWritable in Spark.

[dongjoon-hyun]

Since this PR is invalid, I'll reuse this PR instead of creating new one.

[dongjoon-hyun]

This is a test case for ORC file format. Based on #14471, I'm enabling this.
For Parquet, I think we can proceed separately if ORC is finished.

[SparkQA]

Test build #81806 has finished for PR 19235 at commit c6d2c35.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.