[SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for enhanced Hive partition pruning predicate pushdown

[gatorsmile]

What changes were proposed in this pull request?

This is a follow-up PR of #17633.

This PR is to add a conf spark.sql.hive.advancedPartitionPredicatePushdown.enabled, which can be used to turn the enhancement off.

How was this patch tested?

Add a test case

[SparkQA]

Test build #82944 has finished for PR 19547 at commit 4057602.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Nit, Can we remove the duplication of varcharKeys logic by moving into convertFilters like the following? table was removed from convertBasicFilters and convertCompexFilters, too.

def convertFilters(table: Table, filters: Seq[Expression]): String = {
    // hive varchar is treated as catalyst string, but hive varchar can't be pushed down.
    lazy val varcharKeys = table.getPartitionKeys.asScala
      .filter(col => col.getType.startsWith(serdeConstants.VARCHAR_TYPE_NAME) ||
        col.getType.startsWith(serdeConstants.CHAR_TYPE_NAME))
      .map(col => col.getName).toSet

    if (SQLConf.get.advancedPartitionPredicatePushdownEnabled) {
      convertComplexFilters(filters, varcharKeys)
    } else {
      convertBasicFilters(filters, varcharKeys)
    }
  }

[gatorsmile]

Thanks! We can do it when enhancing convertComplexFilters . Here, it is just to keep the original codes (from Spark 2.2) untouched.

[dongjoon-hyun]

+1, LGTM.
This is really great to have this option. Thank you so much!

[gatorsmile]

Thanks! Merged to master.