[SPARK-6888][SQL] Export driver quirks

[rtreffer]

Make it possible to (temporary) overwrite the driver quirks. This
can be used to overcome problems with specific schemas or to
add new jdbc driver support on the fly.

A very simple implementation to dump the loading can be done like this (spark-shell)

class DumpQuirk extends org.apache.spark.sql.jdbc.DriverQuirks {
  def canHandle(url : String): Boolean = true
  def getCatalystType(sqlType: Int, typeName: String, size: Int, md: org.apache.spark.sql.types.MetadataBuilder): org.apache.spark.sql.types.DataType = {
    println("" + (sqlType, typeName, size, md))
    null
  }
  def getJDBCType(dt: org.apache.spark.sql.types.DataType): (String, Option[Int]) = (null, None)
}
org.apache.spark.sql.jdbc.DriverQuirks.registerQuirks(new DumpQuirk())

Not that this pull request is against 1.3 - I could not create a distribution with the current master.

[srowen]

This needs a JIRA -- see https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark

[rtreffer]

Added a ticket: https://issues.apache.org/jira/browse/SPARK-6888
Will add that to the commit after some sleep .zZzZzZ

[liancheng]

ok to test.

[liancheng]

Please add return type explicitly for all public methods.

[rtreffer]

@liancheng thank you, will updaste the patch.
Just one question: Should I sqash/amend the fixes or should I add a second commit?

[liancheng]

How about this:

quirks.map(_.getCatalystType(sqlType, typeName, size, md)).collectFirst {
  case dataType if dataType != null => dataType
}.orNull

[liancheng]

You may just add new commits to this PR. Also, would you please add tests for this feature?

[SparkQA]

Test build #30332 has finished for PR 5498 at commit 9ca66d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rtreffer]

I still have to write tests for AggregatedQuirks.

[marmbrus]

If we are going to make this a public API we should consider a clearer name. Perhaps. JDBCTypeMapping? You will also need to reopen the PR against master as we don't want to add new APIs in a maintenance branch.

[marmbrus]

Add scala doc to describe the contract for each of these methods.

[SparkQA]

Test build #30375 has finished for PR 5498 at commit 7f23484.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rtreffer]

@marmbrus thank you, I'll fix those issues and open a new one when done.

Regarding naming/api:

It is quite common that there is one class per sql/jdbc dialect. Often called that way (e.g. MySQLDialect on hibernate). I've found quite some projects that use the same naming (via github search).
Anyway, in this case I'd like to match those namings and add a default implementation per method (returning the neutral element).

On the other hand it's currently just doing type mapping. So JDBCTypeMapping would be a very valid name, too. It would restrict the use case more (can be good or bad).

I guess you know better what would suite spark :-)

[marmbrus]

Dialect seems reasonable to me.

[marmbrus]

We will also want to mark all of these @DeveloperApi

[SparkQA]

Test build #30463 has finished for PR 5498 at commit 22d65ca.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rtreffer]

Replaced by #5555