[SPARK-10186][SQL][WIP] Array types using JDBCRDD and postgres

[mariusvniekerk]

This change allows reading from jdbc array column types for the postgresql dialect.

This also opens up some implementation for array types using other jdbc backends.

@JoshRosen @marmbrus

[AmplabJenkins]

Can one of the admins verify this patch?

[mariusvniekerk]

Still need to add some additional types from

https://github.com/pgjdbc/pgjdbc/blob/master/org/postgresql/jdbc2/TypeInfoCache.java#L70

[lepfhty]

missed some underscores here?

[lepfhty]

very cool! nice refactoring too! thanks for picking this up, i haven't had time to revisit.

would you mind adding 'uuid' as Some(StringType) to the PostgresDialect#getCatalystType method? it's probably a one-liner.

[mariusvniekerk]

Sure. Had to refactor a little to work around type erasure warnings

[mariusvniekerk]

I'll add tests once [SPARK-9818] #8101 is merged in

[maropu]

ISTM we need to check if input types are valid for target databases in advance, e.g., in JavaUtils#saveTable.
JavaUtils#savePartition should simply put input data as given typed-data.

[mariusvniekerk]

If the particular dialect does not support these types saveTable should toss an exception when building the nullTypes array

[maropu]

Great work! I left some review comments.
Since ARRAY is one of the SQL99 standard, it'd be better to put this type mapping in JavaUtils#schemaString.
Yes, I know that some databases, e.g, mysql, have no support for ARRAY, so these databases need to throw an exception for unsupported types in JdbcDialect#getJDBCType.

[mariusvniekerk]

I also need to rebase this thing against master again it seems

[mariusvniekerk]

Is the best approach to rebase or just merge master into this and resolve conflicts?

[mariusvniekerk]

@JoshRosen Guess its refactor time due to SPARK-11541. Makes it rather hard if we ever want to add support in 1.5X.

[JoshRosen]

We just merged #9503, so it should now be possible to add integration tests which run against a real Postgres database. Take a look at my PR for examples of where to add these tests.