[SPARK-13667][SQL] Support for specifying custom date format for date and timestamp types at CSV datasource. #11550

HyukjinKwon · 2016-03-07T01:19:45Z

What changes were proposed in this pull request?

This PR adds the support to specify custom date format for DateType and TimestampType.

For TimestampType, this uses the given format to infer schema and also to convert the values
For DateType, this uses the given format to convert the values.
If the dateFormat is not given, then it works with DateTimeUtils.stringToTime() for backwords compatibility.
When it's given, then it uses SimpleDateFormat for parsing data.

In addition, IntegerType, DoubleType and LongType have a higher priority than TimestampType in type inference. This means even if the given format is yyyy or yyyy.MM, it will be inferred as IntegerType or DoubleType. Since it is type inference, I think it is okay to give such precedences.

In addition, I renamed csv.CSVInferSchema to csv.InferSchema as JSON datasource has json.InferSchema. Although they have the same names, I did this because I thought the parent package name can still differentiate each. Accordingly, the suite name was also changed from CSVInferSchemaSuite to InferSchemaSuite.

How was this patch tested?

unit tests are used and ./dev/run_tests for coding style tests.

HyukjinKwon · 2016-03-07T01:22:01Z

@rxin There should be a conflict with #11315 which I think it's supposed to be merged (assuming from your comment).

I will resolve the conflict as soon as either this one or that one is merged.

HyukjinKwon · 2016-03-07T01:22:15Z

@falaki Would you maybe review this please..?

SparkQA · 2016-03-07T03:04:39Z

Test build #52535 has finished for PR 11550 at commit 5c990cd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-03-08T02:55:26Z

@falaki Just let to know, I changed the name CSVInferSchema to InferSchema mainly for consistent names for CSV and JSON data source but maybe they might have to be CSVInferSchema and JSONInferSchema.

SparkQA · 2016-03-08T03:18:24Z

Test build #52621 has finished for PR 11550 at commit c51d4ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-08T04:00:12Z

Test build #52619 has finished for PR 11550 at commit db27259.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DefaultSource extends FileFormat with DataSourceRegister
- class InMemoryCatalog extends ExternalCatalog
- abstract class ExternalCatalog
- case class CatalogTablePartition(
- case class WriteRelation(
- class DefaultSource extends FileFormat with DataSourceRegister
- class DefaultSource extends FileFormat with DataSourceRegister
- case class FileTypes(
- class DefaultSource extends FileFormat with DataSourceRegister
- case class HadoopFsRelation(
- trait FileFormat
- trait FileCatalog
- class HDFSFileCatalog(
- class HiveFileCatalog(
- .doc(\"A comma-separated list of class names of services to add to the scheduler.\")

falaki · 2016-03-08T20:59:48Z

@HyukjinKwon the changes look good to me.

HyukjinKwon · 2016-03-09T03:23:00Z

cc @rxin

SparkQA · 2016-03-14T06:26:34Z

Test build #53049 has finished for PR 11550 at commit e0612f1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-14T07:26:31Z

Test build #53054 has finished for PR 11550 at commit 9d79996.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-14T07:50:14Z

Test build #53055 has finished for PR 11550 at commit 49a8210.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-03-14T08:54:45Z

retest this please

HyukjinKwon · 2016-03-14T09:15:42Z

cc @rxin

SparkQA · 2016-03-14T09:35:43Z

Test build #53065 has finished for PR 11550 at commit 7f701a9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-14T10:00:38Z

Test build #53068 has finished for PR 11550 at commit fd03024.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-14T10:47:00Z

Test build #53063 has finished for PR 11550 at commit 49a8210.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-16T07:38:27Z

Test build #53298 has finished for PR 11550 at commit 6bc0ffb.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-03-16T07:44:33Z

retest this please

SparkQA · 2016-03-16T07:53:17Z

Test build #53301 has finished for PR 11550 at commit 6bc0ffb.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-16T08:57:25Z

Test build #53296 has finished for PR 11550 at commit be4996b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-07T03:11:12Z

Test build #55174 has finished for PR 11550 at commit 3f3ef6f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-11T03:22:32Z

Test build #55496 has finished for PR 11550 at commit c197d52.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-30T02:04:10Z

@HyukjinKwon I want to get this in for 2.0. Can you not do the renaming here? It's always super confusing when classes have the same name, even when they are in different packages. We are trying to move away from that. It will also make the patch much easier to review when the patch is more "precise".

HyukjinKwon · 2016-04-30T02:09:38Z

@rxin Thank you. I will change the name to the original.

HyukjinKwon · 2016-04-30T02:59:40Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala

      nullable: Boolean = true,
-      nullValue: String = ""): Any = {
+      nullValue: String = "",
+      dateFormat: SimpleDateFormat = null): Any = {


This will conflict with #11947 (this is making the last argument as a single option not for the multiple individual parameters like I did for infer() above).

rxin · 2016-04-30T03:08:36Z

Left some minor comments. LGTM otherwise.

SparkQA · 2016-04-30T04:05:54Z

Test build #57397 has finished for PR 11550 at commit c599d92.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-30T04:13:45Z

Test build #57398 has finished for PR 11550 at commit 58159f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-30T04:58:13Z

Test build #57401 has finished for PR 11550 at commit 49732b6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-30T05:11:09Z

Test build #57402 has finished for PR 11550 at commit 3572199.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-30T05:17:14Z

Test build #57404 has finished for PR 11550 at commit c1710d5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-30T05:52:00Z

Thanks - merging in master!

Support for specifying custom date format for date and timestamp types.

5c990cd

HyukjinKwon added 2 commits March 8, 2016 10:40

Resolve conflicts

db27259

Remove unneeded files

c51d4ef

Resolve conflicts

e0612f1

Coding style nit

9d79996

Fix style

49a8210

Resolve conflicts

7f701a9

Remove duplicated test file

fd03024

This was referenced Mar 15, 2016

[SPARK-13866][SQL] Handle decimal type in CSV inference at CSV data source. #11724

Closed

[SPARK-13899][SQL] Produce InternalRow instead of external Row at CSV data source #11717

Closed

HyukjinKwon added 2 commits March 16, 2016 16:22

Resolve conflicts

be4996b

Remove unused imports

6bc0ffb

Resolve conflicts

3f3ef6f

Resolve conflicts

c197d52

HyukjinKwon added 2 commits April 30, 2016 11:31

Do not rename classes to be the same

c599d92

Infer takes single csvOptions instead of taking each option

58159f7

HyukjinKwon reviewed Apr 30, 2016
View reviewed changes

HyukjinKwon added 6 commits April 30, 2016 12:42

Make the structure like the original and add an option argument

d373322

Remove unused variables

48f9fec

Remove unused variables

3d7318e

Fix typo

49732b6

Make some functions private

3572199

Reindent

c1710d5

asfgit closed this in 4bac703 Apr 30, 2016

HyukjinKwon mentioned this pull request Jul 18, 2016

[SPARK-16216][SQL] CSV data source supports custom date format for writing and timezone option for both reading and writing #13912

Closed

HyukjinKwon deleted the SPARK-13667 branch January 2, 2018 03:40

[SPARK-13667][SQL] Support for specifying custom date format for date and timestamp types at CSV datasource. #11550

[SPARK-13667][SQL] Support for specifying custom date format for date and timestamp types at CSV datasource. #11550

Uh oh!

Conversation

HyukjinKwon commented Mar 7, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon commented Mar 7, 2016

Uh oh!

HyukjinKwon commented Mar 7, 2016

Uh oh!

SparkQA commented Mar 7, 2016

Uh oh!

HyukjinKwon commented Mar 8, 2016

Uh oh!

SparkQA commented Mar 8, 2016

Uh oh!

SparkQA commented Mar 8, 2016

Uh oh!

falaki commented Mar 8, 2016

Uh oh!

HyukjinKwon commented Mar 9, 2016

Uh oh!

SparkQA commented Mar 14, 2016

Uh oh!

SparkQA commented Mar 14, 2016

Uh oh!

SparkQA commented Mar 14, 2016

Uh oh!

HyukjinKwon commented Mar 14, 2016

Uh oh!

HyukjinKwon commented Mar 14, 2016

Uh oh!

SparkQA commented Mar 14, 2016

Uh oh!

SparkQA commented Mar 14, 2016

Uh oh!

SparkQA commented Mar 14, 2016

Uh oh!

SparkQA commented Mar 16, 2016

Uh oh!

HyukjinKwon commented Mar 16, 2016

Uh oh!

SparkQA commented Mar 16, 2016

Uh oh!

SparkQA commented Mar 16, 2016

Uh oh!

SparkQA commented Apr 7, 2016

Uh oh!

SparkQA commented Apr 11, 2016

Uh oh!

rxin commented Apr 30, 2016

Uh oh!

HyukjinKwon commented Apr 30, 2016

Uh oh!

HyukjinKwon Apr 30, 2016

Choose a reason for hiding this comment

Uh oh!

rxin commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

rxin commented Apr 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!