[SPARK-3407][SQL]Add Date type support #2344

adrian-wang · 2014-09-10T12:02:15Z

No description provided.

SparkQA · 2014-09-10T12:51:54Z

QA tests have started for PR 2344 at commit ce576b3.

This patch merges cleanly.

SparkQA · 2014-09-10T14:29:13Z

QA tests have finished for PR 2344 at commit ce576b3.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging

SparkQA · 2014-09-11T03:53:10Z

QA tests have started for PR 2344 at commit 7269bba.

This patch merges cleanly.

SparkQA · 2014-09-11T05:37:50Z

QA tests have finished for PR 2344 at commit 7269bba.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging

adrian-wang · 2014-09-11T05:44:46Z

Ha, first pass in recent days!

marmbrus · 2014-09-11T22:20:30Z

.gitignore

We are considering removing the need for this special config file in #2263 so maybe best to leave this out.

marmbrus · 2014-09-11T22:22:40Z

This looks pretty awesome! I'd like to do a quick pass and make sure there aren't any missing places, but overall this looks complete to me. Is there a reason the PR title still says WIP?

adrian-wang · 2014-09-12T01:00:28Z

The last thing here is to enable date_udf.q. Currently there are two issues:

PR [SPARK-3485][SQL] should check parameter type when look for constructors #2355 org.apache.hadoop.hive.serde2.io.DateWritable got two constructors take one parameter, one is int and the other java.sql.Date, would both be int primitiveTypes and cause random failure when call this in a function.
Have to figure out how to enable datediff(TimestampType, ...), currently this would lead to a scala.Matcherror because it seems try to cast TimestampType to TimestampType in ConstantFolding, still looking why this is not solved in SimplifyCasts.

SparkQA · 2014-09-12T04:39:28Z

QA tests have started for PR 2344 at commit 7ed5450.

This patch merges cleanly.

adrian-wang · 2014-09-12T05:32:39Z

@marmbrus Cheng Hao's PR #2368 will resolve the second issue :).
After #2368 and #2355 are merged, I'll rebase my code and add date_udf to white list.

SparkQA · 2014-09-12T06:24:10Z

QA tests have finished for PR 2344 at commit 7ed5450.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
- public class DateType extends DataType

SparkQA · 2014-09-17T09:34:27Z

QA tests have started for PR 2344 at commit 413f946.

This patch merges cleanly.

SparkQA · 2014-09-17T10:47:08Z

QA tests have finished for PR 2344 at commit 413f946.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
- public class DateType extends DataType

adrian-wang · 2014-09-17T13:01:01Z

Just rebase code, the failure is in spark-core and not in this patch.

liancheng · 2014-09-17T22:08:02Z

test this please

SparkQA · 2014-09-17T22:14:50Z

QA tests have started for PR 2344 at commit 413f946.

This patch merges cleanly.

SparkQA · 2014-09-17T22:20:33Z

QA tests have finished for PR 2344 at commit 413f946.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
- public class DateType extends DataType

chenghao-intel · 2014-09-18T00:55:40Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala

I didn't check this in Hive, but probably it's better to convert the Date => Timestamp, since Timestamp is more precise. The same for the rule In .

To get the same result from hive, seems we have to change everything to string here.

chenghao-intel · 2014-09-18T01:18:46Z

A few comments on returns null V.S. raise exception in Cast, and which is the more wider type for Date and Timestamp, the other LGTM.

liancheng · 2014-10-10T10:21:00Z

sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnStats.scala

Using .getTime to do the comparison would be much more efficient. When caching a DateType column, this function is really a critical path.

liancheng · 2014-10-10T10:37:02Z

Left some minor comments, otherwise LGTM. Also, do we want to add Python API in this PR or a separate one?

adrian-wang · 2014-10-10T12:07:31Z

I can see most of the reviews are focused on comparing and ordering. I'd like to fix those comparing rules in a separated PR. I tested what you declared here the other day, that was also different from what Hive did. Also Python API is can be a separated PR.

SparkQA · 2014-10-10T12:59:43Z

QA tests have started for PR 2344 at commit 00fe81f.

This patch merges cleanly.

SparkQA · 2014-10-10T13:04:39Z

QA tests have started for PR 2344 at commit 2038085.

This patch merges cleanly.

SparkQA · 2014-10-10T13:24:39Z

QA tests have started for PR 2344 at commit f15074a.

This patch merges cleanly.

SparkQA · 2014-10-10T13:31:12Z

QA tests have finished for PR 2344 at commit 00fe81f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
- public class DateType extends DataType

AmplabJenkins · 2014-10-10T13:31:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21591/Test FAILed.

SparkQA · 2014-10-10T13:36:22Z

QA tests have finished for PR 2344 at commit 2038085.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
- public class DateType extends DataType

AmplabJenkins · 2014-10-10T13:36:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21592/Test FAILed.

adrian-wang · 2014-10-10T13:47:01Z

Seems we are getting something wrong with orc in jenkins... My local test is fine.

liancheng · 2014-10-10T13:48:59Z

I guess this one is related: 411cf29

SparkQA · 2014-10-10T13:56:16Z

QA tests have finished for PR 2344 at commit f15074a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
- public class DateType extends DataType

AmplabJenkins · 2014-10-10T13:56:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21593/Test FAILed.

adrian-wang · 2014-10-10T14:03:35Z

retest this please.

SparkQA · 2014-10-10T14:09:40Z

QA tests have started for PR 2344 at commit f15074a.

This patch merges cleanly.

SparkQA · 2014-10-10T14:42:09Z

QA tests have finished for PR 2344 at commit f15074a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
- public class DateType extends DataType

AmplabJenkins · 2014-10-10T14:42:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21594/Test FAILed.

adrian-wang · 2014-10-11T02:55:40Z

retest this please.

SparkQA · 2014-10-11T02:59:48Z

QA tests have started for PR 2344 at commit f15074a.

This patch merges cleanly.

SparkQA · 2014-10-11T03:45:13Z

QA tests have finished for PR 2344 at commit f15074a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-11T03:45:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21615/Test PASSed.

marmbrus · 2014-10-13T20:33:45Z

Thanks for working on this huge feature! Merged to master.

adrian-wang changed the title ~~[SPARK-3407][SQL]Add Date type support~~ [SPARK-3407][SQL][wip]Add Date type support Sep 10, 2014

adrian-wang changed the title ~~[SPARK-3407][SQL][wip]Add Date type support~~ [SPARK-3407][SQL]Add Date type support Sep 11, 2014

adrian-wang changed the title ~~[SPARK-3407][SQL]Add Date type support~~ [SPARK-3407][SQL][WIP]Add Date type support Sep 11, 2014

marmbrus reviewed Sep 11, 2014
View reviewed changes

.gitignore Outdated

Copy link

Contributor

marmbrus Sep 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are considering removing the need for this special config file in #2263 so maybe best to leave this out.

adrian-wang changed the title ~~[SPARK-3407][SQL][WIP]Add Date type support~~ [SPARK-3407][SQL]Add Date type support Sep 12, 2014

adrian-wang force-pushed the date branch from 7ed5450 to 413f946 Compare September 17, 2014 09:30

chenghao-intel reviewed Sep 18, 2014
View reviewed changes

liancheng reviewed Oct 10, 2014
View reviewed changes

adrian-wang added 2 commits October 10, 2014 05:54

address lian cheng's comments

00fe81f

update return type

2038085

remove outdated lines

f15074a

liancheng mentioned this pull request Oct 13, 2014

[WIP][SPARK-2883][SQL]initial support ORC in spark sql #2576

Closed

asfgit closed this in 2ac40da Oct 13, 2014

[SPARK-3407][SQL]Add Date type support #2344

[SPARK-3407][SQL]Add Date type support #2344

Uh oh!

Conversation

adrian-wang commented Sep 10, 2014

Uh oh!

SparkQA commented Sep 10, 2014

Uh oh!

SparkQA commented Sep 10, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

adrian-wang commented Sep 11, 2014

Uh oh!

marmbrus Sep 11, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Sep 11, 2014

Uh oh!

adrian-wang commented Sep 12, 2014

Uh oh!

SparkQA commented Sep 12, 2014

Uh oh!

adrian-wang commented Sep 12, 2014

Uh oh!

SparkQA commented Sep 12, 2014

Uh oh!

SparkQA commented Sep 17, 2014

Uh oh!

SparkQA commented Sep 17, 2014

Uh oh!

adrian-wang commented Sep 17, 2014

Uh oh!

liancheng commented Sep 17, 2014

Uh oh!

SparkQA commented Sep 17, 2014

Uh oh!

SparkQA commented Sep 17, 2014

Uh oh!

chenghao-intel Sep 18, 2014

Choose a reason for hiding this comment

Uh oh!

adrian-wang Sep 18, 2014

Choose a reason for hiding this comment

Uh oh!

chenghao-intel commented Sep 18, 2014

Uh oh!

liancheng Oct 10, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng commented Oct 10, 2014

Uh oh!

adrian-wang commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

AmplabJenkins commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

AmplabJenkins commented Oct 10, 2014

Uh oh!

adrian-wang commented Oct 10, 2014

Uh oh!

liancheng commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

AmplabJenkins commented Oct 10, 2014

Uh oh!

adrian-wang commented Oct 10, 2014

Uh oh!