[SPARK-16729][SQL] Throw analysis exception for invalid date casts #14358
Conversation
…casts to date type
|
Test build #62857 has finished for PR 14358 at commit
|
|
@cloud-fan can you take a look? |
| // It is never possible to compare result when hive return with exception, | ||
| // so we can return null | ||
| // NULL is more reasonable here, since the query itself obeys the grammar. | ||
| case _ => _ => null |
There was a problem hiding this comment.
cc @yhuai @liancheng , do you remember why we have this behaviour at the beginning?
There was a problem hiding this comment.
Seems like a design decision made in the original PR. See here: https://github.com/apache/spark/pull/2344/files#diff-258b71121d8d168e4d53cb5b6dc53ffeR166
I don't think we've ever discussed this case explicitly. This change seems reasonable to me.
|
Is this good to merge? |
| // to ensure we test every possible cast situation here | ||
| atomicTypes.zip(atomicTypes).foreach { case (from, to) => | ||
| checkNullCast(from, to) | ||
| if (Cast.canCast(from, to)) { |
There was a problem hiding this comment.
why this check? doesn;t from always equal to to here?
There was a problem hiding this comment.
Not all atomicTypes can cast from each other? E.g. date.
There was a problem hiding this comment.
def canCast(from: DataType, to: DataType): Boolean = (from, to) match {
case (fromType, toType) if fromType == toType => true
......
There was a problem hiding this comment.
ah this is doing self casting - i read it wrong. let me remove it.
|
LGTM, pending jenkins. |
|
Test build #62910 has finished for PR 14358 at commit
|
|
thanks, merging to master! |
|
I would consider this a bug and put it in branch-2.0. I'm going to cherry pick. |
Spark currently throws exceptions for invalid casts for all other data types except date type. Somehow date type returns null. It should be consistent and throws analysis exception as well. Added a unit test case in CastSuite. Author: petermaxlee <petermaxlee@gmail.com> Closes #14358 from petermaxlee/SPARK-16729. (cherry picked from commit ef0ccbc) Signed-off-by: Reynold Xin <rxin@databricks.com>
5 participants
What changes were proposed in this pull request?
Spark currently throws exceptions for invalid casts for all other data types except date type. Somehow date type returns null. It should be consistent and throws analysis exception as well.
How was this patch tested?
Added a unit test case in CastSuite.