[SPARK-9281] [SQL] use decimal or double when parsing SQL

[davies]

Right now, we use double to parse all the float number in SQL. When it's used in expression together with DecimalType, it will turn the decimal into double as well. Also it will loss some precision when using double.

This PR change to parse float number to decimal or double, based on it's using scientific notation or not, see https://msdn.microsoft.com/en-us/library/ms179899.aspx

This is a break change, should we doc it somewhere?

[SparkQA]

Test build #38337 has finished for PR 7642 at commit 1c48e34.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

typo 'e?

[SparkQA]

Test build #1197 has finished for PR 7642 at commit f4a320b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #38379 has finished for PR 7642 at commit eca99de.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class ChangeDecimalPrecision(child: Expression) extends UnaryExpression

[SparkQA]

Test build #38384 has finished for PR 7642 at commit 5e142b6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class ChangeDecimalPrecision(child: Expression) extends UnaryExpression

[SparkQA]

Test build #38679 has finished for PR 7642 at commit 1f576d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

cc @yhuai @rxin

[rxin]

We should add "releasenotes" label to the JIRA ticket.

[rxin]

cc @cloud-fan for review too

[cloud-fan]

should we apply this to CreateArray too?

[davies]

For CreateArray, it's good to make it strict. We can loose the rule later.

[cloud-fan]

LGTM

[rxin]

@davies do we need to change the hive ql part?

[davies]

@rxin HiveQl already does this, it always parse float number as decimal.

[rxin]

Great - merging this in. Thanks.

[scwf]

hi @davies seems this is not compatible with hiveql, HiveQl still parse float number as double. https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala#L1670-L1689

Or i missed sth?

[davies]

Yes, If you use HiveContext, you will be compatible with Hive, but SQLContext will not.