Skip to content

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Jul 21, 2016

What changes were proposed in this pull request?

This PR makes LEAST and GREATEST accept other numeric types (Decimal). This worked fine in HiveContext in 1.6 because , for example, 1.5 was DoubleType in there.

So, this finds a tightest common type from both IntegerType and DoubleType as DoubleType, then it works okay.

But currently it seems 1.5 is being treated as DecimalType for now. So, It casts 1.5 as decimal(2, 1). So, it fails to find a tightest common type from both IntegerType and DecimalType(2, 1).

This PR introduces new function findTightestCommonTypeToDecimal dealing with decimals interacting with each other or with primitive types. This logic is borrowed from JSON schema inference.

How was this patch tested?

Unit tests in TypeCoercionSuite and DataFrameFunctionsSuite.

@HyukjinKwon
Copy link
Member Author

@liancheng and @davies Will this change be appropriate? Could you please take a look?

@SparkQA
Copy link

SparkQA commented Jul 21, 2016

Test build #62652 has finished for PR 14294 at commit 9730afb.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 21, 2016

Test build #62653 has finished for PR 14294 at commit abbcb4e.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

retest this please

@SparkQA
Copy link

SparkQA commented Jul 21, 2016

Test build #62657 has finished for PR 14294 at commit abbcb4e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

Closing this. Please refer https://issues.apache.org/jira/browse/SPARK-16646

@HyukjinKwon HyukjinKwon deleted the SPARK-16646 branch January 2, 2018 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants