Skip to content

Commit

Permalink
[SPARK-36926][SQL] Decimal average mistakenly overflow
Browse files Browse the repository at this point in the history
This bug was introduced by apache#33177

When checking overflow of the sum value in the average function, we should use the `sumDataType` instead of the input decimal type.

fix a regression

Yes, the result was wrong before this PR.

a new test

Closes apache#34180 from cloud-fan/bug.

Lead-authored-by: Wenchen Fan <wenchen@databricks.com>
Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
  • Loading branch information
cloud-fan and cloud-fan committed Oct 6, 2021
1 parent c542297 commit 0c59f93
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,10 @@ case class Average(
// If all input are nulls, count will be 0 and we will get null after the division.
// We can't directly use `/` as it throws an exception under ansi mode.
override lazy val evaluateExpression = child.dataType match {
case d: DecimalType =>
case _: DecimalType =>
DecimalPrecision.decimalAndDecimal()(
Divide(
CheckOverflowInSum(sum, d, !failOnError),
CheckOverflowInSum(sum, sumDataType.asInstanceOf[DecimalType], !failOnError),
count.cast(DecimalType.LongDecimal), failOnError = false)).cast(resultType)
case _: YearMonthIntervalType =>
If(EqualTo(count, Literal(0L)),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1416,6 +1416,12 @@ class DataFrameAggregateSuite extends QueryTest
val df2 = Seq(Period.ofYears(1)).toDF("a").groupBy("a").count()
checkAnswer(df2, Row(Period.ofYears(1), 1))
}

test("SPARK-36926: decimal average mistakenly overflow") {
val df = (1 to 10).map(_ => "9999999999.99").toDF("d")
val res = df.select($"d".cast("decimal(12, 2)").as("d")).agg(avg($"d").cast("string"))
checkAnswer(res, Row("9999999999.990000"))
}
}

case class B(c: Option[Double])
Expand Down

0 comments on commit 0c59f93

Please sign in to comment.