-
Notifications
You must be signed in to change notification settings - Fork 29k
[Do not merge][SPARK-32018][SQL][3.0][FOLLOWUP] Fix a test failure in DataFrameSuite #29448
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| assert(e.getCause.getClass.equals(classOf[ArithmeticException])) | ||
| assert(e.getCause.getMessage.contains("cannot be represented as Decimal")) | ||
| val e = intercept[SparkException] { | ||
| structDf.collect |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add a comment to say that we have to fail overflow with non-ansi mode as well, and link the related JIRA tickets?
|
I am now thinking about reverting the previous commits. |
|
Test build #127497 has finished for PR 29448 at commit
|
|
Actually this is more tricky than I thought. In 3.0/2.4 without the unsafe row bug fix:
(2) is very tricky:
The failed test will not work even if we revert the commit, if we change input DataFrame partition number to 1, to trigger overflow in partial aggregate. To give a summary for 3.0/2.4:
That said, #29404 introduced breaking changes to (2), as it always fails for overflow. Let's revert it. For the unsafe row bug fix #29125, it's important as unsafe row binary is used to check equality in many places (join key, grouping key, window partition key, etc.), but it also makes (1) worse as Spark may return the wrong result. We can simply revert it as well, or we re-apply #29404 only to hash aggregate with GROUP BY to avoid breaking changes (can be complex if we consider distinct aggregate). |
What changes were proposed in this pull request?
Fix a test failure in DataFrameSuite introduced by #29404
Why are the changes needed?
Fix test failure
Does this PR introduce any user-facing change?
No
How was this patch tested?
Unit test