-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-30754][SQL] Reuse results of floorDiv in calculations of floorMod in DateTimeUtils #27491
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #118031 has finished for PR 27491 at commit
|
|
It's pretty trivially faster, I'd imagine, but the logic looks sound and it's not more complex |
|
Can we use a kind of this optimization in HotSpot? |
|
@kiszk The ops are not regular arithmetic |
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
Show resolved
Hide resolved
|
The fix looks fine to me. If you have performance numbers, could you put them in the PR decription? |
|
@MaxGekk Got it. This is not a pair of |
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
Show resolved
Hide resolved
|
Test build #118122 has finished for PR 27491 at commit
|
|
jenkins, retest this, please |
maropu
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine to me. I'll leave this to @srowen and the other committers. cc: @dongjoon-hyun @HyukjinKwon
|
Test build #118140 has finished for PR 27491 at commit
|
|
Merged to master (leaving out of 3.0) |
…Mod in DateTimeUtils
### What changes were proposed in this pull request?
In the case of back-to-back calculation of `floorDiv` and `floorMod` with the same arguments, the result of `foorDiv` can be reused in calculation of `floorMod`. The `floorMod` method is defined as the following in Java standard library:
```java
public static int floorMod(int x, int y) {
int r = x - floorDiv(x, y) * y;
return r;
}
```
If `floorDiv(x, y)` has been already calculated, it can be reused in `x - floorDiv(x, y) * y`.
I propose to modify 2 places in `DateTimeUtils`:
1. `microsToInstant` which is widely used in many date-time functions. `Math.floorMod(us, MICROS_PER_SECOND)` is just replaced by its definition from Java Math library.
2. `truncDate`: `Math.floorMod(oldYear, divider) == 0` is replaced by `Math.floorDiv(oldYear, divider) * divider == oldYear` where `floorDiv(...) * divider` is pre-calculated.
### Why are the changes needed?
This reduces the number of arithmetic operations, and can slightly improve performance of date-time functions.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
By existing test suites `DateTimeUtilsSuite`, `DateFunctionsSuite` and `DateExpressionsSuite`.
Closes apache#27491 from MaxGekk/opt-microsToInstant.
Authored-by: Maxim Gekk <max.gekk@gmail.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
What changes were proposed in this pull request?
In the case of back-to-back calculation of
floorDivandfloorModwith the same arguments, the result offoorDivcan be reused in calculation offloorMod. ThefloorModmethod is defined as the following in Java standard library:If
floorDiv(x, y)has been already calculated, it can be reused inx - floorDiv(x, y) * y.I propose to modify 2 places in
DateTimeUtils:microsToInstantwhich is widely used in many date-time functions.Math.floorMod(us, MICROS_PER_SECOND)is just replaced by its definition from Java Math library.truncDate:Math.floorMod(oldYear, divider) == 0is replaced byMath.floorDiv(oldYear, divider) * divider == oldYearwherefloorDiv(...) * divideris pre-calculated.Why are the changes needed?
This reduces the number of arithmetic operations, and can slightly improve performance of date-time functions.
Does this PR introduce any user-facing change?
No
How was this patch tested?
By existing test suites
DateTimeUtilsSuite,DateFunctionsSuiteandDateExpressionsSuite.