[Feat](Nereids) add numeric functions #40744
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
LiBinfeng-01
force-pushed
the
fold_const_numeric
branch
from
fa2f58a to
76a098e
Compare
|
run buildall |
TPC-H: Total hot run time: 43005 ms |
TPC-DS: Total hot run time: 195717 ms |
ClickBench: Total hot run time: 31.02 s |
LiBinfeng-01
force-pushed
the
fold_const_numeric
branch
2 times, most recently
from
37061e4 to
4d76abb
Compare
|
run buildall |
TPC-H: Total hot run time: 43027 ms |
LiBinfeng-01
force-pushed
the
fold_const_numeric
branch
from
4d76abb to
5d90267
Compare
|
run buildall |
TPC-H: Total hot run time: 42953 ms |
|
run buildall |
TPC-H: Total hot run time: 41212 ms |
|
run performance |
|
run p0 |
TPC-H: Total hot run time: 41979 ms |
TPC-DS: Total hot run time: 199206 ms |
ClickBench: Total hot run time: 32.1 s |
LiBinfeng-01
force-pushed
the
fold_const_numeric
branch
from
309ada3 to
a9d0c39
Compare
|
run buildall |
|
run feut |
|
run performance |
morrySnow
reviewed
Sep 18, 2024
| @@ -84,6 +84,12 @@ public DecimalV3Literal roundFloor(int newScale) { | |||
| value.setScale(newScale, RoundingMode.FLOOR)); | |||
| } | |||
|
|
|||
| public DecimalV3Literal round(int newScale) { | |||
| return new DecimalV3Literal(DecimalV3Type | |||
| .createDecimalV3Type(((DecimalV3Type) dataType).getPrecision(), newScale), | |||
There was a problem hiding this comment.
what scale should be set if newScale bigger than old scale?
There was a problem hiding this comment.
when new scale is larger than original one, it would throw an Invocationexception when calling create Decimal Type in java with message: precision should not smaller than scale and ExpressionEvaluator would catch this exception and go back to original expression. But actually when new scalar bigger than old one, we can return original one when folding constant. Same as be calculation logic
There was a problem hiding this comment.
when new scale is larger than original one, it would throw an Invocationexception when calling create Decimal Type in java with message: precision should not smaller than scale and ExpressionEvaluator would catch this exception and go back to original expression. But actually when new scalar bigger than old one, we can return original one when folding constant. Same as be calculation logic
we should not rely on exception, exception is very expensive, so we should process it without use exception
| */ | ||
| @ExecFunction(name = "ceil") | ||
| public static Expression ceil(DecimalV3Literal first, IntegerLiteral second) { | ||
| return first.roundCeiling(second.getValue()); |
There was a problem hiding this comment.
same as round, if second > old scale, the result is wrong
There was a problem hiding this comment.
move if into roundCeiling
| */ | ||
| @ExecFunction(name = "floor") | ||
| public static Expression floor(DecimalV3Literal first, IntegerLiteral second) { | ||
| return first.roundFloor(second.getValue()); |
| */ | ||
| @ExecFunction(name = "exp") | ||
| public static Expression exp(DoubleLiteral first) { | ||
| return new DoubleLiteral(Math.exp(first.getValue())); |
There was a problem hiding this comment.
what will happen if result of Math.exp is infinity?
There was a problem hiding this comment.
It should return null literal, in java it would return infinite value when it is infinity, would return NaN when the input is Invalid like acos(-2) which input should between [-1, 1]. In sql behavior, it should return NULL, when it is NaN or infinity(positive, negative is same logic). Let me add some boundary check in some functions
There was a problem hiding this comment.
I was wrong about Infinity, actually it should return "inf" in sql and " ∞" in explain string. NaN is right to show by NULL
.../java/org/apache/doris/nereids/trees/expressions/functions/executable/NumericArithmetic.java
Show resolved
Hide resolved
|
run buildall |
TPC-H: Total hot run time: 41728 ms |
TPC-DS: Total hot run time: 194806 ms |
ClickBench: Total hot run time: 33.38 s |
16 tasks
morrySnow
pushed a commit
that referenced
this pull request
Nov 12, 2024
…recision problem (#43422) - Problem function like ```select floor(300.343, 2)``` precision should be 5 and scale should be 2, but now is (6, 2) after compute precision, but after folding const on fe, it changed to (5, 2) but upper level of plan still expect the output of child to be (6, 2). So it would rise an exception when executing. - How it was fixed fix folding constant precision of floor/round/ceil/truncate functions from (5, 2) to (6, 2) in upper case - Notion when second value is negative and it absolute value >= precision - value, it can not be expressed in fe which result is zero with decimal type (3, 0). like 000. So just let it go back and no using folding constant by fe. - Related PR: #40744 - Release note Fix floor/round/ceil functions precision problem in folding constant
github-actions bot
pushed a commit
that referenced
this pull request
Nov 12, 2024
…recision problem (#43422) - Problem function like ```select floor(300.343, 2)``` precision should be 5 and scale should be 2, but now is (6, 2) after compute precision, but after folding const on fe, it changed to (5, 2) but upper level of plan still expect the output of child to be (6, 2). So it would rise an exception when executing. - How it was fixed fix folding constant precision of floor/round/ceil/truncate functions from (5, 2) to (6, 2) in upper case - Notion when second value is negative and it absolute value >= precision - value, it can not be expressed in fe which result is zero with decimal type (3, 0). like 000. So just let it go back and no using folding constant by fe. - Related PR: #40744 - Release note Fix floor/round/ceil functions precision problem in folding constant
github-actions bot
pushed a commit
that referenced
this pull request
Nov 12, 2024
…recision problem (#43422) - Problem function like ```select floor(300.343, 2)``` precision should be 5 and scale should be 2, but now is (6, 2) after compute precision, but after folding const on fe, it changed to (5, 2) but upper level of plan still expect the output of child to be (6, 2). So it would rise an exception when executing. - How it was fixed fix folding constant precision of floor/round/ceil/truncate functions from (5, 2) to (6, 2) in upper case - Notion when second value is negative and it absolute value >= precision - value, it can not be expressed in fe which result is zero with decimal type (3, 0). like 000. So just let it go back and no using folding constant by fe. - Related PR: #40744 - Release note Fix floor/round/ceil functions precision problem in folding constant
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Nov 12, 2024
…recision problem (apache#43422) - Problem function like ```select floor(300.343, 2)``` precision should be 5 and scale should be 2, but now is (6, 2) after compute precision, but after folding const on fe, it changed to (5, 2) but upper level of plan still expect the output of child to be (6, 2). So it would rise an exception when executing. - How it was fixed fix folding constant precision of floor/round/ceil/truncate functions from (5, 2) to (6, 2) in upper case - Notion when second value is negative and it absolute value >= precision - value, it can not be expressed in fe which result is zero with decimal type (3, 0). like 000. So just let it go back and no using folding constant by fe. - Related PR: apache#40744 - Release note Fix floor/round/ceil functions precision problem in folding constant
16 tasks
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Nov 12, 2024
…recision problem (apache#43422) - Problem function like ```select floor(300.343, 2)``` precision should be 5 and scale should be 2, but now is (6, 2) after compute precision, but after folding const on fe, it changed to (5, 2) but upper level of plan still expect the output of child to be (6, 2). So it would rise an exception when executing. - How it was fixed fix folding constant precision of floor/round/ceil/truncate functions from (5, 2) to (6, 2) in upper case - Notion when second value is negative and it absolute value >= precision - value, it can not be expressed in fe which result is zero with decimal type (3, 0). like 000. So just let it go back and no using folding constant by fe. - Related PR: apache#40744 - Release note Fix floor/round/ceil functions precision problem in folding constant
16 tasks
py023
pushed a commit
to py023/doris
that referenced
this pull request
Nov 13, 2024
…recision problem (apache#43422) - Problem function like ```select floor(300.343, 2)``` precision should be 5 and scale should be 2, but now is (6, 2) after compute precision, but after folding const on fe, it changed to (5, 2) but upper level of plan still expect the output of child to be (6, 2). So it would rise an exception when executing. - How it was fixed fix folding constant precision of floor/round/ceil/truncate functions from (5, 2) to (6, 2) in upper case - Notion when second value is negative and it absolute value >= precision - value, it can not be expressed in fe which result is zero with decimal type (3, 0). like 000. So just let it go back and no using folding constant by fe. - Related PR: apache#40744 - Release note Fix floor/round/ceil functions precision problem in folding constant
16 tasks
morrySnow
pushed a commit
that referenced
this pull request
Nov 18, 2024
…cion (#44022) Related PR: #40744 when executing floor(1) it would castTo decimalV3(3,0) because it need (3,0) to contain it's message. But after fold const, it lost precision(3) because decimalV3 literal class does not have mechanism to save precision Solved: after folding constant, we need to change result type to the type we wanted
github-actions bot
pushed a commit
that referenced
this pull request
Nov 18, 2024
…cion (#44022) Related PR: #40744 when executing floor(1) it would castTo decimalV3(3,0) because it need (3,0) to contain it's message. But after fold const, it lost precision(3) because decimalV3 literal class does not have mechanism to save precision Solved: after folding constant, we need to change result type to the type we wanted
github-actions bot
pushed a commit
that referenced
this pull request
Nov 18, 2024
…cion (#44022) Related PR: #40744 when executing floor(1) it would castTo decimalV3(3,0) because it need (3,0) to contain it's message. But after fold const, it lost precision(3) because decimalV3 literal class does not have mechanism to save precision Solved: after folding constant, we need to change result type to the type we wanted
16 tasks
16 tasks
morrySnow
pushed a commit
that referenced
this pull request
Feb 19, 2025
…o match with be execution behavior (#47966) ### What problem does this PR solve? Related PR: #40744， #47228 Problem Summary: When numeric function input or output out of boundary when fe folding constant, it would throw an exception before. Now we change it to match with be execution result, which is NullLiteral
github-actions bot
pushed a commit
that referenced
this pull request
Feb 19, 2025
…o match with be execution behavior (#47966) ### What problem does this PR solve? Related PR: #40744， #47228 Problem Summary: When numeric function input or output out of boundary when fe folding constant, it would throw an exception before. Now we change it to match with be execution result, which is NullLiteral
lzyy2024
pushed a commit
to lzyy2024/doris
that referenced
this pull request
Feb 21, 2025
…o match with be execution behavior (apache#47966) ### What problem does this PR solve? Related PR: apache#40744， apache#47228 Problem Summary: When numeric function input or output out of boundary when fe folding constant, it would throw an exception before. Now we change it to match with be execution result, which is NullLiteral
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Feb 24, 2025
…o match with be execution behavior (apache#47966) Related PR: apache#40744， apache#47228 Problem Summary: When numeric function input or output out of boundary when fe folding constant, it would throw an exception before. Now we change it to match with be execution result, which is NullLiteral
16 tasks
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Feb 25, 2025
…o match with be execution behavior (apache#47966) ### What problem does this PR solve? Related PR: apache#40744， apache#47228 Problem Summary: When numeric function input or output out of boundary when fe folding constant, it would throw an exception before. Now we change it to match with be execution result, which is NullLiteral
16 tasks
yiguolei
pushed a commit
that referenced
this pull request
Feb 25, 2025
…y behavior to match with be execution behavior #47966 (#48237) pick:#47966 Related PR: #40744， #47228 Problem Summary: When numeric function input or output out of boundary when fe folding constant, it would throw an exception before. Now we change it to match with be execution result, which is NullLiteral ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
dataroaring
pushed a commit
that referenced
this pull request
Feb 26, 2025
…y behavior to match with be execution behavior (#47966) (#48281) pick:#47966 ### What problem does this PR solve? Related PR: #40744， #47228 Problem Summary: When numeric function input or output out of boundary when fe folding constant, it would throw an exception before. Now we change it to match with be execution result, which is NullLiteral
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
add fold constant on fe of numeric functions:
Coalesce, Round, Ceil, Floor, Exp, Ln, Log, Log10, Log2, Sqrt, Power, Sin, Cos, Tan, Acos, Asin, Atan, Atan2, Sign, Bin, BitCount, BitLength, Cbrt, Cosh, Tanh, Dexp, Dlog10, Dlog1, Dpow, Dsqrt, Fmod, Fpow, Radians, Degrees, Xor, Pi, E, Conv, Truncate, CountEqual, Pmod.