[feat](nereids) opt condition expression in filter / join / if /case when #57025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yujun777
force-pushed
the
replace-null-for-filter
branch
from
7e13974 to
14f64e7
Compare
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 190165 ms |
ClickBench: Total hot run time: 30.6 s |
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 190510 ms |
ClickBench: Total hot run time: 30.45 s |
Contributor
Author
|
run buildall |
yujun777
force-pushed
the
replace-null-for-filter
branch
from
aa203f9 to
0ec20a4
Compare
Contributor
Author
|
run buildall |
yujun777
changed the title
yujun777
changed the title
Contributor
FE Regression Coverage ReportIncrement line coverage |
yujun777
force-pushed
the
replace-null-for-filter
branch
from
0ec20a4 to
95ca6c5
Compare
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 188578 ms |
ClickBench: Total hot run time: 30.89 s |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 190321 ms |
ClickBench: Total hot run time: 31.03 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
ClickBench: Total hot run time: 30.36 s |
924060929
approved these changes
Oct 20, 2025
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
starocean999
approved these changes
Oct 22, 2025
dwdwqfwe
pushed a commit
to dwdwqfwe/doris
that referenced
this pull request
Oct 24, 2025
…when (apache#57025) For a condition expression, replace null to false, null safe equal to equal. The condition include filter condition, join condition, if condition, case when condition. And for the condition expression, only replace those sub expression which its ancestors to the condition root are all AND / OR / CASE WHEN / IF. So, for a expression `null and not(null)` in a filter, the first null can be replaced, but the second null cann't be replaced because its parent is NOT, not in AND / OR / CASE WHEN / IF. For a expression `x <=> y` in condition, if one of them is not-nullable, then it can rewrite to `x = y`. Note that if a expression is not a condition, `x <=> y` can rewrite to `x = y` require that both x and y are not-nullable. Here is some example: 1. replace null with false ``` null and (null or null) and not(null) => false and (false or false) and not(null) case when null and not (null) then ... => case when false and not (null) then ... if(null and not(null), ...) => if (false and not(null), ...) ``` 2. replace null safe equal to equal if at least one of its child is non-nullable ``` x <=> 3 and not(x <=> 3) => x = 3 and not(x <=> 3) case when x <=> 3 then ... => case when x = 3 then ... if(x <=> 3, ...) => if(x=3, ...) ``` 3. rewrite if expression in condition if its else branch is true / false literal. suppose the if is in a filter, then will have ``` if(c, p, true) => not(c <=> true) or p if(c, p, false) => c and p ``` Currently, the expression rewrite framework cann't known the expression's plan, In order to indicate that the expression is on the filter / join, we add the expression's plan and source to the ExpressionRewriteContext to indicate the expression's location.
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Nov 20, 2025
…when (apache#57025) For a condition expression, replace null to false, null safe equal to equal. The condition include filter condition, join condition, if condition, case when condition. And for the condition expression, only replace those sub expression which its ancestors to the condition root are all AND / OR / CASE WHEN / IF. So, for a expression in a filter, the first null can be replaced, but the second null cann't be replaced because its parent is NOT, not in AND / OR / CASE WHEN / IF. For a expression in condition, if one of them is not-nullable, then it can rewrite to . Note that if a expression is not a condition, can rewrite to require that both x and y are not-nullable.
16 tasks
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Dec 1, 2025
…when (apache#57025) For a condition expression, replace null to false, null safe equal to equal. The condition include filter condition, join condition, if condition, case when condition. And for the condition expression, only replace those sub expression which its ancestors to the condition root are all AND / OR / CASE WHEN / IF. So, for a expression in a filter, the first null can be replaced, but the second null cann't be replaced because its parent is NOT, not in AND / OR / CASE WHEN / IF. For a expression in condition, if one of them is not-nullable, then it can rewrite to . Note that if a expression is not a condition, can rewrite to require that both x and y are not-nullable.
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Dec 2, 2025
…when (apache#57025) For a condition expression, replace null to false, null safe equal to equal. The condition include filter condition, join condition, if condition, case when condition. And for the condition expression, only replace those sub expression which its ancestors to the condition root are all AND / OR / CASE WHEN / IF. So, for a expression in a filter, the first null can be replaced, but the second null cann't be replaced because its parent is NOT, not in AND / OR / CASE WHEN / IF. For a expression in condition, if one of them is not-nullable, then it can rewrite to . Note that if a expression is not a condition, can rewrite to require that both x and y are not-nullable.
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Dec 8, 2025
…when (apache#57025) For a condition expression, replace null to false, null safe equal to equal. The condition include filter condition, join condition, if condition, case when condition. And for the condition expression, only replace those sub expression which its ancestors to the condition root are all AND / OR / CASE WHEN / IF. So, for a expression in a filter, the first null can be replaced, but the second null cann't be replaced because its parent is NOT, not in AND / OR / CASE WHEN / IF. For a expression in condition, if one of them is not-nullable, then it can rewrite to . Note that if a expression is not a condition, can rewrite to require that both x and y are not-nullable.
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jan 9, 2026
…when (apache#57025) For a condition expression, replace null to false, null safe equal to equal. The condition include filter condition, join condition, if condition, case when condition. And for the condition expression, only replace those sub expression which its ancestors to the condition root are all AND / OR / CASE WHEN / IF. So, for a expression in a filter, the first null can be replaced, but the second null cann't be replaced because its parent is NOT, not in AND / OR / CASE WHEN / IF. For a expression in condition, if one of them is not-nullable, then it can rewrite to . Note that if a expression is not a condition, can rewrite to require that both x and y are not-nullable.
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jan 12, 2026
…when (apache#57025) For a condition expression, replace null to false, null safe equal to equal. The condition include filter condition, join condition, if condition, case when condition. And for the condition expression, only replace those sub expression which its ancestors to the condition root are all AND / OR / CASE WHEN / IF. So, for a expression in a filter, the first null can be replaced, but the second null cann't be replaced because its parent is NOT, not in AND / OR / CASE WHEN / IF. For a expression in condition, if one of them is not-nullable, then it can rewrite to . Note that if a expression is not a condition, can rewrite to require that both x and y are not-nullable.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For a condition expression, replace null to false, null safe equal to equal. The condition include filter condition, join condition, if condition, case when condition.
And for the condition expression, only replace those sub expression which its ancestors to the condition root are all AND / OR / CASE WHEN / IF. So, for a expression
null and not(null)in a filter, the first null can be replaced, but the second null cann't be replaced because its parent is NOT, not in AND / OR / CASE WHEN / IF.For a expression
x <=> yin condition, if one of them is not-nullable, then it can rewrite tox = y. Note that if a expression is not a condition,x <=> ycan rewrite tox = yrequire that both x and y are not-nullable.Here is some example:
suppose the if is in a filter, then will have
Currently, the expression rewrite framework cann't known the expression's plan, In order to indicate that the expression is on the filter / join, we add the expression's plan and source to the ExpressionRewriteContext to indicate the expression's location.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)