Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: derive col is not null from col op col conditions #8603

Merged
merged 3 commits into from
Jan 30, 2019
Merged

*: derive col is not null from col op col conditions #8603

merged 3 commits into from
Jan 30, 2019

Conversation

eurekaka
Copy link
Contributor

@eurekaka eurekaka commented Dec 6, 2018 •
edited
Loading

What problem does this PR solve?

Fix #8587

What is changed and how it works?

  • Check whether the col op col condition is null rejected, if yes, derive col is not null for both sides of inner join, and for inner side of outer join.
  • Remove duplicate conditions before pushing them down. Note that we check if the condition contains mutable functions before de-duplicating.
  • Check NotNull flag of child column before generating is not null expression to avoid unnecessary conditions.
  • Handle anti join specially.

Check List

Tests

  • Unit test

Code changes

  • Has exported function/method change

Side effects

  • Increased code complexity

Related changes

N/A

This change is Reviewable

@eurekaka eurekaka added type/enhancement The issue or PR belongs to an enhancement. status/WIP sig/planner SIG: Planner component/expression labels Dec 6, 2018
@eurekaka
Copy link
Contributor Author

/run-all-tests

@eurekaka
Copy link
Contributor Author

/run-all-tests

@eurekaka
Copy link
Contributor Author

/run-unit-test

@eurekaka
Copy link
Contributor Author

/rebuid

@eurekaka
Copy link
Contributor Author

/run-all-tests

@eurekaka
Copy link
Contributor Author

randgen test failures can be fixed by #8725, wait it to be merged first.

@XuHuaiyu XuHuaiyu changed the title plan: derive col is not null from col op col conditions *: derive col is not null from col op col conditions Dec 19, 2018
@eurekaka
Copy link
Contributor Author

/run-all-tests

@eurekaka
Copy link
Contributor Author

/run-mybatis-test

@eurekaka
Copy link
Contributor Author

@zz-jason @winoros @lamxTyler PTAL

@eurekaka
Copy link
Contributor Author

@lamxTyler comments addressed, PTAL

alivxxx
alivxxx reviewed Dec 21, 2018
View reviewed changes
expression/util.go Outdated Show resolved Hide resolved
expression/util.go Outdated Show resolved Hide resolved
@eurekaka
Copy link
Contributor Author

@lamxTyler comments addressed, PTAL

alivxxx
alivxxx reviewed Dec 21, 2018
View reviewed changes
Copy link
Contributor

@alivxxx alivxxx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alivxxx alivxxx added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 21, 2018
@eurekaka eurekaka force-pushed the derive_not_null branch 2 times, most recently from 88075a6 to d61a4ef Compare December 28, 2018 12:49
@eurekaka
Copy link
Contributor Author

@zz-jason PTAL. I have updated the PR to not derive is not null from column equal condition of LeftOuterSemiJoin either, for example, for queries like select a, a in (select a from t t2 where t2.b > t1.b) from t t1, no t2.a is not null is generated, but we would still generate t2.b is not null, which looks correct?

mysql> explain select a, a in (select a from t t2 where t2.b > t1.b) from t t1;
+--------------------------+-------+------+-----------------------------------------------------------------------------------------------+
| id                       | count | task | operator info                                                                                 |
+--------------------------+-------+------+-----------------------------------------------------------------------------------------------+
| Projection_7             | 4.00  | root | t1.a, 6_aux_0                                                                                 |
| └─HashLeftJoin_8         | 4.00  | root | left outer semi join, inner:TableReader_13, equal:[eq(t1.a, t2.a)], other cond:gt(t2.b, t1.b) |
|   ├─TableReader_10       | 4.00  | root | data:TableScan_9                                                                              |
|   │ └─TableScan_9        | 4.00  | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo                                   |
|   └─TableReader_13       | 4.00  | root | data:Selection_12                                                                             |
|     └─Selection_12       | 4.00  | cop  | not(isnull(t2.b))                                                                             |
|       └─TableScan_11     | 4.00  | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo                                   |
+--------------------------+-------+------+-----------------------------------------------------------------------------------------------+

winoros
winoros reviewed Jan 2, 2019
View reviewed changes
executor/join_test.go Show resolved Hide resolved
expression/constant_propagation.go Show resolved Hide resolved
@eurekaka
plan: derive col is not null from col op col conditions
3e864de 
Check whether the `op` condition is null rejected, if yes, derive
`col is not null` for both sides of inner join, and for inner side
of outer join.

Also, remove duplicate conditions before pushing them down. Check
if the condition is mutable before removing duplicates.

For AntiSemiJoin, AntiLeftOuterSemiJoin and LeftOuterSemiJoin, do
not generate new `is not null` conditions.
@eurekaka
Copy link
Contributor Author

/run-all-tests

@eurekaka
Copy link
Contributor Author

/run-integration-ddl-test
/run-unit-test

@eurekaka eurekaka requested review from zz-jason and winoros and removed request for winoros January 16, 2019 09:26
zz-jason
zz-jason reviewed Jan 29, 2019
View reviewed changes
Copy link
Member

@zz-jason zz-jason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zz-jason zz-jason added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 29, 2019
@codecov-io
Copy link

Codecov Report

Merging #8603 into master will increase coverage by <.01%.
The diff coverage is 86.45%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8603      +/-   ##
==========================================
+ Coverage    67.2%   67.21%   +<.01%     
==========================================
  Files         371      371              
  Lines       76985    77050      +65     
==========================================
+ Hits        51741    51791      +50     
- Misses      20624    20631       +7     
- Partials     4620     4628       +8
Impacted Files Coverage Δ
expression/builtin.go 84% <ø> (ø) ⬆️
planner/core/logical_plans.go 73.72% <100%> (ø) ⬆️
planner/core/expression_rewriter.go 73.61% <100%> (-0.17%) ⬇️
planner/core/logical_plan_builder.go 74.86% <100%> (+0.17%) ⬆️
expression/constant_propagation.go 95.56% <100%> (+0.74%) ⬆️
expression/util.go 72% <77.77%> (+0.48%) ⬆️
planner/core/rule_predicate_push_down.go 89.16% <79.41%> (-1.48%) ⬇️
util/systimemon/systime_mon.go 80% <0%> (-20%) ⬇️
ddl/delete_range.go 75.13% <0%> (-4.24%) ⬇️
executor/merge_join.go 80.45% <0%> (-1.15%) ⬇️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20c1b95...b04bd72. Read the comment docs.

@eurekaka eurekaka merged commit 09a5c80 into pingcap:master Jan 30, 2019
@eurekaka eurekaka deleted the derive_not_null branch January 30, 2019 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alivxxx alivxxx alivxxx left review comments

@XuHuaiyu XuHuaiyu XuHuaiyu left review comments

@zz-jason zz-jason zz-jason approved these changes

@winoros winoros Awaiting requested review from winoros

Assignees
No one assigned
Labels
component/expression sig/planner SIG: Planner status/LGT2 Indicates that a PR has LGTM 2. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@eurekaka @codecov-io @zz-jason @winoros @XuHuaiyu @alivxxx