docs: add support late materialization rfc #39654
Conversation
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
ti-chi-bot
added
release-note-none
ti-chi-bot
requested review from
fixdb,
winoros,
chrysan,
flowbehappy and
windtalker
|
@Lloyd-Pottiger: GitHub didn't allow me to request PR reviews from the following users: JinheLin. Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
| - Read the needed data pack of rest columns on table t | ||
| - Run the filter `REGEXP_LIKE(col_str2, 'new\\*.\\*line')` |
There was a problem hiding this comment.
After filtering col_str1, why we read all the rest columns instead of only col_str2 column?
There was a problem hiding this comment.
Good point! The most direct reason is most other AP systems do not do like this. 😄 Maybe it is not worth doing that. We can have a try when having enough time.
Co-authored-by: xzhangxian1008 <xzhangxian@foxmail.com>
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
thanks, fixed |
|
It is recommended to write clearly the stage where TiDB selects late materialization predicates. |
Since we need to use statistics, so maybe it can be work as a rule of CBO? |
|
There is no cost in logical rules. Your strategy needs to be executed after the statistics derivation is complete. |
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
|
|
||
| Since we need statistics to calculate the selectivity of the filter conditions, so this algorithm should be executed in postOptimize phase. | ||
|
|
||
| Obviously, beacuse the selectivity of the filter conditions is accurate, and the algorithm is not optimal, we can not guarantee that the pushed down filter conditions are the best. In order to patch a workaround for performance degradation, we will add a optimizer hint `/*+ enable_late_materialization=false */` to disable optimizer to push down filter conditions. |
There was a problem hiding this comment.
I don't recommend add this specific hint, we will have a hint set variable that is applicable to almost all the variable. @Reminiscent
There was a problem hiding this comment.
Do you mean to add a variable rather than a hint? And then we can use set variable enable_late_materialization=false for each query?
There was a problem hiding this comment.
lgtm, I will revise the docs later.
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
Lloyd-Pottiger
force-pushed
the
rfc-support-late-materialization
branch
from
841c792
to
0fc97d2
Compare
|
@flowbehappy: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
|
||
| Since we need statistics to calculate the selectivity of the filter conditions, so this algorithm should be executed in postOptimize phase. | ||
|
|
||
| Obviously, beacuse the selectivity of the filter conditions is accurate, and the algorithm is not optimal, we can not guarantee that the pushed down filter conditions are the best. In order to patch a workaround for performance degradation, we will add a variable `/*+ set_var(enable_late_materialization=false) */` to disable optimizer to push down filter conditions. |
There was a problem hiding this comment.
The set_var hint is not a GA hint ( It is developed by community and no document).
So if you want to set var, please 'set xxx=xxx'
| - Filter conditions that apply to the same column are either pushed down or not pushed down at all. | ||
| - The pushed down filter conditions should not contain too many columns. | ||
|
|
||
| The algorithm is: |
There was a problem hiding this comment.
It needs to be clearly stated in which optimizer phase this function is called. This is very important
There was a problem hiding this comment.
postOptimize
mentioned in line 96
Since we need statistics to calculate the selectivity of the filter conditions, so this algorithm should be executed in postOptimize phase.…e-materialization
ti-chi-bot
added
status/LGT2
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: 88d053c
|
ti-chi-bot
added
the
status/can-merge
|
/retest-required |
You should check the failed test cases first. |
|
/retest-required |
1 similar comment
|
/retest-required |
Signed-off-by: Lloyd-Pottiger yan1579196623@gmail.com
What problem does this PR solve?
Issue Number: ref #40601
Problem Summary:
What is changed and how it works?
add a rfc for supporting late materialization
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.