-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor for optimizer #277
Labels
Comments
what's the structure for PostgreSQL? |
There are many SQL projects, all of them have execution code separated from statements. |
LGTM, We could move step by step. |
YuJuncen
pushed a commit
to YuJuncen/tidb
that referenced
this issue
Apr 23, 2021
* restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com>
xhebox
pushed a commit
to xhebox/tidb
that referenced
this issue
Sep 28, 2021
xhebox
pushed a commit
to xhebox/tidb
that referenced
this issue
Oct 8, 2021
ti-chi-bot
pushed a commit
that referenced
this issue
Oct 9, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Optimizer need to know everything to do the job, like transform the statement/expression, choose best index to use. But currently the only optimization work is done inside plan.Fillter method, which is not capable for the job because a plan knows nothing more than itself.
The execution flow can be seperated in three stages.
prepare
Validate the statement is legal, make sure table and identifier reference does exists, etc.
optimize
Transform the statement into optimized and normalized form and choose index to use.
execute
Do execution.
Currently the execution is performed directly in
Statment.Exec
method, as a result, thestmts
package imports a lot of dependencies, makes it hard to be imported by other packages.We have difficulty in doing optimisation right now because of cyclic import restriction.
plans and rsets can not import stmts.
So the proposal is to move the execution functionality out of
stmt/stmts
package, add a newexec
package and do the optimisation and execution work there. Sostmt/stmts
only keeps basic functionality and can be imported by plans.The result dependency relation is going to be:
expression -> stmts -> plans -> rsets -> exec
Plan interface and Statement interface and their implmentations are going to be in the same package, this also makes it easier to apply visitor pattern.
The text was updated successfully, but these errors were encountered: