Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor for optimizer #277

Closed
coocood opened this issue Sep 25, 2015 · 3 comments
Closed

Refactor for optimizer #277

coocood opened this issue Sep 25, 2015 · 3 comments
Labels

Comments

@coocood
Copy link
Member

coocood commented Sep 25, 2015

Optimizer need to know everything to do the job, like transform the statement/expression, choose best index to use. But currently the only optimization work is done inside plan.Fillter method, which is not capable for the job because a plan knows nothing more than itself.

The execution flow can be seperated in three stages.

prepare

Validate the statement is legal, make sure table and identifier reference does exists, etc.

optimize

Transform the statement into optimized and normalized form and choose index to use.

execute

Do execution.

Currently the execution is performed directly in Statment.Exec method, as a result, the stmts package imports a lot of dependencies, makes it hard to be imported by other packages.

We have difficulty in doing optimisation right now because of cyclic import restriction.

plans and rsets can not import stmts.

So the proposal is to move the execution functionality out of stmt/stmts package, add a new exec package and do the optimisation and execution work there. So stmt/stmts only keeps basic functionality and can be imported by plans.

The result dependency relation is going to be:

expression -> stmts -> plans -> rsets -> exec

Plan interface and Statement interface and their implmentations are going to be in the same package, this also makes it easier to apply visitor pattern.

@siddontang
Copy link
Member

what's the structure for PostgreSQL?
Maybe we can refer it.

@coocood
Copy link
Member Author

coocood commented Sep 26, 2015

There are many SQL projects, all of them have execution code separated from statements.
We can just extract Exec first, then we can decide how to do optimisation later.

@ngaut
Copy link
Member

ngaut commented Sep 26, 2015

LGTM, We could move step by step.

@coocood coocood closed this as completed Dec 7, 2015
YuJuncen pushed a commit to YuJuncen/tidb that referenced this issue Apr 23, 2021
* restore-util: Implement split/scatter (pingcap#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (pingcap#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (pingcap#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (pingcap#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (pingcap#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (pingcap#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (pingcap#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (pingcap#299)

* restore-util: add method to set placement rules and store labels (pingcap#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (pingcap#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (pingcap#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (pingcap#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (pingcap#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
xhebox pushed a commit to xhebox/tidb that referenced this issue Sep 28, 2021
xhebox pushed a commit to xhebox/tidb that referenced this issue Oct 8, 2021
ti-chi-bot pushed a commit that referenced this issue Oct 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants