Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cte executors done #4

Open
wants to merge 19 commits into
base: split_seed_and_recursive
Choose a base branch
from

Conversation

guo-shaoge
Copy link

  1. row conatiner refine done
  2. union all/union distinct done

Signed-off-by: guo-shaoge shaoge1994@163.com

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Proposal: xxx

What's Changed:

How it Works:

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM
  • Breaking backward compatibility

Release note

@guo-shaoge guo-shaoge force-pushed the new_cte_exec branch 2 times, most recently from 6155af2 to b7070ac Compare April 29, 2021 09:33
@guo-shaoge guo-shaoge force-pushed the new_cte_exec branch 3 times, most recently from e256c8c to 6f89165 Compare May 8, 2021 06:29
guo-shaoge and others added 15 commits May 10, 2021 13:33
2. row conatiner refine done
3. union all/union distinct done

Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
1. add setupCTEStorageTracker. fix tracker infinite recursion
2. add begCh to CTEStorage. CTEExec will notify CTETableReaderExec when iterInTbl is first setup

Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Move CTEStorage to StmtCtx. ref count is deprecated.
We need a better way to fix subquery problem.(CTEStorage of subquery is
not shared with outer CTEStorage).

Signed-off-by: guo-shaoge <shaoge1994@163.com>
1. unlock() manually -> defer unlock.
2. reopen CTEStorage when we got error during the filling of CTEStorage.

Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge force-pushed the new_cte_exec branch 2 times, most recently from 1471826 to f492570 Compare May 14, 2021 10:14
Signed-off-by: guo-shaoge <shaoge1994@163.com>
2. move CTEStorage to uilt/cte_storage package

Signed-off-by: guo-shaoge <shaoge1994@163.com>
wjhuang2016 pushed a commit that referenced this pull request Jun 16, 2022
create sysvar TiDBFastDDL (#3)
* Change TiDBFastDDL to Global scope.
* TiDBFastDDL Variable usage
1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set
reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job.
2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process.
Complete lightning env initlize procdeure (#4)
1. Get max open file limition for lightning, default 1024
2. Set light sorted path:
   2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning
   2.2 otherwise set path to DataDir + "lightning"
   Check whether the lightning sorted path is exist and a dir, if yes then keep use it.
   otherwise create new one.
3, Set the lightning dir quota, default 10G
Add one unit test
uniform lightning related errmessage text to lightning_error file. (#5)
complete memory track module work.
refacter memmory manager to resource manager for expand to control concurrent
base available  cpu core number.
Finsih integrate with lightning concurrency process logic.
Add log for lightning processing.
refactor some code`
complete memory track module work. (#7)
complete user cancel and exception part logic implement
* complete memory track module work.
* refacter memmory manager to resource manager for expand to control concurrent
base available  cpu core number.

Finsih integrate with lightning concurrency process logic.

* Add log for lightning processing.
refactor some code`
restore logic first stage
adjust metric of ddl process progress value for lightning solution
refine import cycle
set up disk quota
refine code and add some ut and ft.
Add config paramemter TiDBlightningSortPath for setting sort parth for add index.
fix mem reclaim problems
Add log infromation
combine the optimizes:
1, prundecode
2, json expression
Add some log information
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants