Complete memory track module work. #7
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
knull-cn
reviewed
May 24, 2022
| @@ -156,7 +160,7 @@ func FlushKeyValSync(ctx context.Context, keyEngineInfo string, cache *WorkerKVC | |||
| err = ei.unsafeImportAndReset(ctx) | |||
| if err != nil { | |||
| // LogError("unsafeImportAndReset %s cost %v", size2str(sn.szInc.encodeSize), time.Now().Sub(start)) | |||
| // 仅仅是导入失败，下次还是可以继续导入，不影响. | |||
| // Only import failed, next time can import continue. | |||
| return nil | |||
There was a problem hiding this comment.
i think ,there can print some log
knull-cn
reviewed
May 24, 2022
| @@ -229,7 +233,7 @@ func fetchTableRegionSizeStats(tblId int64, exec sqlexec.RestrictedSQLExecutor) | |||
| return ret, nil | |||
| } | |||
|
|
|||
| // TODO: 如果多个 线程 同时调用该函数，是否存在这样情况? 如果存在，该怎么处理? | |||
| // TODO: If multi thread call this function, how to handle this logic? | |||
| func FinishIndexOp(ctx context.Context, keyEngineInfo string, tbl table.Table, unique bool) (err error) { | |||
There was a problem hiding this comment.
这里的多个线程调用，大约是这么个情况：定时查看是否 完成，如果完成，则会走 finish。但是，可能触发了多次，导致多个线程同时调用。这里需要处理下 —— 我记得这就是为啥 要 mutex 的一个原因
Benjamin2037
pushed a commit
that referenced
this pull request
Jun 1, 2022
create sysvar TiDBFastDDL (#3) * Change TiDBFastDDL to Global scope. * TiDBFastDDL Variable usage 1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job. 2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process. Complete lightning env initlize procdeure (#4) 1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test uniform lightning related errmessage text to lightning_error file. (#5) complete memory track module work. refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. Add log for lightning processing. refactor some code` complete memory track module work. (#7) complete user cancel and exception part logic implement * complete memory track module work. * refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. * Add log for lightning processing. refactor some code` restore logic first stage adjust metric of ddl process progress value for lightning solution refine import cycle set up disk quota refine code and add some ut and ft. Add config paramemter TiDBlightningSortPath for setting sort parth for add index. fix mem reclaim problems Add log infromation combine the optimizes: 1, prundecode 2, json expression Add some log information
Benjamin2037
pushed a commit
that referenced
this pull request
Jun 1, 2022
create sysvar TiDBFastDDL (#3) * Change TiDBFastDDL to Global scope. * TiDBFastDDL Variable usage 1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job. 2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process. Complete lightning env initlize procdeure (#4) 1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test uniform lightning related errmessage text to lightning_error file. (#5) complete memory track module work. refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. Add log for lightning processing. refactor some code` complete memory track module work. (#7) complete user cancel and exception part logic implement * complete memory track module work. * refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. * Add log for lightning processing. refactor some code` restore logic first stage adjust metric of ddl process progress value for lightning solution refine import cycle set up disk quota refine code and add some ut and ft. Add config paramemter TiDBlightningSortPath for setting sort parth for add index. fix mem reclaim problems Add log infromation combine the optimizes: 1, prundecode 2, json expression Add some log information
Benjamin2037
added a commit
that referenced
this pull request
Jun 1, 2022
create sysvar TiDBFastDDL (#3) * Change TiDBFastDDL to Global scope. * TiDBFastDDL Variable usage 1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job. 2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process. Complete lightning env initlize procdeure (#4) 1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test uniform lightning related errmessage text to lightning_error file. (#5) complete memory track module work. refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. Add log for lightning processing. refactor some code` complete memory track module work. (#7) complete user cancel and exception part logic implement * complete memory track module work. * refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. * Add log for lightning processing. refactor some code` restore logic first stage adjust metric of ddl process progress value for lightning solution refine import cycle set up disk quota refine code and add some ut and ft. Add config paramemter TiDBlightningSortPath for setting sort parth for add index. fix mem reclaim problems Add log infromation combine the optimizes: 1, prundecode 2, json expression Add some log information
Benjamin2037
added a commit
that referenced
this pull request
Jun 15, 2022
create sysvar TiDBFastDDL (#3) * Change TiDBFastDDL to Global scope. * TiDBFastDDL Variable usage 1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job. 2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process. Complete lightning env initlize procdeure (#4) 1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test uniform lightning related errmessage text to lightning_error file. (#5) complete memory track module work. refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. Add log for lightning processing. refactor some code` complete memory track module work. (#7) complete user cancel and exception part logic implement * complete memory track module work. * refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. * Add log for lightning processing. refactor some code` restore logic first stage adjust metric of ddl process progress value for lightning solution refine import cycle set up disk quota refine code and add some ut and ft. Add config paramemter TiDBlightningSortPath for setting sort parth for add index. fix mem reclaim problems Add log infromation combine the optimizes: 1, prundecode 2, json expression Add some log information
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
What is changed and how it works?
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.