add priv step1 #281
Merged
add priv step1 #281
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Prepare for grant statement.
| @@ -147,6 +147,10 @@ func TypeToStr(tp byte, binary bool) string { | |||
| return "timestamp" | |||
| case mysql.TypeBit: | |||
| return "bit" | |||
| case mysql.TypeEnum: | |||
There was a problem hiding this comment.
It's better. But I am blocked here and can not wait.
|
LGTM |
1 similar comment
|
LGTM |
YuJuncen
pushed a commit
to YuJuncen/tidb
that referenced
this pull request
Apr 23, 2021
* restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com>
YuJuncen
pushed a commit
to YuJuncen/tidb
that referenced
this pull request
Apr 23, 2021
* lightning: split large csv file if possible
* gofmt
* gofmt
* unit test
* add unit test
* tiny change
* tiny refine
* fix ci
* remove useless code
* fix ci
* fix ci
* address comments
* go fmt for all
* Replace CSV Ragel parser by a hand-written parser copied from encoding/csv
Conflicts:
lightning/mydump/csv_parser_generated.go
* fix conflict
* update
* update again
* send a batch of kv in encodeLoop
* use sync.Pool
* Close channel instead of push one entry.
* Use copy instead append
* Fix test and failpoint version
* Reuse slice of record
This expected to avoid about 3.5% of alloc_objects
alloc_objects:
Total: 773496750 773873722 (flat, cum) 7.18%
177 . . parser.fieldIndexes = parser.fieldIndexes[:0]
178 . .
179 . . isEmptyLine := true
...
225 386621314 386621314 str := string(parser.recordBuffer) // Convert to string once to batch allocations
226 386875436 386875436 dst := make([]string, len(parser.fieldIndexes))
* Use pool for mutation
This take most alloc in WriteRows:
ROUTINE ======================== github.com/pingcap/tidb-lightning/lightning/backend.(*importer).WriteRows in /Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/backend/importer.go
797370418 980241246 (flat, cum) 9.09% of Total
. . 155: kvs := rows.(kvPairs)
...
...
. . 192: for i, pair := range kvs {
772641868 772641868 193: mutations[i] = &kv.Mutation{
. . 194: Op: kv.Mutation_Put,
. . 195: Key: pair.Key,
. . 196: Value: pair.Val,
. . 197: }
. . 198: }
* Set GC percent as 500 default
Lightning allocates too many transient objects and heap size is small,
so garbage collections happen too frequently and lots of time is spent in GC component.
In a test of loading the table `order_line.csv` of 14k TPCC.
The time need of `encode kv data and write` step reduce from 52m4s to 37m30s when change
GOGC from 100 to 500, the total time needed reduce near 15m too.
The cost of this is the memory of lightnin at runtime grow from about 200M to 700M, but it's acceptable.
So we set the gc percentage as 500 default to reduce the GC frequency instead of 100.
* Remove MaxKVPairs in Mydump
has been move to Importer part
* Remove outdate code
* Update tidb version
For pingcap@495f8b7
disable UpdateDeltaForTable if TxnCtx is nil
* Address comment
* Remain append
Co-authored-by: xuhuaiyu <391585975@qq.com>
xhebox
pushed a commit
to xhebox/tidb
that referenced
this pull request
Sep 28, 2021
xhebox
pushed a commit
to xhebox/tidb
that referenced
this pull request
Oct 8, 2021
ti-chi-bot
pushed a commit
that referenced
this pull request
Oct 9, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prepare for grant statement.