This repository has been archived by the owner on Dec 8, 2021. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 66
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
optimize the performance of lightning (#281)
* lightning: split large csv file if possible * gofmt * gofmt * unit test * add unit test * tiny change * tiny refine * fix ci * remove useless code * fix ci * fix ci * address comments * go fmt for all * Replace CSV Ragel parser by a hand-written parser copied from encoding/csv Conflicts: lightning/mydump/csv_parser_generated.go * fix conflict * update * update again * send a batch of kv in encodeLoop * use sync.Pool * Close channel instead of push one entry. * Use copy instead append * Fix test and failpoint version * Reuse slice of record This expected to avoid about 3.5% of alloc_objects alloc_objects: Total: 773496750 773873722 (flat, cum) 7.18% 177 . . parser.fieldIndexes = parser.fieldIndexes[:0] 178 . . 179 . . isEmptyLine := true ... 225 386621314 386621314 str := string(parser.recordBuffer) // Convert to string once to batch allocations 226 386875436 386875436 dst := make([]string, len(parser.fieldIndexes)) * Use pool for mutation This take most alloc in WriteRows: ROUTINE ======================== github.com/pingcap/tidb-lightning/lightning/backend.(*importer).WriteRows in /Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/backend/importer.go 797370418 980241246 (flat, cum) 9.09% of Total . . 155: kvs := rows.(kvPairs) ... ... . . 192: for i, pair := range kvs { 772641868 772641868 193: mutations[i] = &kv.Mutation{ . . 194: Op: kv.Mutation_Put, . . 195: Key: pair.Key, . . 196: Value: pair.Val, . . 197: } . . 198: } * Set GC percent as 500 default Lightning allocates too many transient objects and heap size is small, so garbage collections happen too frequently and lots of time is spent in GC component. In a test of loading the table `order_line.csv` of 14k TPCC. The time need of `encode kv data and write` step reduce from 52m4s to 37m30s when change GOGC from 100 to 500, the total time needed reduce near 15m too. The cost of this is the memory of lightnin at runtime grow from about 200M to 700M, but it's acceptable. So we set the gc percentage as 500 default to reduce the GC frequency instead of 100. * Remove MaxKVPairs in Mydump has been move to Importer part * Remove outdate code * Update tidb version For pingcap/tidb@495f8b7 disable UpdateDeltaForTable if TxnCtx is nil * Address comment * Remain append Co-authored-by: xuhuaiyu <391585975@qq.com>
- Loading branch information
Showing
11 changed files
with
154 additions
and
81 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters