restore: optimize SQL processing speed #110

lonng · 2018-12-25T02:57:58Z

DNM: Based on #109 for simplicity of development.

What problem does this PR solve?

Optimizing SQL processing speed

Result:

PR 109: TableConcurrency = 20, RegionConcurrency = 40, the metrics data has been lost due to cluster be cleanup.
```
Data size: 340G
Rate: ~45MB/s
Total time: ~= 2h30m (import time: 40m)
```

PR 110: TableConcurrency = 20, RegionConcurrency = 20, IOConcurrency = 5, Test1 metrics snapshot, Test2 metrics snapshot

CAN NOT REPREDUCE [IO Delay unstable]

Test1: 
Data size: 146G
Rate: 90~160MB/s
Total time: ~= 48m (import time: 27m)

Test2:
Data size: 146G
Rate: 130~190MB/s
Total time: ~= 46m (import time: 28m)

Test3
coming ...

PR 110: TableConcurrency = 40, RegionConcurrency = 40, IOConcurrency = 10 160G, Metrics

2018/12/30 01:16:32.871 restore.go:477: [info] restore all tables data takes 52m59.59960385s
2018/12/30 01:16:32.871 restore.go:366: [info] Everything imported, stopping periodic actions
2018/12/30 01:16:32.871 restore.go:208: [error] run cause error : [types:1292]invalid time format: '{2038 1 19 4 4 36 0}'
2018/12/30 01:16:32.871 restore.go:214: [info] the whole procedure takes 53m7.986292573s

Early conclusion:

Concurrent IO causes delays to lengthen, which lengthens SQL processing time

What is changed and how it works?

Limiting IO concurrency

Check List

Tests

Unit test

Code changes

Side effects

Related changes

sre-bot · 2018-12-25T02:58:01Z

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

lonng · 2018-12-25T03:10:03Z

/run-integration-test

lonng · 2018-12-25T10:30:26Z

/run-integration-test

lonng · 2018-12-26T05:41:27Z

/run-integration-test

lonng · 2018-12-26T07:12:10Z

/run-integration-test

IANTHEREAL · 2018-12-26T09:35:46Z

add test result into description, like io conc = 1, default, region conc...

… block buffer

lonng · 2018-12-30T04:00:02Z

/run-integration-test

lonng · 2018-12-30T06:48:36Z

@GregoryIan @kennytm @july2993 PTAL

kennytm · 2018-12-30T07:18:25Z

/run-unit-test

go.mod

lightning/metric/metric.go

lightning/mydump/parser.go

tidb-lightning.toml

lonng · 2018-12-30T08:04:31Z

/run-all-tests

kennytm

Rest LGTM

tidb-lightning.toml

lightning/config/config.go

lonng · 2018-12-30T08:28:55Z

/run-all-tests

tidb-lightning.toml

lightning/worker/worker.go

IANTHEREAL · 2019-01-02T06:24:43Z

Rest LGTM

lonng · 2019-01-02T08:02:21Z

/run-all-tests

IANTHEREAL · 2019-01-02T12:23:11Z

some problems with ci

IANTHEREAL · 2019-01-02T12:25:53Z

well done! LGTM

kennytm · 2019-01-02T13:28:34Z

/run-integration-test

lonng added the status/DNM Do not merge, test is failing or blocked by another PR label Dec 25, 2018

lonng force-pushed the lonng/opt-sql-proc branch 2 times, most recently from 9249430 to 0bb02d6 Compare December 25, 2018 07:03

lonng force-pushed the lonng/opt-sql-proc branch from 459ceaf to 07232e3 Compare December 26, 2018 05:41

lonng changed the base branch from master to kennytm/1-chunk-1-file December 26, 2018 06:34

lonng changed the base branch from kennytm/1-chunk-1-file to master December 26, 2018 06:35

lonng added the Should Update Docs Should update docs after this PR is merged. Remove this label once the docs are updated label Dec 26, 2018

lonng force-pushed the lonng/opt-sql-proc branch from 2a3e7f9 to e96ca9c Compare December 26, 2018 07:12

lonng force-pushed the lonng/opt-sql-proc branch 2 times, most recently from c7f283b to 7315ef5 Compare December 26, 2018 08:58

lonng added 3 commits December 27, 2018 08:05

restore: add metrics

1028ef1

mydump,restore: mydumper.read-block-size to init ChunkParser read…

f1535dc

… block buffer

*: limit IO concurrency

1880258

lonng force-pushed the lonng/opt-sql-proc branch 2 times, most recently from ccfe5ef to 6baedce Compare December 27, 2018 00:34

*: add apply worker metrics

eb011cd

lonng force-pushed the lonng/opt-sql-proc branch 3 times, most recently from 5fc836f to 79efc70 Compare December 30, 2018 03:59

lonng added status/PTAL This PR is ready for review. Add this label back after committing new changes priority/normal type/feature New feature and removed status/DNM Do not merge, test is failing or blocked by another PR labels Dec 30, 2018

kennytm reviewed Dec 30, 2018

View reviewed changes

lonng force-pushed the lonng/opt-sql-proc branch 2 times, most recently from c15fa92 to 9139e2e Compare December 30, 2018 07:55

kennytm reviewed Dec 30, 2018

View reviewed changes

tidb-lightning.toml Outdated Show resolved Hide resolved

lightning/config/config.go Outdated Show resolved Hide resolved

lonng force-pushed the lonng/opt-sql-proc branch from 9139e2e to fef33c2 Compare December 30, 2018 08:28

kennytm reviewed Dec 30, 2018

View reviewed changes

tidb-lightning.toml Outdated Show resolved Hide resolved

kennytm added status/LGT1 One reviewer already commented LGTM (LGTM1) and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Dec 30, 2018

mydump: set read block buffer size to BlockSize * config.BufferSizeScale

54688f3

lonng force-pushed the lonng/opt-sql-proc branch from fef33c2 to 54688f3 Compare December 30, 2018 08:54

IANTHEREAL reviewed Jan 2, 2019

View reviewed changes

tidb-lightning.toml Show resolved Hide resolved

IANTHEREAL reviewed Jan 2, 2019

View reviewed changes

lightning/worker/worker.go Show resolved Hide resolved

worker: add unit test

4f1dced

IANTHEREAL added status/LGT2 Two reviewers already commented LGTM, ready for merge (LGTM2) and removed status/LGT1 One reviewer already commented LGTM (LGTM1) labels Jan 2, 2019

kennytm merged commit fdf4d5b into master Jan 2, 2019

kennytm deleted the lonng/opt-sql-proc branch January 2, 2019 13:52

kennytm added Should Update Ansible The config in TiDB-Ansible should be updated and removed Should Update Docs Should update docs after this PR is merged. Remove this label once the docs are updated labels Jan 8, 2019

kennytm removed the Should Update Ansible The config in TiDB-Ansible should be updated label May 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restore: optimize SQL processing speed #110

restore: optimize SQL processing speed #110

lonng commented Dec 25, 2018 •

edited

Loading

sre-bot commented Dec 25, 2018

lonng commented Dec 25, 2018

lonng commented Dec 25, 2018

lonng commented Dec 26, 2018

lonng commented Dec 26, 2018

IANTHEREAL commented Dec 26, 2018 •

edited

Loading

lonng commented Dec 30, 2018

lonng commented Dec 30, 2018

kennytm commented Dec 30, 2018

lonng commented Dec 30, 2018

kennytm left a comment

lonng commented Dec 30, 2018

IANTHEREAL commented Jan 2, 2019

lonng commented Jan 2, 2019

IANTHEREAL commented Jan 2, 2019

IANTHEREAL commented Jan 2, 2019

kennytm commented Jan 2, 2019

restore: optimize SQL processing speed #110

restore: optimize SQL processing speed #110

Conversation

lonng commented Dec 25, 2018 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

sre-bot commented Dec 25, 2018

lonng commented Dec 25, 2018

lonng commented Dec 25, 2018

lonng commented Dec 26, 2018

lonng commented Dec 26, 2018

IANTHEREAL commented Dec 26, 2018 • edited Loading

lonng commented Dec 30, 2018

lonng commented Dec 30, 2018

kennytm commented Dec 30, 2018

lonng commented Dec 30, 2018

kennytm left a comment

Choose a reason for hiding this comment

lonng commented Dec 30, 2018

IANTHEREAL commented Jan 2, 2019

lonng commented Jan 2, 2019

IANTHEREAL commented Jan 2, 2019

IANTHEREAL commented Jan 2, 2019

kennytm commented Jan 2, 2019

lonng commented Dec 25, 2018 •

edited

Loading

IANTHEREAL commented Dec 26, 2018 •

edited

Loading