executor: load data statement, separate data preparing routine and commit routine #11533

cfzjywxk · 2019-07-30T14:09:35Z

What problem does this PR solve?

experiments diff on
#11517

What is changed and how it works?

load data statement

separate data preparing routine and commit routine(maybe could concurrent commit within multiple txns in the future)
tests on different workloads

bench workload1

sysbench schema
1m rows = 185MB
10m rows = 1.9GB

batch size = 20k

data	mode	rt(sec)	batch size(row)	prepare(us)	commit(us)	avg prepare(us)	avg commit(us)
1m	concurrent	27.56	20k	5578125	27436263	111562	548725
1m	concurrent	28.93	20k	5646682	28809710	112933	576194
1m	concurrent	28.59	20k	6077118	28429610	121542	568592
1m	sequential	33.56	20k	5295292	28251475	105905	565029
1m	sequential	31.64	20k	`5221046`	26403697	104420	528073
1m	sequential	32.74	20k	5463142	27259847	109262	545196

batch size = 200k

data	mode	rt(sec)	batch size(row)	prepare(us)	commit(us)	avg prepare(us)	avg commit(us)
1m	concurrent	29.41	200k	5740404	28639015	1148080	5727803
1m	sequential	33.87	200k	5580477	28238641	1116095	5647728

batch size = 4k

data	mode	rt(sec)	batch size(row)	prepare(us)	commit(us)	avg prepare(us)	avg commit(us)
1m	concurrent	35.79	4k	5653526	35739674	22614	142958
1m	sequential	37.52	4k	`5349413`	32139162	21397	128556

data size = 10m

data	mode	rt(sec)	batch size(row)	prepare(us)	commit(us)	avg prepare(us)	avg commit(us)
10m	concurrent	299	20k	62993307	299338296	125986	598676
10m	concurrent	308	20k	60778661	308309945	121557	616619
10m	concurrent	310	20k	60284459	309862211	120568	619724
10m	sequential	371	20k	64699475	306366299	129398	612732
10m	sequential	348	20k	62781449	285804457	125562	571608
10m	sequential	346	20k	61517956	285356229	123035	570712

bench workload2

this workload task generator routine and commit routine cooperate well, producing and committing seem concurrently progressing. Master branch do all these things sequentially

CREATE TABLE `model` (
  `pk` int(11) DEFAULT NULL,
  `col1` int(11) DEFAULT NULL,
  `col2` int(11) DEFAULT NULL,
  ...
  `col99` int(11) DEFAULT NULL,
  KEY `pk` (`pk`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin

column content = 99999999

100columns 1m rows = 852MB
500columns 1m rows = 4.2GB

data	mode	rt(sec)	batch size(row)	prepare(us)	commit(us)	avg prepare(us)	avg commit(us)
1m	concurrent	80.44	20k	55995202	79229461	1119904	1584589
1m	concurrent	78.07	20k	55592061	76927103	1111841	1538542
1m	concurrent	83.41	40k	55236614	81160118	`2209464`	3246404
1m	sequential	121.60	20k	50197243	71241453	1003944	`1424829`

500columns

data	mode	rt(sec)	batch size(row)	prepare(us)	commit(us)	avg prepare(us)	avg commit(us)
1m	concurrent	457.73	20k	268403660	452392863	5368073	`9047857`
1m	concurrent	392.51	20k	264366292	387069215	5287325	7741384
1m	sequential	589.52	20k	236941284	351913873	`4738825`	7038277
1m	sequential	585.56	20k	237462383	347632453	4749247	6952649

bench workload3

wide table with 100 columns (all columns with BLOB or TEXT fields) , big column

work queue size 1000(max wait job numbers)
( txn limit 100MB)

schema like

CREATE TABLE `tin_n_10wb` (
  `pk` int(11) DEFAULT NULL,
  `col1` blob DEFAULT NULL,
  `col2` text DEFAULT NULL,
  ...
 `col95` blob DEFAULT NULL,
 `col96` text DEFAULT NULL,
  ...

column content like
len("G2jOVOMOg3Cnub2jCuXtn0AO1V5hONfLftt0cBHX74itPKMm2uVj1Ax1GfykLLR6PxGclMhPRQxoCe8ZvYI096goa6CsxKHiBvpl7SOp0ax5vl1QNYUZq2x2") = 120

100k rows = 1.2GB

data	mode	rt(sec)	batch size(row)	prepare(us)	commit(us)	avg prepare(us)	avg commit(us)
100k	concurrent	24.70	5k(raftstore.raft-max-inflight-msgs)	22384533	23709265	1119226	`1185463`
100k	sequential	43.15	5k(reconfiguring raftstore.raft-max-inflight-msgs)	20503449	22651435	1025172	1132571
100k	concurrent	21.87	1k	20900265	21574010	209002	215740
100k	sequential	51.21	1k	19181370	31973015	191813	319730
100k	sequential	108.33	1k	19770424	88490433	197704	884904
100k	concurrent	23.52	200	20573994	23393318	41147	46786
100k	sequential	40.94	200	19083036	21767787	38166	43535

some conclusions up to now

greater avg prepare and commit time cost in concurrent mode compared with sequential processing.
costs, seperate processing will consume more memory resources during processing. Sequential processing some row buffer could be reused.
batch size will impact performance, default 20k batch size seems good on most situations
workload3 batch size = 5k and batch size = 1k have much different performance, batch size = 5k
many error logs in like

~~from monitor data on grafana, we see prewrite on kv is quit slow, the root cause should be the prewrite request is quite slow(32sec on 5k batch size)~~

tikv/tikv#5279
has some explanations on prewrite slow on tikv executing workload3(big rowsize)
this could be solved by increase tikv raftstore.raft-max-inflight-msgs configuration(default -> 4096), allow more un ack raft messages in queue, after changing this config,

Check List

Tests

Unit test

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change

Side effects

Increased code complexity

Related changes

cfzjywxk · 2019-07-30T14:47:57Z

/run-all-tests

executor/load_data.go

ngaut · 2019-07-31T05:34:17Z

Could you do a test for many columns(such as 100 columns) ?

cfzjywxk · 2019-07-31T05:56:31Z

Could you do a test for many columns(such as 100 columns) ?

en. will construct such testing scenarios， and profiling defailed cpu/memory usage

cfzjywxk · 2019-08-01T05:02:58Z

@jackysp @ngaut PTAL

cfzjywxk · 2019-08-01T10:24:23Z

/run-all-tests

executor/load_data.go

jackysp · 2019-08-01T11:07:36Z

executor/load_data.go

+// EnqOneTask feed one batch commit task to commit routine
+func (e *LoadDataInfo) EnqOneTask() {
+	if e.curBatchCnt > 0 {
+		logutil.BgLogger().Info("tried to enqueue one task current len", zap.Uint64("len", e.curBatchCnt))


We could make the committed batch size configurable. It is better to set it to debug level.

executor/load_data.go

executor/write_test.go

executor/load_data.go

tiancaiamao · 2019-08-02T07:36:05Z

executor/load_data.go

+// InitQueues initialize task queue and error report queue
+func (e *LoadDataInfo) InitQueues() {
+	e.commitTaskQueue = make(chan commitTask, 1000)
+	e.commitErrQueue = make(chan error, 5)


make(chan error)

executor/load_data.go

tiancaiamao · 2019-08-02T07:53:10Z

server/conn.go

+		}
+	}()
+	// start commit worker routine
+	go loadDataInfo.CommitRoutine(ctx)


Why not make the load process into a separate goroutine and keep the commit process in this goroutine?

changed PTAL

server/conn.go

zz-jason

please fix ci.

codecov · 2019-08-05T14:02:54Z

Codecov Report

Merging #11533 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #11533   +/-   ##
===========================================
  Coverage   81.2905%   81.2905%           
===========================================
  Files           454        454           
  Lines         98720      98720           
===========================================
  Hits          80250      80250           
  Misses        12725      12725           
  Partials       5745       5745

cfzjywxk · 2019-08-05T14:08:48Z

/run-all-tests

jackysp · 2019-08-06T03:26:03Z

sessionctx/variable/tidb_vars.go

@@ -286,6 +286,9 @@ const (

 	// TiDBEnableNoopFuncs set true will enable using fake funcs(like get_lock release_lock)
 	TiDBEnableNoopFuncs = "tidb_enable_noop_functions"
+
+	// TiDBLoadDataSeqProcess set true will make load data process and commit sequentially in one routine
+	TiDBLoadDataSeqProcess = "tidb_load_data_seq_process"


Please add some tests for it.

ngaut · 2019-08-21T05:54:04Z

executor/load_data.go

+	defer func() {
+		r := recover()
+		if r != nil {
+			buf := make([]byte, 4096)


Zap log library has a Stack function to get stack tracing.

roger, changed

ngaut · 2019-08-21T05:54:58Z

executor/load_data.go

+		logutil.Logger(ctx).Error("commit error refresh", zap.Error(err))
+		return err
+	}
+	// this only set in sequential mode, e.rows buffer will be reused in sequential mode


What is sequential mode?

ngaut · 2019-08-21T05:55:22Z

executor/load_data.go

+				return err
+			}
+		}
+		logutil.Logger(ctx).Debug("one task enqueued ", zap.Int("current queue len", len(e.commitTaskQueue)))


Remove debug message.

ngaut · 2019-08-21T05:56:12Z

executor/load_data.go

+					break
+				}
+				tasks++
+				logutil.Logger(ctx).Debug("commit one task finished",


ngaut · 2019-08-21T05:56:20Z

executor/load_data.go

+					zap.Int("pending tasks count", len(e.commitTaskQueue)))
+			} else {
+				end = true
+				logutil.Logger(ctx).Info("commit work all finished",


ngaut · 2019-08-21T05:57:39Z

server/conn.go

-		}
-		if err = loadDataInfo.Ctx.StmtCommit(); err != nil {
-			return nil, err
+		if enqTask {


We do not need to use enqTask argument.

param removed

ngaut · 2019-08-21T05:58:07Z

server/conn.go

+	defer func() {
+		r := recover()
+		if r != nil {
+			buf := make([]byte, 4096)


Use zap library instead.

ngaut · 2019-08-21T05:59:07Z

Please do not force push your commit. Just commit is fine.

server/conn.go

executor/load_data.go

zz-jason · 2019-08-28T07:05:55Z

executor/load_data.go

+
+	// these fields are used for pipeline data prepare and commit
+	commitTaskQueue chan CommitTask
+	QuitCommit      chan struct{}


I think we can do it by only using one signal channel.

executor/load_data.go

server/conn.go

cfzjywxk · 2019-09-05T08:30:39Z

ping @lysu @jackysp PTAL thx

cfzjywxk · 2019-09-05T09:20:40Z

/run-unit-test

sre-bot · 2019-09-16T11:28:14Z

/run-all-tests

sre-bot · 2019-09-16T11:32:16Z

@cfzjywxk merge failed.

jackysp · 2019-09-16T11:35:17Z

/rebuild

cfzjywxk requested review from lysu, ngaut, tiancaiamao and jackysp July 30, 2019 14:09

cfzjywxk added sig/execution SIG execution type/performance labels Jul 30, 2019

crazycs520 reviewed Jul 31, 2019

View reviewed changes

executor/load_data.go Outdated Show resolved Hide resolved

ngaut removed their request for review August 1, 2019 05:38

cfzjywxk requested a review from zz-jason August 1, 2019 08:15

jackysp reviewed Aug 1, 2019

View reviewed changes

executor/load_data.go Outdated Show resolved Hide resolved

jackysp reviewed Aug 1, 2019

View reviewed changes

executor/load_data.go Outdated Show resolved Hide resolved

jackysp reviewed Aug 1, 2019

View reviewed changes

executor/write_test.go Show resolved Hide resolved

jackysp reviewed Aug 1, 2019

View reviewed changes

executor/write_test.go Show resolved Hide resolved

tiancaiamao reviewed Aug 2, 2019

View reviewed changes

zz-jason reviewed Aug 2, 2019

View reviewed changes

zz-jason added the status/ReqChange label Aug 5, 2019

cfzjywxk force-pushed the ppload branch from 7e2adac to 3898f7e Compare August 5, 2019 13:57

cfzjywxk requested a review from jackysp August 5, 2019 14:06

jackysp added the status/all tests passed label Aug 6, 2019

cfzjywxk requested review from tiancaiamao and zz-jason August 6, 2019 02:34

jackysp reviewed Aug 6, 2019

View reviewed changes

ngaut reviewed Aug 21, 2019

View reviewed changes

executor/load_data.go Outdated

break

}

tasks++

logutil.Logger(ctx).Debug("commit one task finished",

Copy link

Member

ngaut Aug 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

ngaut reviewed Aug 21, 2019

View reviewed changes

server/conn.go Outdated

defer func() {

r := recover()

if r != nil {

buf := make([]byte, 4096)

Copy link

Member

ngaut Aug 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use zap library instead.

cfzjywxk added 2 commits August 21, 2019 19:59

remove debug log, seq proc related params

a631f75

remove log

700d86d

cfzjywxk requested review from ngaut, zz-jason and jackysp August 22, 2019 07:19

zz-jason reviewed Aug 28, 2019

View reviewed changes

cfzjywxk added 4 commits August 28, 2019 15:33

Merge branch 'master' into ppload

e07a050

change to one force quit channel, add panic error to log

1057e4f

Merge branch 'master' into ppload

aa3a04b

Merge branch 'master' into ppload

d954d76

cfzjywxk added 2 commits September 5, 2019 17:20

Merge branch 'master' into ppload

596d071

Merge branch 'master' into ppload

768bd60

jackysp approved these changes Sep 16, 2019

View reviewed changes

jackysp added the status/can-merge Indicates a PR has been approved by a committer. label Sep 16, 2019

jackysp merged commit d438e10 into pingcap:master Sep 16, 2019

cfzjywxk deleted the ppload branch September 16, 2019 11:46

youjiali1995 mentioned this pull request Jan 21, 2021

LOAD DATA statement doesn't respect tidb_dml_batch_size variable #10524

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: load data statement, separate data preparing routine and commit routine #11533

executor: load data statement, separate data preparing routine and commit routine #11533

cfzjywxk commented Jul 30, 2019 •

edited

Loading

cfzjywxk commented Jul 30, 2019

ngaut commented Jul 31, 2019

cfzjywxk commented Jul 31, 2019

cfzjywxk commented Aug 1, 2019

cfzjywxk commented Aug 1, 2019

jackysp Aug 1, 2019

tiancaiamao Aug 2, 2019

tiancaiamao Aug 2, 2019

cfzjywxk Aug 5, 2019

zz-jason left a comment

codecov bot commented Aug 5, 2019 •

edited

Loading

cfzjywxk commented Aug 5, 2019

jackysp Aug 6, 2019

ngaut Aug 21, 2019

cfzjywxk Aug 21, 2019

ngaut Aug 21, 2019

ngaut Aug 21, 2019

ngaut Aug 21, 2019

ngaut Aug 21, 2019

ngaut Aug 21, 2019

cfzjywxk Aug 21, 2019

ngaut Aug 21, 2019

ngaut commented Aug 21, 2019

zz-jason Aug 28, 2019

cfzjywxk commented Sep 5, 2019

cfzjywxk commented Sep 5, 2019

sre-bot commented Sep 16, 2019

sre-bot commented Sep 16, 2019

jackysp commented Sep 16, 2019

executor: load data statement, separate data preparing routine and commit routine #11533

executor: load data statement, separate data preparing routine and commit routine #11533

Conversation

cfzjywxk commented Jul 30, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

bench workload1

bench workload2

bench workload3

Check List

cfzjywxk commented Jul 30, 2019

ngaut commented Jul 31, 2019

cfzjywxk commented Jul 31, 2019

cfzjywxk commented Aug 1, 2019

cfzjywxk commented Aug 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zz-jason left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 5, 2019 • edited Loading

Codecov Report

cfzjywxk commented Aug 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngaut commented Aug 21, 2019

Choose a reason for hiding this comment

cfzjywxk commented Sep 5, 2019

cfzjywxk commented Sep 5, 2019

sre-bot commented Sep 16, 2019

sre-bot commented Sep 16, 2019

jackysp commented Sep 16, 2019

cfzjywxk commented Jul 30, 2019 •

edited

Loading

codecov bot commented Aug 5, 2019 •

edited

Loading