pkg/loader: add pkg to load data to mysql #436

july2993 · 2019-01-07T05:44:37Z

What problem does this PR solve?

a pkg to use to load data to mysql in realtime, can be use in test and reparo, drainer later

#438 contains only update vendor part, you can review and merge that first

What is changed and how it works?

see tests/kafka/kafka.go as a example,
most logic is same as drainer, but if the table has PK, it will merge the same PK record(see merge.go), after merge, only have one record for one PK, so we can easily do bulk insert. this is also beneficial if the upstream update some hot key frequently.

more thinks maybe we can do:

currently like in drainer, for update event we get all field and set all field in the update sql, but in most time only one or a few field is updated really, we can only set that fields to reduce data send to downstream.

Check List

Tests

Integration test

Code changes

Side effects

Related changes

july2993 · 2019-01-07T05:59:40Z

/run-all-tests

IANTHEREAL · 2019-01-07T09:45:48Z

currently like in drainer, for update event we get all field and set all field in the update sql, but in most time only one or a few field is updated really, we can only set that fields to reduce data send to downstream.

we did it in syncer ago, but some column had trigger attributes, if we can solve it, I like this way

IANTHEREAL · 2019-01-07T10:00:26Z

update bench test result？

july2993 · 2019-01-07T13:18:02Z

a bench test of bench_test.go running locally with downstream mysql 5.7

worker-count&batch-size: 16 128
BenchmarkInsertKey 1000000 26680 ns/op 37481op/second
BenchmarkInsertNoKey 300000 72654 ns/op 13763 op/second

change worker-count&batch-size: 32 256
BenchmarkInsertKey 3000000 11767 ns/op 84983 op/second
BenchmarkInsertNoKey 300000 71042 ns/op 14076 op/second

july2993 · 2019-01-08T09:09:41Z

@GregoryIan PTAL

use same table schema to bench merge or not

pkg/loader/executor.go

pkg/loader/merge.go

pkg/loader/load.go

pkg/loader/model.go

WangXiangUSTC · 2019-01-10T07:08:12Z

pkg/loader/util.go

+	return
+}
+
+func quoteSchema(schema string, table string) string {


we can use the https://github.com/pingcap/tidb-tools/blob/15b677aee094de9c7244f2c691ac9e427b92fe93/pkg/dbutil/common.go#L592

some problem about tidb version dependency, keep this simple func

pkg/loader/load.go

pkg/loader/model.go

IANTHEREAL

two important things about it:

export input channel and successful channel to control load
how does user know TXNs were all finished?
compare one by one,
or sort them and get one successful TXN, then we can know the TXNs before it were already finished.

It's better to have a README.md and example

lack of metrics and log

pkg/loader/load.go

pkg/loader/util.go

pkg/loader/load.go

pkg/loader/model.go

pkg/loader/load.go

IANTHEREAL · 2019-01-11T07:43:40Z

pkg/loader/load.go

+	var singleDMLs []*DML
+
+	for tableName, tableDMLs := range tables {
+		if len(tableDMLs[0].primaryKeys()) > 0 && s.merge {


as we said, we should consider unique keys relations on multiple rows, I think we should check primary key count = 1 and unique key = 0

see

tidb-binlog/pkg/loader/executor.go

Line 238 in 03d05d1

// we merge dmls by primary key, after merge by key, we

but i can add the limit or add an option to open this feature now, maybe use it after more test and prove, or when meet performance problem

update: 5f5873b

IANTHEREAL · 2019-01-12T04:57:07Z

Ping @july2993, we can speed this pr

july2993 · 2019-01-14T06:04:55Z

add example and metrics @GregoryIan @WangXiangUSTC PTAL

Co-Authored-By: july2993 <july2993@gmail.com>

july2993 · 2019-01-16T08:44:42Z

@GregoryIan @kennytm @WangXiangUSTC PTAL

kennytm

Rest LGTM

kennytm · 2019-01-17T08:51:03Z

pkg/loader/executor.go

+	tableName := inserts[0].Table
+
+	builder := new(strings.Builder)
+	builder.WriteString(fmt.Sprintf("REPLACE INTO %s(%s) VALUES ", quoteSchema(dbName, tableName), buildColumnList(info.columns)))


@july2993 This isn't resolved 🙃

july2993 · 2019-01-17T09:10:26Z

@WangXiangUSTC @kennytm PTAL

WangXiangUSTC · 2019-01-17T10:00:53Z

LGTM

* pkg/loader: add pkg to load data to mysql (#436) * tests/* add swap unique index value test (#437) * fix json type and may lost data when restart (#463) * translate.go use string type instead of []byte for json field

july2993 added the status/PTAL label Jan 7, 2019

pkg/loader: add pkg to load data to mysql

32a119f

july2993 force-pushed the hjh/loader branch from 85aec34 to 32a119f Compare January 7, 2019 05:45

july2993 added the type/feature label Jan 7, 2019

vendor/* update vendor add errgroup

bc7d1a6

july2993 force-pushed the hjh/loader branch from a7b632e to bc7d1a6 Compare January 7, 2019 06:26

july2993 added the priority/important label Jan 7, 2019

Merge branch 'master' into hjh/loader

c12019c

july2993 added 2 commits January 8, 2019 13:52

add a update pk&uk case and add comment about mergeByKey

9c9fd60

Merge remote-tracking branch 'origin/hjh/loader' into hjh/loader

296f878

july2993 added 2 commits January 9, 2019 20:12

executor.go: Fix forgot to return err and quote column name

45d68fd

bench_test.go set merge or not and add delete&update bench

f85ec8b

use same table schema to bench merge or not

WangXiangUSTC reviewed Jan 10, 2019

View reviewed changes

july2993 added 2 commits January 10, 2019 16:28

loader/* don't always chagne insert -> replace refine some code

d4f5cf7

Merge branch 'master' into hjh/loader

07cb6c9

july2993 mentioned this pull request Jan 10, 2019

tool sync from kafka to mysql #441

Merged

IANTHEREAL reviewed Jan 11, 2019

View reviewed changes

pkg/loader/load.go Outdated Show resolved Hide resolved

WangXiangUSTC reviewed Jan 11, 2019

View reviewed changes

pkg/loader/load.go Show resolved Hide resolved

pkg/loader/load.go Show resolved Hide resolved

pkg/loader/model.go Outdated Show resolved Hide resolved

IANTHEREAL reviewed Jan 11, 2019

View reviewed changes

july2993 added 3 commits January 14, 2019 10:35

add example and refine code

b229f43

add metrics of loader, change NewLoader api

b9eb9ce

Merge remote-tracking branch 'origin/hjh/loader' into hjh/loader

03d05d1

load.go: merge pk and batch if and only if have pk and no uk

5f5873b

kennytm and others added 10 commits January 16, 2019 11:14

Update pkg/loader/example_loader_test.go

9c0bb48

Co-Authored-By: july2993 <july2993@gmail.com>

Update pkg/loader/README.md

83ba23c

Co-Authored-By: july2993 <july2993@gmail.com>

Update pkg/loader/README.md

e11c7ac

Co-Authored-By: july2993 <july2993@gmail.com>

Update pkg/loader/README.md

38850e1

Co-Authored-By: july2993 <july2993@gmail.com>

Update pkg/loader/merge_test.go

84c637a

Co-Authored-By: july2993 <july2993@gmail.com>

Update pkg/loader/load.go

81fb7b1

Co-Authored-By: july2993 <july2993@gmail.com>

Update pkg/loader/README.md

b36ff7c

Co-Authored-By: july2993 <july2993@gmail.com>

Update pkg/loader/README.md

54aa4d8

Co-Authored-By: july2993 <july2993@gmail.com>

loader: address comments and use strings.Builder

c23895f

merge.go factor some common case out

22076e0

kennytm self-assigned this Jan 16, 2019

july2993 assigned WangXiangUSTC Jan 17, 2019

kennytm reviewed Jan 17, 2019

View reviewed changes

kennytm removed their assignment Jan 17, 2019

executor.go: Use fmt.Fprintf write data to builder

ff844d0

WangXiangUSTC added status/LGT1 and removed status/PTAL labels Jan 17, 2019

kennytm approved these changes Jan 17, 2019

View reviewed changes

july2993 added status/LGT2 and removed status/LGT1 labels Jan 17, 2019

july2993 merged commit b614c9c into master Jan 17, 2019

july2993 added a commit that referenced this pull request Feb 26, 2019

pkg/loader: add pkg to load data to mysql (#436)

11ef08e

july2993 mentioned this pull request Feb 26, 2019

cherry pick pkg/loader #471

Merged

july2993 added a commit that referenced this pull request Feb 26, 2019

pkg/loader: add pkg to load data to mysql (#436)

96c645c

july2993 added a commit that referenced this pull request Feb 26, 2019

pkg/loader: add pkg to load data to mysql (#436)

8c041a3

july2993 deleted the hjh/loader branch May 4, 2019 13:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/loader: add pkg to load data to mysql #436

pkg/loader: add pkg to load data to mysql #436

july2993 commented Jan 7, 2019 •

edited

Loading

july2993 commented Jan 7, 2019

IANTHEREAL commented Jan 7, 2019

IANTHEREAL commented Jan 7, 2019

july2993 commented Jan 7, 2019

july2993 commented Jan 8, 2019

WangXiangUSTC Jan 10, 2019

july2993 Jan 10, 2019

IANTHEREAL left a comment •

edited

Loading

IANTHEREAL Jan 11, 2019

july2993 Jan 14, 2019

july2993 Jan 14, 2019

IANTHEREAL commented Jan 12, 2019

july2993 commented Jan 14, 2019

july2993 commented Jan 16, 2019

kennytm left a comment

kennytm Jan 17, 2019

july2993 commented Jan 17, 2019

WangXiangUSTC commented Jan 17, 2019

pkg/loader: add pkg to load data to mysql #436

pkg/loader: add pkg to load data to mysql #436

Conversation

july2993 commented Jan 7, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

july2993 commented Jan 7, 2019

IANTHEREAL commented Jan 7, 2019

IANTHEREAL commented Jan 7, 2019

july2993 commented Jan 7, 2019

july2993 commented Jan 8, 2019

WangXiangUSTC Jan 10, 2019

Choose a reason for hiding this comment

july2993 Jan 10, 2019

Choose a reason for hiding this comment

IANTHEREAL left a comment • edited Loading

Choose a reason for hiding this comment

IANTHEREAL Jan 11, 2019

Choose a reason for hiding this comment

july2993 Jan 14, 2019

Choose a reason for hiding this comment

july2993 Jan 14, 2019

Choose a reason for hiding this comment

IANTHEREAL commented Jan 12, 2019

july2993 commented Jan 14, 2019

july2993 commented Jan 16, 2019

kennytm left a comment

Choose a reason for hiding this comment

kennytm Jan 17, 2019

Choose a reason for hiding this comment

july2993 commented Jan 17, 2019

WangXiangUSTC commented Jan 17, 2019

july2993 commented Jan 7, 2019 •

edited

Loading

IANTHEREAL left a comment •

edited

Loading