Detect duplicate data from TiKV #1144

Little-Wallace · 2021-05-25T09:55:19Z

What problem does this PR solve?

part of #1110

When there are multiple lightning instance importing data to TiKV together, the data of them may be duplicate.

What is changed and how it works?

We use different commit timestamps to distinguish the data imported by different lightning ( see details in #1101) . After import jobs all end, we will scan all data of TiKV to collect duplicate data.

Check List

Tests

Unit test
Integration test

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Release note

No release note.

ti-chi-bot · 2021-05-25T09:55:21Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

gozssky
kennytm

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

CLAassistant · 2021-05-25T09:55:25Z

All committers have signed the CLA.

pkg/lightning/backend/kv/kv2sql.go

pkg/lightning/restore/duplicate.go

go.mod1

commit 1ed2b5b Author: Little-Wallace <bupt2013211450@gmail.com> Date: Wed Jul 14 12:11:19 2021 +0800 fix kvproto Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 3187304 Author: Little-Wallace <bupt2013211450@gmail.com> Date: Tue Jun 8 16:27:42 2021 +0800 update getValues to public Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit b8a36b4 Author: Little-Wallace <bupt2013211450@gmail.com> Date: Tue Jun 8 16:12:25 2021 +0800 refactor duplicate index Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit bab4072 Author: Little-Wallace <bupt2013211450@gmail.com> Date: Mon May 31 17:58:40 2021 +0800 fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 2dfebb0 Author: Little-Wallace <bupt2013211450@gmail.com> Date: Mon May 31 17:30:40 2021 +0800 use io.EOF to judge end Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit b9b6a31 Author: Little-Wallace <bupt2013211450@gmail.com> Date: Thu May 27 17:08:25 2021 +0800 add some node Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 34b2150 Author: Little-Wallace <bupt2013211450@gmail.com> Date: Tue May 25 20:37:12 2021 +0800 duplicate data before checksum Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 4341adc Author: Little-Wallace <bupt2013211450@gmail.com> Date: Tue May 25 17:44:38 2021 +0800 support decode kv Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 18edf13 Author: Little-Wallace <bupt2013211450@gmail.com> Date: Mon May 24 23:58:33 2021 +0800 retry region error Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit bdd64be Author: Little-Wallace <bupt2013211450@gmail.com> Date: Mon May 24 20:02:40 2021 +0800 add duplicate manager Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

tests: add test for cross engine duplicate detection

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

pkg/lightning/backend/backend.go

kennytm · 2021-07-30T08:28:41Z

tests/lightning_duplicate_detection/config1.toml

+pattern = '(?i)^(?:[^/]*/)*([^/.]+)\.(.*?)\.0\.sql$'
+schema = '$1'
+table = '$2'
+key = '0'


(optional)

perhaps easier just use two distinct data-source-dir with no-schema = true.

I do not understand...

pkg/lightning/backend/kv/kv2sql.go

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

kennytm · 2021-07-30T10:27:18Z

pkg/lightning/backend/local/local.go

+
+	"github.com/pingcap/parser/mysql"
+
 	"github.com/cockroachdb/pebble"


Suggested change

"github.com/pingcap/parser/mysql"

"github.com/cockroachdb/pebble"

"github.com/pingcap/parser/mysql"

"github.com/cockroachdb/pebble"

@Little-Wallace still not fixed 🤔

kennytm · 2021-07-30T10:39:49Z

pkg/lightning/backend/local/local.go

+		return errors.Annotate(err, "collect local duplicate keys failed")
+	}
+	if err = duplicateManager.CollectDuplicateRowsFromTiKV(ctx, tbl); err != nil {
+		return errors.Annotate(err, "duplicate table failed")


Suggested change

return errors.Annotate(err, "duplicate table failed")

return errors.Annotate(err, "collect remote duplicate keys failed")

pkg/lightning/backend/local/local.go

pkg/lightning/backend/noop/noop.go

kennytm · 2021-07-30T14:22:46Z

pkg/lightning/backend/local/duplicate.go

+
+	"golang.org/x/sync/errgroup"
+
+	split "github.com/pingcap/br/pkg/restore"


this should go to the import group with prefix github.com/pingcap/br/*

and why rename it to split 🤣

I just copy it from another place.....

kennytm · 2021-07-30T14:23:27Z

pkg/lightning/backend/local/duplicate.go

+	"github.com/cockroachdb/pebble"
+	"github.com/pingcap/errors"
+	sst "github.com/pingcap/kvproto/pkg/import_sstpb"
+	kvrpc "github.com/pingcap/kvproto/pkg/kvrpcpb"
+	tikv "github.com/pingcap/kvproto/pkg/tikvpb"


these should join all other third-party imports (the group shared by zap, grpc, tidb, etc.)

pkg/lightning/backend/local/duplicate.go

kennytm

no other issues from me.

pkg/lightning/backend/local/duplicate.go

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

pkg/lightning/backend/backend.go

Little-Wallace · 2021-08-03T04:46:48Z

/test

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

kennytm

rest LGTM

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

pkg/lightning/backend/local/duplicate.go

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

kennytm · 2021-08-03T14:22:35Z

/merge

ti-chi-bot · 2021-08-03T14:22:38Z

This pull request has been accepted and is ready to merge.

Commit hash: fb64eaa

ti-chi-bot added the size/XXL label May 25, 2021

ti-chi-bot requested review from glorv and overvenus May 25, 2021 09:55

lonng reviewed May 27, 2021

View reviewed changes

pkg/lightning/backend/kv/kv2sql.go Show resolved Hide resolved

ti-chi-bot added needs-rebase and removed needs-rebase labels May 27, 2021

Little-Wallace force-pushed the duplicate branch from 71c8655 to bab4072 Compare May 31, 2021 10:58

ti-chi-bot added the needs-rebase label May 31, 2021

kennytm reviewed Jun 24, 2021

View reviewed changes

pkg/lightning/restore/duplicate.go Outdated Show resolved Hide resolved

kennytm reviewed Jun 24, 2021

View reviewed changes

go.mod1 Outdated Show resolved Hide resolved

ti-chi-bot removed the needs-rebase label Jul 14, 2021

Little-Wallace force-pushed the duplicate branch from 15848b2 to 9657cef Compare July 14, 2021 04:18

Little-Wallace and others added 11 commits July 14, 2021 12:19

fix conflict

7d57e17

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

fic fmt

1f03ab1

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

fix test compile

beab74a

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

move duplicate to local

abbf24b

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

add method

42f5fac

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

fix get values bug

14ac67e

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

fix read remote tikv

abe67b9

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

Report duplicate keys

b8b9dbe

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

add test for cross engine duplicate detection

9ccde16

Merge pull request #1 from gozssky/dup-test

62a271f

tests: add test for cross engine duplicate detection

Little-Wallace force-pushed the duplicate branch 3 times, most recently from 785b622 to dd6c16a Compare July 25, 2021 09:10

Little-Wallace added 2 commits July 25, 2021 17:11

fix range bug

dd6c16a

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

fix fmt

0af24e9

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

kennytm reviewed Jul 30, 2021

View reviewed changes

pkg/lightning/backend/kv/kv2sql.go Outdated Show resolved Hide resolved

kennytm reviewed Jul 30, 2021

View reviewed changes

pkg/lightning/backend/kv/kv2sql.go Show resolved Hide resolved

address comment

d9e2de4

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

kennytm reviewed Jul 30, 2021

View reviewed changes

pkg/lightning/backend/local/duplicate.go Outdated Show resolved Hide resolved

Little-Wallace added 3 commits August 2, 2021 19:03

Merge branch 'master' into duplicate

30e0978

address comment

3339bd5

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

fix duplicaterequest field

7cb54b3

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

sleepymole reviewed Aug 3, 2021

View reviewed changes

pkg/lightning/backend/backend.go Show resolved Hide resolved

Little-Wallace added 3 commits August 3, 2021 13:03

fix warn

6010c07

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

split local and remote keys

618b406

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

fix fmt

366204e

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

kennytm reviewed Aug 3, 2021

View reviewed changes

fix fmt

ddb54ba

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

kennytm approved these changes Aug 3, 2021

View reviewed changes

pkg/lightning/backend/local/duplicate.go Outdated Show resolved Hide resolved

ti-chi-bot added the status/LGT1 LGTM1 label Aug 3, 2021

fix fmt

fb64eaa

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

sleepymole approved these changes Aug 3, 2021

View reviewed changes

ti-chi-bot added status/LGT2 LGTM2 and removed status/LGT1 LGTM1 labels Aug 3, 2021

Little-Wallace added the status/can-merge label Aug 3, 2021

ti-chi-bot removed the status/can-merge label Aug 3, 2021

Little-Wallace added the component/import label Aug 3, 2021

ti-chi-bot added the status/can-merge label Aug 3, 2021

ti-chi-bot merged commit 0c84977 into pingcap:master Aug 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect duplicate data from TiKV #1144

Detect duplicate data from TiKV #1144

Little-Wallace commented May 25, 2021

ti-chi-bot commented May 25, 2021 •

edited

Loading

CLAassistant commented May 25, 2021 •

edited

Loading

kennytm Jul 30, 2021

Little-Wallace Jul 30, 2021

kennytm Jul 30, 2021

kennytm Aug 3, 2021

kennytm Jul 30, 2021

kennytm Jul 30, 2021

Little-Wallace Aug 3, 2021

kennytm Jul 30, 2021

kennytm left a comment

Little-Wallace commented Aug 3, 2021

kennytm left a comment

kennytm commented Aug 3, 2021

ti-chi-bot commented Aug 3, 2021


		"github.com/pingcap/parser/mysql"

		"github.com/cockroachdb/pebble"

	return errors.Annotate(err, "duplicate table failed")
	return errors.Annotate(err, "collect remote duplicate keys failed")


		"golang.org/x/sync/errgroup"

		split "github.com/pingcap/br/pkg/restore"

Detect duplicate data from TiKV #1144

Detect duplicate data from TiKV #1144

Conversation

Little-Wallace commented May 25, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented May 25, 2021 • edited Loading

CLAassistant commented May 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennytm left a comment

Choose a reason for hiding this comment

Little-Wallace commented Aug 3, 2021

kennytm left a comment

Choose a reason for hiding this comment

kennytm commented Aug 3, 2021

ti-chi-bot commented Aug 3, 2021

ti-chi-bot commented May 25, 2021 •

edited

Loading

CLAassistant commented May 25, 2021 •

edited

Loading