br: support PiTR feature #34409

joccau · 2022-05-06T07:02:52Z

What problem does this PR solve?

Issue Number: ref #29501

Problem Summary:

What is changed and how it works?

The new feature: PiTR

Take a backup of key-value recored in TiKV If you run br log start ...
Restore table and record from snapshot-ts to restore-ts if you run br restore point ...

Relative design/tests

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Support the new feature of PiTR.

* stream: initial commit of stream backup * br/stream: basic logic of stream backup metadata * br/stream: add prefix scanner * br/stream: use types from kv package * br/stream: unexport prefix scanner(remove it to utils package and export it in need) * br/stream: move kvproto to br-stream branch, added a playground * br/stream: added some new functions for playground * br/stream: added a builder for task * br/stream: remove playground, use math.MaxUInt64 * br: replace kvproto with br-stream * br/stream: update error code * br/stream: fix tests * br/stream: added copyright

* Add stream start/pause/resume/stop command Signed-off-by: joccau <zak.zhao@pingcap.com>

* Add stream start/pause/resume/stop command Signed-off-by: joccau <zak.zhao@pingcap.com> * Add build observe ranges Signed-off-by: joccau <zak.zhao@pingcap.com> * Compare startTS and endTS Signed-off-by: joccau <zak.zhao@pingcap.com> * test for stream task Signed-off-by: joccau <zak.zhao@pingcap.com> * test for stream task Signed-off-by: joccau <zak.zhao@pingcap.com> * Deal comments in pr Signed-off-by: joccau <zak.zhao@pingcap.com> * Add comment for function Signed-off-by: joccau <zak.zhao@pingcap.com> * modify comments Signed-off-by: joccau <zak.zhao@pingcap.com> * update building of key-range for tableID Signed-off-by: joccau <zak.zhao@pingcap.com> * resolve the check issue Signed-off-by: joccau <zak.zhao@pingcap.com> * Deal comments in pr Signed-off-by: joccau <zak.zhao@pingcap.com> * Debug stream log to test flow Signed-off-by: joccau <zak.zhao@pingcap.com> * Deal comments in PR Signed-off-by: joccau <zak.zhao@pingcap.com> * use + / rather join Signed-off-by: joccau <zak.zhao@pingcap.com> * maintain test case for backup stream Signed-off-by: joccau <zak.zhao@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Add stream start/pause/resume/stop command Signed-off-by: joccau <zak.zhao@pingcap.com>

* add restore command * coarse implement restore

* add restore_ts for apply * update kvproto

* fix file rewrite end key Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * run go fmt Signed-off-by: Yu Juncen <yu745514916@live.com> * run go mod tidy Signed-off-by: Yu Juncen <yu745514916@live.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* br-stream: add region context in applyRequest

* add ObjPrefix to speed up walkDir in br-stream * add start/end key to filter unnecessay key

* Add code for stream restore metaKV Signed-off-by: joccau <zak.zhao@pingcap.com>

* add switch check before stream starts

* added truncate log Signed-off-by: Yu Juncen <yu745514916@live.com>

* observer all data in stream task Signed-off-by: joccau <zak.zhao@pingcap.com>

* support checkSum during restore dml kv-events. Signed-off-by: joccau <zak.zhao@pingcap.com> * update usage API Signed-off-by: joccau <zak.zhao@pingcap.com> * fix bug in stream start Signed-off-by: joccau <zak.zhao@pingcap.com> * maintain testcase Signed-off-by: joccau <zak.zhao@pingcap.com>

* restore full and stream by command "br stream restore" Signed-off-by: joccau <zak.zhao@pingcap.com>

* support task info Signed-off-by: Yu Juncen <yu745514916@live.com> * added QPS Signed-off-by: Yu Juncen <yu745514916@live.com> address comments Signed-off-by: Yu Juncen <yu745514916@live.com> * set the start ts as the default global checkpoint Signed-off-by: Yu Juncen <yu745514916@live.com>

commit 981c9c39f9b7fe4cbe6e6f3cbf2a0c57838c1224 Author: Yu Juncen <yu745514916@live.com> Date: Thu Mar 17 17:22:53 2022 +0800 fix tests Signed-off-by: Yu Juncen <yu745514916@live.com> commit b3bafa8f0f8fd1f243c990179cbbfe8ecf2fed7e Author: Yu Juncen <yu745514916@live.com> Date: Thu Mar 17 11:14:07 2022 +0800 retry if epoch not match during finding leader Signed-off-by: Yu Juncen <yu745514916@live.com> commit cf0bf4b0d61e163eb527db032dd880dbbabed809 Author: Yu Juncen <yu745514916@live.com> Date: Wed Mar 16 16:34:27 2022 +0800 fix import error ignored Signed-off-by: Yu Juncen <yu745514916@live.com> commit 0ae287d65465856b013680bc2183d60a08c6b37f Author: Yu Juncen <yu745514916@live.com> Date: Fri Mar 11 14:48:06 2022 +0800 fix typo Signed-off-by: Yu Juncen <yu745514916@live.com> commit 4eba96e5cc6110734233376b39216fbb66a88d8d Author: Yu Juncen <yu745514916@live.com> Date: Fri Mar 11 14:09:31 2022 +0800 run tests sequential Signed-off-by: Yu Juncen <yu745514916@live.com> commit 4b659359963c141ab405dca52f3607f10f2be6a2 Author: Yu Juncen <yu745514916@live.com> Date: Fri Mar 11 12:10:48 2022 +0800 update go.mod Signed-off-by: Yu Juncen <yu745514916@live.com> commit fc2d9290cd8d5782455565175c78edbfb43aac60 Author: Yu Juncen <yu745514916@live.com> Date: Fri Mar 11 11:53:43 2022 +0800 update kvproto Signed-off-by: Yu Juncen <yu745514916@live.com> commit 365975b6ef2341f4307ac39b43af9bdd037b74b4 Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu Mar 10 20:33:10 2022 +0800 added test cases Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 08502b5c0670989474b6de5239235dede72f581f Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed Mar 9 18:23:58 2022 +0800 added some tools for retry Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit b8d518b69dd827680f074ec1899cadf1b00647d3 Merge: 5c329e46f ce0b12c Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed Mar 9 16:06:36 2022 +0800 Merge branch 'br-stream' of https://github.com/pingcap/tidb into retry-apply Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 5c329e46f70424c27df3862a407d4a9751af1824 Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed Mar 9 15:55:09 2022 +0800 added retry Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 044d804 Merge: 0451199 ebd27f8 Author: 3pointer <luancheng@pingcap.com> Date: Tue Mar 8 16:59:22 2022 +0800 Merge branch 'br-stream' into key_not_found commit 0451199 Author: 3pointer <luancheng@pingcap.com> Date: Tue Mar 8 16:47:47 2022 +0800 address comment commit b5470a9 Author: 3pointer <luancheng@pingcap.com> Date: Mon Mar 7 19:30:23 2022 +0800 fix check commit 00d49d8 Author: 3pointer <luancheng@pingcap.com> Date: Mon Mar 7 18:57:58 2022 +0800 format commit 232a646 Author: 3pointer <luancheng@pingcap.com> Date: Mon Mar 7 18:49:32 2022 +0800 update kvproto commit d46105e Author: 3pointer <luancheng@pingcap.com> Date: Fri Mar 4 12:28:58 2022 +0800 add start/end key to filter unnecessay key commit d61a01f Author: 3pointer <luancheng@pingcap.com> Date: Fri Mar 4 12:26:17 2022 +0800 add ObjPrefix to speed up walkDir in br-stream Signed-off-by: Yu Juncen <yu745514916@live.com>

…k here (#33725) * refactor the status command && print "no task" when no task here Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * added some test cases Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * added the status of backup task Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * fix typo Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * make write success always Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * apply suggestions from comment Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* br: refine cmd stream restore to restore log

* support pause/resume stream task Signed-off-by: joccau <zak.zhao@pingcap.com>

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* show errors Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * fix remove last error Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * ignore tiflash when getting status Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * Update br/pkg/stream/stream_status.go Co-authored-by: 3pointer <qdlc2010@gmail.com> * fix build Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Co-authored-by: 3pointer <qdlc2010@gmail.com>

* remove checkpointTs in pd when stop stream task Signed-off-by: joccau <zak.zhao@pingcap.com>

* br-stream: split put file and delete file in restore * br-stream: implement cleanup files

) * update command name Signed-off-by: joccau <zak.zhao@pingcap.com> * support restoring autoTable Signed-off-by: joccau <zak.zhao@pingcap.com> * report other tidb after update global schema version Signed-off-by: joccau <zak.zhao@pingcap.com> * support restoring sequence id Signed-off-by: joccau <zak.zhao@pingcap.com> * make fmt

sre-bot · 2022-05-06T12:17:40Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/4dcbb0f8df492c6c6c80864c064d070d913d3ebb

joccau · 2022-05-07T02:13:28Z

/component br

br/pkg/utils/schema.go

Signed-off-by: joccau <zak.zhao@pingcap.com>

br/cmd/br/stream.go

br/pkg/gluetidb/glue.go

br/pkg/restore/client.go

wjhuang2016 · 2022-05-09T09:37:59Z

br/pkg/restore/client.go

+}
+
+// UpdateSchemaVersion updates schema version by transaction way.
+func (rc *Client) UpdateSchemaVersion(ctx context.Context) error {


It's not compatible with the concurrent DDL, we need to take care of it.

Because PiTR is used in an empty cluster, the concurrent scenarios does not happen.
The PiTR is contains 2 parts:

restore snapshot

restore log

br/pkg/restore/import_retry.go

br/pkg/restore/util.go

* rewrite db Signed-off-by: joccau <zak.zhao@pingcap.com> * rewrite table Signed-off-by: joccau <zak.zhao@pingcap.com> * restore log from start-ts to restore-ts Signed-off-by: joccau <zak.zhao@pingcap.com> * safe old schemas after restore log Signed-off-by: joccau <zak.zhao@pingcap.com> * batch put raw kv Signed-off-by: joccau <zak.zhao@pingcap.com> * remove useless function and set 64 to rawkv put batch count Signed-off-by: joccau <zak.zhao@pingcap.com> * rename backup-stream to log-backup Signed-off-by: joccau <zak.zhao@pingcap.com> * deal comments in PR Signed-off-by: joccau <zak.zhao@pingcap.com>

3pointer · 2022-05-10T12:18:44Z

/run-integration-br-tests

Signed-off-by: joccau <zak.zhao@pingcap.com>

joccau · 2022-05-11T04:34:08Z

/run-integration-br-test

YuJuncen

Given all commits LGTM, trivially(?) this PR LGTM.

3pointer

LGTM

* remove the duplicated meta key entry

Signed-off-by: joccau <zak.zhao@pingcap.com>

joccau · 2022-05-12T06:42:12Z

/merge

ti-chi-bot · 2022-05-12T06:42:15Z

This pull request has been accepted and is ready to merge.

Commit hash: 952ea7c

joccau · 2022-05-12T07:29:55Z

/run-all-tests

joccau · 2022-05-12T07:55:08Z

/run-all-tests

joccau · 2022-05-12T09:01:01Z

/run-all-tests

sre-bot · 2022-05-12T09:40:40Z

TiDB MergeCI notify

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci/integration-cdc-test	🟢 all 34 tests passed	26 min	Existing passed
idc-jenkins-ci-tidb/integration-common-test	🟢 all 11 tests passed	13 min	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	7 min 5 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	6 min 55 sec	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 12 tests passed	6 min 54 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	6 min 40 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	6 min 30 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	3 min 27 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 2 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

YuJuncen and others added 30 commits March 21, 2022 17:48

br: backup log command(stream start/stop) (#30308)

de8a63d

* Add stream start/pause/resume/stop command Signed-off-by: joccau <zak.zhao@pingcap.com>

br stream: set default startTS and print steam task into screen (#31020)

4af4f47

* Add stream start/pause/resume/stop command Signed-off-by: joccau <zak.zhao@pingcap.com>

br-stream: implement restore command (#31821)

ce0538a

* add restore command * coarse implement restore

br-stream: backup record key only (#32168)

f09418f

br-stream: add restore_ts for apply (#32250)

6ce6e00

* add restore_ts for apply * update kvproto

rebase to master

bde51d8

br-stream: add region context in applyRequest (#32657)

e7174fb

* br-stream: add region context in applyRequest

br/restore: Added progress bar for restore (#32248)

6f736f1

br-stream: fix the issue that key is not in region. (#32877)

041c1d9

* add ObjPrefix to speed up walkDir in br-stream * add start/end key to filter unnecessay key

br: Stream restore ddl-events from observing key-value on TiKV (#32687)

a7e8432

* Add code for stream restore metaKV Signed-off-by: joccau <zak.zhao@pingcap.com>

br-stream: add precheck before create task (#33088)

6b94fd7

* add switch check before stream starts

rebase to master

a32b7f8

br/stream: added truncate log (#33092)

05ba1e8

* added truncate log Signed-off-by: Yu Juncen <yu745514916@live.com>

br: observe all of tables and support create/truncate table (#33211)

6a5a70c

* observer all data in stream task Signed-off-by: joccau <zak.zhao@pingcap.com>

br: restore full and stream by one command (#33451)

46483b4

* restore full and stream by command "br stream restore" Signed-off-by: joccau <zak.zhao@pingcap.com>

br: refine cmd stream restore to restore log (#33734)

e8c87fd

* br: refine cmd stream restore to restore log

fix infinite help function loop (#33994)

7c9f385

br: support pause/resume stream task (#33872)

5020e3c

* support pause/resume stream task Signed-off-by: joccau <zak.zhao@pingcap.com>

basic libraries for error handing (#33884)

7582cb5

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

br: fix bug in PiTR (#34086)

ab8bf9d

* remove checkpointTs in pd when stop stream task Signed-off-by: joccau <zak.zhao@pingcap.com>

br-stream: implement cleanup files. (#34103)

8a6e686

* br-stream: split put file and delete file in restore * br-stream: implement cleanup files

ti-chi-bot added the component/br This issue is related to BR of TiDB. label May 7, 2022

3pointer reviewed May 7, 2022

View reviewed changes

br/pkg/utils/schema.go Outdated Show resolved Hide resolved

make tidy (#34426)

af0d5f8

Signed-off-by: joccau <zak.zhao@pingcap.com>

wjhuang2016 reviewed May 9, 2022

View reviewed changes

3pointer and others added 2 commits May 10, 2022 20:46

Merge branch 'master' into br-stream

dccf367

support empty schema in PiTR. (#34551)

f894c48

Signed-off-by: joccau <zak.zhao@pingcap.com>

YuJuncen approved these changes May 11, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 11, 2022

3pointer approved these changes May 11, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 11, 2022

3pointer and others added 3 commits May 11, 2022 21:14

backup-log: remove duplicated meta key in a batch. (#34577)

4431dc3

* remove the duplicated meta key entry

Merge branch 'master' into br-stream

347dbd4

move meta file into v1 dir (#34401)

952ea7c

Signed-off-by: joccau <zak.zhao@pingcap.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 12, 2022

Merge branch 'master' into br-stream

4dcbb0f

ti-chi-bot merged commit c6fe032 into master May 12, 2022

ti-chi-bot deleted the br-stream branch May 12, 2022 09:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

br: support PiTR feature #34409

br: support PiTR feature #34409

joccau commented May 6, 2022 •

edited by 3pointer

Loading

sre-bot commented May 6, 2022 •

edited

Loading

joccau commented May 7, 2022

wjhuang2016 May 9, 2022

joccau May 10, 2022 •

edited

Loading

3pointer commented May 10, 2022

joccau commented May 11, 2022

YuJuncen left a comment

3pointer left a comment

joccau commented May 12, 2022

ti-chi-bot commented May 12, 2022

joccau commented May 12, 2022

joccau commented May 12, 2022

joccau commented May 12, 2022

sre-bot commented May 12, 2022

br: support PiTR feature #34409

br: support PiTR feature #34409

Conversation

joccau commented May 6, 2022 • edited by 3pointer Loading

What problem does this PR solve?

What is changed and how it works?

Relative design/tests

Check List

Release note

sre-bot commented May 6, 2022 • edited Loading

joccau commented May 7, 2022

wjhuang2016 May 9, 2022

Choose a reason for hiding this comment

joccau May 10, 2022 • edited Loading

Choose a reason for hiding this comment

3pointer commented May 10, 2022

joccau commented May 11, 2022

YuJuncen left a comment

Choose a reason for hiding this comment

3pointer left a comment

Choose a reason for hiding this comment

joccau commented May 12, 2022

ti-chi-bot commented May 12, 2022

joccau commented May 12, 2022

joccau commented May 12, 2022

joccau commented May 12, 2022

sre-bot commented May 12, 2022

TiDB MergeCI notify

joccau commented May 6, 2022 •

edited by 3pointer

Loading

sre-bot commented May 6, 2022 •

edited

Loading

joccau May 10, 2022 •

edited

Loading