lightning: support restore data into tables that contains data #784

glorv · 2021-03-03T07:10:42Z

What problem does this PR solve?

Support use lightning local/importer backend to load data into tables that already contain data.
Support running multiple lightning in parallel to load data into tables that already contain data.

What is changed and how it works?

Add a new metatable mysql.brie_sub_tasks in TiDB to store the meta data for each table restore task.
Use show table next_row_id for the auto_id allocator base value. This can avoid auto_id conflict with the existing data. If there are other task record for the same table in brie_sub_tasks, the rowid rang will be calculated based on the recored info.
Prefetch the checksum value and store it in the table checkpoint. Also add this value into local checksum when compares local vs remote checksum.
Only do checksum and analyze after the last parallel task is finished, other tasks only put their local checksum info into the metatable, so the last only actually do the checksum can sum all the local checksum to get the right result.

Alternatives:

It's not necessary to put the metatable into mysql db. We can create a temporary table like the checkpoints db on-demand and drop it after all lightning tasks are finished.
- Pros: avoid ask the admin permisssion.
Store the metadata at pd instead of in TiDB.
- pro: maybe easier to manage.
- Drawback: this will only support newer cluster.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has persistent data change

Side effects

Possible performance regression
Increased code complexity

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Release Note

Support load data into tables that contain data.

glorv · 2021-03-03T07:11:56Z

Need to update the Glue interface to support scan or transaction to better support the show table next_row_id result.

…al-restore

glorv · 2021-05-27T02:04:05Z

/run-all-tests

Little-Wallace

LGTM

…-restore

kennytm

rest LGTM

kennytm · 2021-06-03T08:30:57Z

tests/lightning_incremental/run.sh

+for backend in importer local; do
+    if [ "$backend" = 'local' ]; then
+        check_cluster_version 4 0 0 'local backend' || continue
+    fi


this check is useless

kennytm · 2021-06-03T08:59:33Z

pkg/lightning/restore/restore.go

+		Logger:       log.L(),
+		HideQueryLog: redact.NeedRedact(),
+	}
+	metaDBSQL := fmt.Sprintf("CREATE DATABASE IF NOT EXISTS `%s`", b.schema)


the b.schema here and the two below should be identifier-escaped (foo`bar → `foo``bar`)

kennytm · 2021-06-03T09:02:53Z

pkg/lightning/restore/restore.go

+	}
+
+	// avoid override existing metadata if the meta is already inserted.
+	stmt := fmt.Sprintf("DROP DATABASE %s;", m.schemaName)


kennytm · 2021-06-03T09:03:54Z

pkg/lightning/restore/restore.go

@@ -2733,3 +2933,789 @@ func (cr *chunkRestore) restore(
 		return ctx.Err()
 	}
 }
+
+type metaMgrBuilder interface {


let's move these meta-manager stuff into its own .go file to avoid further bloating the restore.go file (it's over 3700 lines after this PR 😅)

…-restore

glorv · 2021-06-04T05:31:30Z

/run-all-tests

ti-chi-bot · 2021-06-04T07:59:45Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Little-Wallace
kennytm

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

glorv · 2021-06-04T08:07:31Z

/merge

ti-chi-bot · 2021-06-04T08:07:33Z

This pull request has been accepted and is ready to merge.

Commit hash: a352c3a

backup/checksum: add cluster index support for ranges (pingcap#1120) lightning/restore: support ingset multi ssts for same range (pingcap#1089) * ingest ssts for the same range in a batch * make ingest compatible with old tikv tests: add row count check for br clustered index tests (pingcap#1151) storage: mkdirAll for local storage even when SkipCheckPath is true (pingcap#1156) tests/br: fix bug in issue pingcap#1158 (pingcap#1160) tests/br/compatibility: fix failed for manifest unknown (pingcap#1161) *: upgrade go version from 1.13/1.15 to 1.16 (pingcap#1159) action trigger: fix compatibility trigger bug on push (pingcap#1170) address comment fix test address comment address comment fix build add ut for ignore columns go.mod: update tidb to the new version (pingcap#1153) lightning: support restore data into tables that contains data (pingcap#784) fix conflict after merge master

…ap#784)

glorv added 2 commits March 2, 2021 16:02

support restore data into tables that contains data

cc0337e

add integration test

5fdbb50

glorv added needs-cherry-pick-release-4.0 component/import labels Mar 3, 2021

glorv added 4 commits March 9, 2021 11:02

allow run multi lightning in parallel

df174b6

reuse task id when recover from checkpoint

224906c

Merge branch 'master' of https://github.com/pingcap/br into increment…

d459855

…al-restore

fix lint

bcba066

glorv force-pushed the incremental-restore branch 4 times, most recently from 8c03754 to b2c981c Compare March 10, 2021 07:16

fix meta table field name

03399d7

glorv force-pushed the incremental-restore branch 4 times, most recently from 9dcb8a6 to c500ab5 Compare March 10, 2021 09:30

fix integration tests

6d89f9d

glorv force-pushed the incremental-restore branch from c500ab5 to 6d89f9d Compare March 10, 2021 10:49

glorv added 10 commits March 10, 2021 20:23

fix rowid check for auto-random

bdcbf3b

Merge branch 'master' into incremental-restore

b0ba237

fix version

d7d0db1

fix auto random

a55d286

Merge branch 'master' of https://github.com/pingcap/br into increment…

e8dceff

…al-restore

fix allocate row_id

0c4d45f

fix

4c977d1

fix close rows

f818bff

add unit test and fix bug

7d64011

Merge branch 'master' into incremental-restore

b7ed935

glorv added 4 commits May 25, 2021 10:42

fix meta mgr for tidb backend

3df592b

remove table empty check

953aa9c

fix schedulers

8da7fd4

add importer backend

dcf1b7d

Little-Wallace approved these changes May 27, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 LGTM1 label May 27, 2021

Little-Wallace removed needs-cherry-pick-release-4.0 needs-cherry-pick-release-5.0 labels May 31, 2021

glorv added this to the v5.1.0 milestone Jun 2, 2021

glorv added 2 commits June 3, 2021 14:29

Merge branch 'master' of ssh://github.com/pingcap/br into incremental…

0c42432

…-restore

return error if target table is in checksum phase

965c7eb

kennytm reviewed Jun 3, 2021

View reviewed changes

glorv added 5 commits June 3, 2021 18:55

resolve comments

059175a

remove useless code

ef782e1

fmt code

a895b15

Merge branch 'master' of ssh://github.com/pingcap/br into incremental…

6a3be0b

…-restore

add copy right header

a352c3a

kennytm approved these changes Jun 4, 2021

View reviewed changes

ti-chi-bot added status/LGT2 LGTM2 and removed status/LGT1 LGTM1 labels Jun 4, 2021

ti-chi-bot added the status/can-merge label Jun 4, 2021

ti-chi-bot merged commit 05beea0 into pingcap:master Jun 4, 2021

glorv deleted the incremental-restore branch June 4, 2021 08:12

3pointer pushed a commit to 3pointer/br that referenced this pull request Jun 4, 2021

lightning: support restore data into tables that contains data (pingc…

ac452ad

…ap#784)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lightning: support restore data into tables that contains data #784

lightning: support restore data into tables that contains data #784

glorv commented Mar 3, 2021 •

edited

Loading

glorv commented Mar 3, 2021

glorv commented May 27, 2021

Little-Wallace left a comment

kennytm left a comment

kennytm Jun 3, 2021

kennytm Jun 3, 2021

kennytm Jun 3, 2021

kennytm Jun 3, 2021

glorv commented Jun 4, 2021

ti-chi-bot commented Jun 4, 2021

glorv commented Jun 4, 2021

ti-chi-bot commented Jun 4, 2021

lightning: support restore data into tables that contains data #784

lightning: support restore data into tables that contains data #784

Conversation

glorv commented Mar 3, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release Note

glorv commented Mar 3, 2021

glorv commented May 27, 2021

Little-Wallace left a comment

Choose a reason for hiding this comment

kennytm left a comment

Choose a reason for hiding this comment

kennytm Jun 3, 2021

Choose a reason for hiding this comment

kennytm Jun 3, 2021

Choose a reason for hiding this comment

kennytm Jun 3, 2021

Choose a reason for hiding this comment

kennytm Jun 3, 2021

Choose a reason for hiding this comment

glorv commented Jun 4, 2021

ti-chi-bot commented Jun 4, 2021

glorv commented Jun 4, 2021

ti-chi-bot commented Jun 4, 2021

glorv commented Mar 3, 2021 •

edited

Loading