lightning: fix id too large after parallel import #57398

D3Hunter · 2024-11-15T04:32:49Z

What problem does this PR solve?

Issue Number: close #56814

Problem Summary:

What changed and how does it work?

previously with parallel import, the auto id after import is the estimated ones even though there are IDs inside the data source file, we change to the ones inside the data source file to avoid next auto id too large

to do this, we need to save different values for different auto id in checkpoint

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
- parallel import with 2 instances, next id is the max id in data files + 1
- open parallel import, but start instance B after the first instance finish, so we can see B is checksuming existing data, and success
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-11-15T04:33:04Z

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2024-11-15T04:39:58Z

lightning/pkg/importer/meta_manager.go

+			// if another instance finished import before below checksum logic,
+			// it will cause checksum mismatch, but it's very rare.


codecov · 2024-11-15T04:50:52Z

Codecov Report

Attention: Patch coverage is 78.41410% with 49 lines in your changes missing coverage. Please review.

Project coverage is 74.0038%. Comparing base (a5832b4) to head (3889d53).
Report is 181 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57398        +/-   ##
================================================
+ Coverage   72.9011%   74.0038%   +1.1026%     
================================================
  Files          1668       1702        +34     
  Lines        461134     480713     +19579     
================================================
+ Hits         336172     355746     +19574     
+ Misses       104308     103452       -856     
- Partials      20654      21515       +861

Flag	Coverage Δ
integration	`46.4629% <74.8898%> (?)`
unit	`72.4956% <56.2500%> (+0.2109%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7673% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`44.4313% <ø> (-0.6780%)`	⬇️

D3Hunter · 2024-11-18T06:25:42Z

@lance6716 ping

lance6716

(will review soon)

lance6716 · 2024-11-18T06:11:19Z

Makefile

@@ -530,7 +530,7 @@ br_bins:

 .PHONY: data_parsers
 data_parsers: tools/bin/vfsgendev pkg/lightning/mydump/parser_generated.go lightning_web
-	PATH="$(GOPATH)/bin":"$(PATH)":"$(TOOLS)" protoc -I. -I"$(GOPATH)/src" pkg/lightning/checkpoints/checkpointspb/file_checkpoints.proto --gogofaster_out=.
+	PATH="$(GOPATH)/bin":"$(PATH)":"$(TOOLS)" protoc -I. -I"$(GOPATH)/pkg/mod" pkg/lightning/checkpoints/checkpointspb/file_checkpoints.proto --gogofaster_out=.


maybe use GOMODCACHE value from go env

https://go.dev/ref/mod#environment-variables

lance6716

15 / 32 files viewed

lance6716 · 2024-11-18T07:53:00Z

lightning/pkg/importer/meta_manager.go

@@ -188,22 +188,31 @@ func (m *dbTableMetaMgr) AllocTableRowIDs(ctx context.Context, rawRowIDMax int64
 		DB:     conn,
 		Logger: m.tr.logger,
 	}
-	var newRowIDBase, newRowIDMax int64
-	curStatus := metaStatusInitial
+	// (myStartRowID, myEndRowID] is the range of row_id that current instance


can we use [start, end)? This is a bit strange and we have to -1 in GetMaxAutoIDBAse

that's how auto id allocator works now, both in TiDB and lightning, won't do it here

lance6716

27 / 32 files viewed

lance6716 · 2024-11-18T08:39:05Z

lightning/pkg/importer/import.go

@@ -1990,20 +1990,12 @@ type deliverResult struct {
 }

 func saveCheckpoint(rc *Controller, t *TableImporter, engineID int32, chunk *checkpoints.ChunkCheckpoint) {
-	// We need to update the AllocBase every time we've finished a file.


this line of comment can be kept. Maybe it reminds us to keep the consistency of "ID used for encoding data" and "ID saved in checkpoint"

lightning/pkg/importer/table_import.go

ti-chi-bot · 2024-11-18T11:29:25Z

@lance6716: Your lgtm message is repeated, so it is ignored.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

GMHDBJD

LGTM

ti-chi-bot · 2024-11-19T05:58:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: GMHDBJD, lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [GMHDBJD,lance6716]
~~lightning/OWNERS~~ [GMHDBJD,lance6716]
~~pkg/disttask/OWNERS~~ [GMHDBJD,lance6716]
~~pkg/executor/importer/OWNERS~~ [GMHDBJD,lance6716]
~~pkg/lightning/OWNERS~~ [GMHDBJD,lance6716]
~~pkg/table/OWNERS~~ [GMHDBJD]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-11-19T05:58:17Z

[LGTM Timeline notifier]

Timeline:

2024-11-18 11:29:13.95091031 +0000 UTC m=+874116.141779300: ☑️ agreed by lance6716.
2024-11-19 05:58:16.14053605 +0000 UTC m=+940658.331405044: ☑️ agreed by GMHDBJD.

tiprow · 2024-11-19T06:09:44Z

@D3Hunter: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`3889d53`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

D3Hunter · 2024-11-19T06:32:55Z

/retest

tiprow · 2024-11-19T06:33:16Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2024-11-19T11:04:14Z

In response to a cherrypick label: new pull request created to branch release-8.5: #57516.

close #56814

ti-chi-bot · 2024-12-03T08:12:12Z

In response to a cherrypick label: new pull request created to branch release-7.5: #57928.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close #56814

D3Hunter added 7 commits November 13, 2024 13:03

add fields

6061a6e

base

74b83cb

change

a7d8625

test

c516fdb

change

613e9c2

bazel

9c02c2c

change

03e16f7

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 15, 2024

ti-chi-bot bot added do-not-merge/needs-triage-completed and removed do-not-merge/needs-linked-issue labels Nov 15, 2024

D3Hunter commented Nov 15, 2024

View reviewed changes

change

c04f322

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Nov 15, 2024

D3Hunter requested a review from lance6716 November 15, 2024 04:59

D3Hunter added 3 commits November 15, 2024 16:56

change

c833bcf

simplify

105f401

comment

4b485e0

lance6716 reviewed Nov 18, 2024

View reviewed changes

ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Nov 18, 2024

D3Hunter mentioned this pull request Nov 18, 2024

dm: after physical import, the auto id might have a large jump pingcap/tiflow#11768

Closed

lance6716 reviewed Nov 18, 2024

View reviewed changes

lance6716 approved these changes Nov 18, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 18, 2024

lance6716 approved these changes Nov 18, 2024

View reviewed changes

fix comments

3889d53

GMHDBJD approved these changes Nov 19, 2024

View reviewed changes

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 19, 2024

ti-chi-bot bot merged commit ecca340 into pingcap:master Nov 19, 2024
24 of 25 checks passed

ti-chi-bot mentioned this pull request Nov 19, 2024

lightning: fix id too large after parallel import (#57398) #57516

Merged

13 tasks

D3Hunter deleted the fix-id-jump-in-parallel branch November 19, 2024 12:41

ti-chi-bot bot pushed a commit that referenced this pull request Nov 20, 2024

lightning: fix id too large after parallel import (#57398) (#57516)

82376c7

close #56814

ti-chi-bot bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Dec 3, 2024

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 3, 2024

This is an automated cherry-pick of pingcap#57398

f28a242

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Dec 3, 2024

lightning: fix id too large after parallel import (#57398) #57928

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Dec 6, 2024

lightning: fix id too large after parallel import (#57398) (#57928)

f8d5c54

close #56814

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lightning: fix id too large after parallel import #57398

lightning: fix id too large after parallel import #57398

D3Hunter commented Nov 15, 2024 •

edited

Loading

tiprow bot commented Nov 15, 2024

D3Hunter Nov 15, 2024

codecov bot commented Nov 15, 2024 •

edited

Loading

D3Hunter commented Nov 18, 2024

lance6716 left a comment

lance6716 Nov 18, 2024

lance6716 left a comment

lance6716 Nov 18, 2024

D3Hunter Nov 18, 2024

lance6716 left a comment

lance6716 Nov 18, 2024

ti-chi-bot bot commented Nov 18, 2024

GMHDBJD left a comment

ti-chi-bot bot commented Nov 19, 2024

ti-chi-bot bot commented Nov 19, 2024

tiprow bot commented Nov 19, 2024

D3Hunter commented Nov 19, 2024

tiprow bot commented Nov 19, 2024

ti-chi-bot commented Nov 19, 2024

ti-chi-bot commented Dec 3, 2024

		// if another instance finished import before below checksum logic,
		// it will cause checksum mismatch, but it's very rare.

lightning: fix id too large after parallel import #57398

lightning: fix id too large after parallel import #57398

Conversation

D3Hunter commented Nov 15, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Nov 15, 2024

D3Hunter Nov 15, 2024

Choose a reason for hiding this comment

codecov bot commented Nov 15, 2024 • edited Loading

Codecov Report

D3Hunter commented Nov 18, 2024

lance6716 left a comment

Choose a reason for hiding this comment

lance6716 Nov 18, 2024

Choose a reason for hiding this comment

lance6716 left a comment

Choose a reason for hiding this comment

lance6716 Nov 18, 2024

Choose a reason for hiding this comment

D3Hunter Nov 18, 2024

Choose a reason for hiding this comment

lance6716 left a comment

Choose a reason for hiding this comment

lance6716 Nov 18, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 18, 2024

GMHDBJD left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 19, 2024

ti-chi-bot bot commented Nov 19, 2024

[LGTM Timeline notifier]

tiprow bot commented Nov 19, 2024

D3Hunter commented Nov 19, 2024

tiprow bot commented Nov 19, 2024

ti-chi-bot commented Nov 19, 2024

ti-chi-bot commented Dec 3, 2024

D3Hunter commented Nov 15, 2024 •

edited

Loading

codecov bot commented Nov 15, 2024 •

edited

Loading