Restore: implement restorer for compacted SST/Snapshot/log files #57208

3pointer · 2024-11-07T10:12:42Z

What problem does this PR solve?

Issue Number: close #57209

Problem Summary:

This pull request introduces a unified structure for managing compacted SST, Snapshot, and log files. Previously, these file types were handled independently, leading to redundant code and potential inconsistencies. The new common struct simplifies file management, improves code maintainability, and reduces the likelihood of errors.

What changed and how does it work?

Unified Struct Abstraction: A shared struct has been created to manage compacted SST, Snapshot, and log files collectively, providing a consistent interface across these file types.

// FileRestorer defines the essential methods required for restoring SST files, 
// covering different types of backups and compacted logs, including:
// 1. Raw backup SSTs
// 2. Txn backup SSTs
// 3. TiDB backup SSTs
// 4. Compacted log SSTs
// 
// Base structures for these restorations include:
// - SimpleRestorer: Handles raw, txn, and compacted log backups.
// - MultiTablesRestorer: Manages snapshot-based multi-table restorations.
type FileRestorer interface {
	// Restore imports files to TiKV.
	Restore(onProgress func(int64), files ...BatchRestoreFilesInfo) error
	// WaitUntilFinish waits for all pending restore files to complete.
	WaitUntilFinish() error
	// Close releases resources.
	Close() error
}

Impact: This update significantly streamlines the restoration process, reducing complexity and improving performance by adopting a unified handling approach. Components across the system have been updated to leverage the new struct, fostering consistency and reducing maintenance overhead.
NOTE: This PR doesn't consider the checkpoint logic for compacted log files. I'll do it in another PR.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2024-11-07T10:12:47Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

tiprow · 2024-11-07T10:13:16Z

Hi @3pointer. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-11-12T10:19:21Z

Codecov Report

Attention: Patch coverage is 74.02597% with 280 lines in your changes missing coverage. Please review.

Project coverage is 74.8449%. Comparing base (d86a366) to head (fed41b5).
Report is 17 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57208        +/-   ##
================================================
+ Coverage   72.8440%   74.8449%   +2.0009%     
================================================
  Files          1672       1721        +49     
  Lines        462716     474632     +11916     
================================================
+ Hits         337061     355238     +18177     
+ Misses       104867      97306      -7561     
- Partials      20788      22088      +1300

Flag	Coverage Δ
integration	`49.5579% <48.1447%> (?)`
unit	`72.4389% <47.8664%> (+0.2170%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7673% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`60.9165% <74.0259%> (+15.8380%)`	⬆️

---- 🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

YuJuncen

rest lgtm

YuJuncen · 2024-11-14T07:50:20Z

br/pkg/restore/log_client/compacted_file_strategy.go

+	}
+	if len(sstOutputs) != len(subCompaction.SstOutputs) {
+		log.Info("partial files in sub compaction skipped due to checkpoint")
+		subCompaction.SstOutputs = sstOutputs


It is a little strange that a predicate function modifies its argument. Would you add some comments in the interface?

This is because the subCompaction is not a basic restore unit during restore, the restore unit is file. and subCompaction might contains multiple files. sometimes we may need to skip partial files in one subCompaction.

br/pkg/restore/log_client/log_file_manager_test.go

YuJuncen · 2024-11-14T08:10:01Z

br/pkg/restore/split/splitter.go

+	// Skip the file by checkpoints or invalid files
+	ShouldSkip(T) bool
+	// GetAccumulations returns an iterator for the accumulated values.
+	GetAccumulations() *SplitHelperIterator


It is a little strange that Accumulate receives T but GetAccumulations returns Valued? It seems this method actually returns the keys to be splitted.

Yes, I want to refactor SplitHelperIterator too, but I suggest doing this in future PR.

br/pkg/restore/split/splitter.go

br/pkg/restore/restorer.go

YuJuncen · 2024-11-14T09:14:29Z

br/pkg/restore/restorer.go

+	return f[:idx]
+}
+
+type PipelineRestorerWrapper[T any] struct {


Perhaps this can be a plain function. So the type argument can be deduced, no explicit annotations like restore.PipelineRestorerWrapper[*logclient.LogDataFileInfo] needed.

br/pkg/restore/split/splitter.go

3pointer · 2024-11-17T14:06:43Z

/test unit-test

ti-chi-bot · 2024-11-17T14:06:46Z

@3pointer: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-e2e-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_br_integration_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_lightning_integration_test
pingcap/tidb/pull_mysql_client_test

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

3pointer · 2024-11-17T14:06:48Z

/test pull-lightning-integration-test

tiprow · 2024-11-17T14:07:06Z

@3pointer: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2024-11-17T14:07:10Z

@3pointer: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2024-11-19T10:02:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Leavrth, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~br/OWNERS~~ [Leavrth,YuJuncen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-11-19T10:02:27Z

[LGTM Timeline notifier]

Timeline:

2024-11-18 02:14:05.854802798 +0000 UTC m=+840808.045671795: ☑️ agreed by YuJuncen.
2024-11-19 10:02:25.461273552 +0000 UTC m=+955307.652142544: ☑️ agreed by Leavrth.

Leavrth · 2024-11-19T10:03:15Z

/hold

3pointer · 2024-11-20T02:04:29Z

/unhold

3pointer · 2024-11-20T02:34:08Z

/test pull-lightning-integration-test

tiprow · 2024-11-20T02:34:29Z

@3pointer: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

restore: implement restorer for compacted SST/Snapshot/log files

1783421

ti-chi-bot bot added do-not-merge/needs-triage-completed and removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Nov 7, 2024

3pointer added 6 commits November 7, 2024 18:25

resolve conflicts

d449c73

Merge branch 'master' into feat/compacted_logs_2nd

5fab677

refine code after merge master

73959f4

add test cases

c6d643a

rename some names && more test cases

3674e0e

remove useless code

7ef0f7d

3pointer marked this pull request as ready for review November 12, 2024 09:59

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 12, 2024

3pointer added 4 commits November 13, 2024 11:11

fix build

23a5d18

polish code

0ec3cb2

add NewSnapFileImporterOptions

c775e12

add tryMap for br iter

329bf69

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Nov 14, 2024

3pointer added 2 commits November 14, 2024 11:54

fix bazel

4dcd764

fix integration tests

ed5d95e

YuJuncen reviewed Nov 14, 2024

View reviewed changes

3pointer added 2 commits November 14, 2024 17:57

address comments

57d15db

address comments

57191e0

3pointer added 2 commits November 17, 2024 21:08

update case

5e2eac4

update cases

fafb0d2

3pointer added 3 commits November 17, 2024 22:19

Merge branch 'master' into feat/compacted_logs_2nd

d182593

fix the test after merge master

1f1b636

fix cases

b9651ed

YuJuncen approved these changes Nov 18, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 18, 2024

3pointer added 3 commits November 18, 2024 11:15

fix ut

c406c7a

fix ut after filterout moving forward

5346932

resolve conflicts

b7bda58

Leavrth approved these changes Nov 19, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 19, 2024

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 19, 2024

address comment

fed41b5

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 20, 2024

ti-chi-bot bot merged commit 9530fdc into pingcap:master Nov 20, 2024
29 checks passed

3pointer mentioned this pull request Nov 20, 2024

Implement different split strategy for logs and compacted sst files. #57212

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore: implement restorer for compacted SST/Snapshot/log files #57208

Restore: implement restorer for compacted SST/Snapshot/log files #57208

3pointer commented Nov 7, 2024 •

edited

Loading

ti-chi-bot bot commented Nov 7, 2024

tiprow bot commented Nov 7, 2024

codecov bot commented Nov 12, 2024 •

edited

Loading

YuJuncen left a comment

YuJuncen Nov 14, 2024

3pointer Nov 14, 2024

YuJuncen Nov 14, 2024

3pointer Nov 14, 2024

YuJuncen Nov 14, 2024

3pointer commented Nov 17, 2024

ti-chi-bot bot commented Nov 17, 2024

3pointer commented Nov 17, 2024

tiprow bot commented Nov 17, 2024

tiprow bot commented Nov 17, 2024

ti-chi-bot bot commented Nov 19, 2024

ti-chi-bot bot commented Nov 19, 2024

Leavrth commented Nov 19, 2024

3pointer commented Nov 20, 2024

3pointer commented Nov 20, 2024

tiprow bot commented Nov 20, 2024

Restore: implement restorer for compacted SST/Snapshot/log files #57208

Restore: implement restorer for compacted SST/Snapshot/log files #57208

Conversation

3pointer commented Nov 7, 2024 • edited Loading

What problem does this PR solve?

Problem Summary:

What changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Nov 7, 2024

tiprow bot commented Nov 7, 2024

codecov bot commented Nov 12, 2024 • edited Loading

Codecov Report

YuJuncen left a comment

Choose a reason for hiding this comment

YuJuncen Nov 14, 2024

Choose a reason for hiding this comment

3pointer Nov 14, 2024

Choose a reason for hiding this comment

YuJuncen Nov 14, 2024

Choose a reason for hiding this comment

3pointer Nov 14, 2024

Choose a reason for hiding this comment

YuJuncen Nov 14, 2024

Choose a reason for hiding this comment

3pointer commented Nov 17, 2024

ti-chi-bot bot commented Nov 17, 2024

3pointer commented Nov 17, 2024

tiprow bot commented Nov 17, 2024

tiprow bot commented Nov 17, 2024

ti-chi-bot bot commented Nov 19, 2024

ti-chi-bot bot commented Nov 19, 2024

[LGTM Timeline notifier]

Leavrth commented Nov 19, 2024

3pointer commented Nov 20, 2024

3pointer commented Nov 20, 2024

tiprow bot commented Nov 20, 2024

3pointer commented Nov 7, 2024 •

edited

Loading

codecov bot commented Nov 12, 2024 •

edited

Loading