lightning: make every engine use a unique commitTS #1101
Conversation
|
|
|
/run-check-release-note |
|
/build |
|
/run-check-release-note |
|
/build |
|
/run-integration-test |
|
/build |
1 similar comment
|
/build |
|
/run-integration-test |
|
/hold |
|
/unhold |
|
/build |
|
/run-integration-test |
pkg/lightning/restore/restore.go
Outdated
| dataSynced = cr.maybeSaveCheckpoint(rc, t, engineID, cr.chunk, dataEngine, indexEngine) | ||
| // When enabled DiskQuota, `enforceDiskQuota` may force the backend to import the content of a large engine | ||
| // into the target and then reset the engine to empty. When this happen, we need to use a new commitTS. | ||
| if err = rc.backend.AllocateTSIfNotExists(ctx, t.tableName, engineID); err != nil { |
There was a problem hiding this comment.
Why not do this in OpenEngine and resetEngine
There was a problem hiding this comment.
Sounds very reasonable. Maybe we can do it inside each backend's implementation and remove AllocateTSIfNotExists.
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
|
@kennytm PTAL again since some interface has changed |
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: 0bb61ab
|
What problem does this PR solve?
close #1108
To support duplicate detection for local backend, we need to ensure a same commitTS is used for the same engine to avoid ingesting kv pairs more than once with different commitTS. And we should make every engine use a unique commitTS so that we can detect cross-engine duplicates
Do not merge this pr before #1080 .
What is changed and how it works?
Check List
Tests
Code changes
Side effects
Related changes
Release Note