Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge lightning preprocess duplicate detection and post-import conflict detection #51036

Closed
lyzx2001 opened this issue Feb 7, 2024 · 2 comments
Assignees
Labels
affects-8.0 type/enhancement The issue or PR belongs to an enhancement.

Comments

@lyzx2001
Copy link
Contributor

lyzx2001 commented Feb 7, 2024

Enhancement

Merge lightning preprocess duplicate detection and post-import conflict detection

[conflict]
strategy = "none"
  1. none: Do not conduct conflict detection. If the imported data has conflict, checksum phase will fail and return error.
  2. replace: Remove all the rows that have conflicts except the latest ones.
  3. error: Fail and return error if conflicts are detected during import phase.

We remove the old configuration options, including record and remove, which are less useful and confusing for users.
tikv-importer.duplicate-resolutionwill not be used after preprocess duplicate detection and post-import conflict detection are merged.
Add a parameter conflict.precheck-conflict-before-import to control whether to turn on preprocess duplicate detection. The valid value is true and false, and the default is false.

Users should be able to check table lightning_task_info.conflict_error_v2 for all the conflicts, including conflicts from both preprocess duplicate detection and post-import conflict detection. Previously, conflicts from preprocess duplicate detection were stored in table lightning_task_info.conflict_records, so here we need to integrate them into lightning_task_info.conflict_error_v2.

Pre-work: Support replace mode for lightning post-import conflict detection #45774

@lyzx2001 lyzx2001 added the type/enhancement The issue or PR belongs to an enhancement. label Feb 7, 2024
@lyzx2001
Copy link
Contributor Author

lyzx2001 commented Feb 7, 2024

/assign @lyzx2001

ti-chi-bot bot pushed a commit that referenced this issue Feb 19, 2024
ti-chi-bot bot pushed a commit that referenced this issue Mar 15, 2024
@lyzx2001
Copy link
Contributor Author

lyzx2001 commented Apr 2, 2024

Further optimization will be tracked by #52306

@lyzx2001 lyzx2001 closed this as completed Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-8.0 type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

No branches or pull requests

2 participants