Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ARCTIC-1150] Minor optimizing with many small files and TransactionId not in order #1151

Merged
merged 10 commits into from
Feb 27, 2023
Merged

[ARCTIC-1150] Minor optimizing with many small files and TransactionId not in order #1151

merged 10 commits into from
Feb 27, 2023

Conversation

wangtaohz
Copy link
Contributor

@wangtaohz wangtaohz commented Feb 23, 2023 •
edited
Loading

Why are the changes needed?

fix #1150

Brief change log

  • ChangeTableIncrementalScan support toSequence, corresponding to fromSequence
  • Minor optimizing use formSequence and toSequence to limit the file cnt it scans
  • Minor optimizing scan all change files grouped by snapshot and calculate the max sequence for the toSequence

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduces a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added module:ams-server Ams server module module:core Core module module:ams-dashboard Ams dashboard module labels Feb 23, 2023
@wangtaohz wangtaohz merged commit ce76de9 into apache:master Feb 27, 2023
@wangtaohz wangtaohz deleted the fix-1150 branch February 27, 2023 11:23
This was referenced Apr 21, 2023
Open
Closed
zhoujinsong pushed a commit that referenced this pull request May 31, 2023
@wangtaohz
[ARCTIC-1150] Minor optimizing with many small files and TransactionI…
627d990 
…d not in order (#1151)

* minor optimize plan many change files with transaction id not in order
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
@wangtaohz @ShawHee
[ARCTIC-1150] Minor optimizing with many small files and TransactionI…
b70535b 
…d not in order (apache#1151)

* minor optimize plan many change files with transaction id not in order
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhoujinsong zhoujinsong zhoujinsong approved these changes

Assignees
No one assigned
Labels
module:ams-dashboard Ams dashboard module module:ams-server Ams server module module:core Core module
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: Minor optimizing breaks data consistency with many small files and TransactionId not in order
2 participants
@wangtaohz @zhoujinsong