Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARCTIC-1190][AMS] Table expire service should not expire snapshot which optimizing planned with #1193

Merged
merged 9 commits into from
Mar 6, 2023

Conversation

wangtaohz
Copy link
Contributor

@wangtaohz wangtaohz commented Mar 4, 2023

Why are the changes needed?

fix #1190

Brief change log

  • table expire should not expire snapshot for optimizing plan, but there is no need to check the snapshot of Change Store
  • add unit test

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduces a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

wangtaohz and others added 8 commits March 1, 2023 16:12
2.refactor method fetchLatestFlinkCommittedSnapshotTime
change to hiveLocations
remove baseTable == null, changeTable == null
change to baseExcludePaths
add some comment for fetchLatestFlinkCommittedSnapshotTime
import a static variable for flink.max-committed-checkpoint-id
@github-actions github-actions bot added module:ams-server Ams server module module:ams-dashboard Ams dashboard module labels Mar 4, 2023
@codecov
Copy link

codecov bot commented Mar 4, 2023

Codecov Report

Patch coverage: 89.65% and project coverage change: +0.13 🎉

Comparison is base (fdac57e) 26.30% compared to head (b063243) 26.44%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1193      +/-   ##
============================================
+ Coverage     26.30%   26.44%   +0.13%     
- Complexity     3821     3839      +18     
============================================
  Files           486      486              
  Lines         52932    52956      +24     
  Branches       6130     6133       +3     
============================================
+ Hits          13926    14006      +80     
+ Misses        37597    37543      -54     
+ Partials       1409     1407       -2     
Flag Coverage Δ
ams 26.44% <89.65%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...se/arctic/ams/server/optimize/OptimizeService.java 50.34% <ø> (+10.13%) ⬆️
...ic/ams/server/service/impl/TableExpireService.java 70.96% <89.65%> (+7.60%) ⬆️
.../arctic/ams/server/optimize/TableOptimizeItem.java 20.61% <0.00%> (+1.11%) ⬆️
...se/arctic/ams/server/mybatis/Long2TsConvertor.java 41.17% <0.00%> (+11.76%) ⬆️
.../arctic/ams/server/model/TableOptimizeRuntime.java 62.24% <0.00%> (+14.28%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@zhoujinsong zhoujinsong merged commit 702f8c9 into apache:master Mar 6, 2023
@wangtaohz wangtaohz deleted the fix-1190 branch March 6, 2023 05:54
zhoujinsong pushed a commit that referenced this pull request May 31, 2023
…ich optimizing planned with (#1193)

* the last snapshot with flink.max-committed-checkpoint-id should not be expired

* 1.refactor TableExpireServicec for unit test
2.refactor method fetchLatestFlinkCommittedSnapshotTime

* remove the use of traceId
change to hiveLocations
remove baseTable == null, changeTable == null
change to baseExcludePaths
add some comment for fetchLatestFlinkCommittedSnapshotTime
import a static variable for flink.max-committed-checkpoint-id

* table expire should not expire snapshot for optimizing plan

* add some comment for optimizer
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
…ich optimizing planned with (apache#1193)

* the last snapshot with flink.max-committed-checkpoint-id should not be expired

* 1.refactor TableExpireServicec for unit test
2.refactor method fetchLatestFlinkCommittedSnapshotTime

* remove the use of traceId
change to hiveLocations
remove baseTable == null, changeTable == null
change to baseExcludePaths
add some comment for fetchLatestFlinkCommittedSnapshotTime
import a static variable for flink.max-committed-checkpoint-id

* table expire should not expire snapshot for optimizing plan

* add some comment for optimizer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:ams-dashboard Ams dashboard module module:ams-server Ams server module
Projects
None yet
2 participants