Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARCTIC-1167][core][hive] fix Trash for restoring deleted files #1223

Merged
merged 6 commits into from
Mar 15, 2023

Conversation

wangtaohz
Copy link
Contributor

@wangtaohz wangtaohz commented Mar 14, 2023

Why are the changes needed?

fix #1167, add some hot-fix

Brief change log

  • make TableTrashManager serializable
  • fix the ArcticHadoopFileIO cast error for hive sync
  • moving the file to Trash should overwrite the file that already exists
  • remove useless ArcticFileIoDummy

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduces a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

wangtaohz and others added 5 commits March 14, 2023 21:23
…part of partitions at a time (apache#1220)

* support partition ordered by PartitionWeight for OptimizePlan

* if not all partitions are optimized, current change snapshot id should set to -1
@codecov
Copy link

codecov bot commented Mar 14, 2023

Codecov Report

Patch coverage: 50.00% and project coverage change: -23.80 ⚠️

Comparison is base (8162c6d) 52.69% compared to head (318dd02) 28.90%.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #1223       +/-   ##
=============================================
- Coverage     52.69%   28.90%   -23.80%     
- Complexity      523     5158     +4635     
=============================================
  Files            43      663      +620     
  Lines          3704    69592    +65888     
  Branches        354     8021     +7667     
=============================================
+ Hits           1952    20114    +18162     
- Misses         1622    47542    +45920     
- Partials        130     1936     +1806     
Flag Coverage Δ
core 27.56% <50.00%> (?)
trino 52.78% <ø> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...java/com/netease/arctic/io/ArcticHadoopFileIO.java 58.42% <ø> (ø)
...com/netease/arctic/io/RecoverableArcticFileIO.java 60.86% <0.00%> (ø)
.../com/netease/arctic/io/BasicTableTrashManager.java 88.31% <100.00%> (ø)
...etease/arctic/hive/utils/HiveMetaSynchronizer.java 79.10% <100.00%> (ø)

... and 617 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@zhoujinsong zhoujinsong merged commit 9b73beb into apache:master Mar 15, 2023
@wangtaohz wangtaohz deleted the fix-1167-1 branch March 15, 2023 06:33
wangtaohz added a commit to wangtaohz/amoro that referenced this pull request Mar 15, 2023
zhoujinsong pushed a commit that referenced this pull request Mar 20, 2023
* cherry pick [ARCTIC-1167][core] Introduce `Trash` for restoring deleted files (#1169)
cherry pick [ARCTIC-1016] Moving files will fail if the target parent directory doesn't exist in HDFS #1046

* cherry pick [ARCTIC-1167][core][hive] fix Trash for restoring deleted files (#1223)

* add getSyncFileInfoCachePool back for ThreadPool

* [ARCTIC-1167][core][hive] fix Unkeyed Table location and metadata file pattern for `Trash` (#1229)
zhoujinsong pushed a commit that referenced this pull request May 31, 2023
* fix ArcticHadoopFileIO cast error

* overwrite file in trash when move

* [ARCTIC-1213] Optimizing of Mixed Format Table supports optimizing a part of partitions at a time (#1220)

* support partition ordered by PartitionWeight for OptimizePlan

* if not all partitions are optimized, current change snapshot id should set to -1

* fix checkstyle

* TableTrashManager should extends Serializable
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
…he#1223)

* fix ArcticHadoopFileIO cast error

* overwrite file in trash when move

* [ARCTIC-1213] Optimizing of Mixed Format Table supports optimizing a part of partitions at a time (apache#1220)

* support partition ordered by PartitionWeight for OptimizePlan

* if not all partitions are optimized, current change snapshot id should set to -1

* fix checkstyle

* TableTrashManager should extends Serializable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature]: Support move files to Trash when deleting files and restore files from Trash later
2 participants