Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Support move files to Trash when deleting files and restore files from Trash later #1167

Closed
2 tasks done
wangtaohz opened this issue Feb 27, 2023 · 0 comments · Fixed by #1169, #1223, #1224 or #1229
Closed
2 tasks done
Labels
type:feature Feature Requests

Comments

@wangtaohz
Copy link
Contributor

Description

We should introduce a new feature to improve table security in the situation of deleting files, like snapshots expiration, and orphan files clean, making it easy to restore files when they are deleted by mistake.

Use case/motivation

When files are deleted by mistake, whether they are metadata files or data files, it will cause serious problems for the iceberg table, like loading table errors or reading/writing errors.

And also, it's very difficult to recover the table from the abnormal state above; sometimes, restoring the files is the only way to recover the table.

Describe the solution

  1. We should introduce a new feature of Trash to restore deleted files
  2. The Trash is an optional config for each Table
  3. When deleting files of a table with Trash enabled, files are supposed to be moved to a specific directory, instead of deleted directly from the file system; the Trash directory is also a config for each Table
  4. We can restore deleted files from Trash when needed
  5. The Trash directory is cleaned periodically with an expiration time by AMS

Get more details from the doc https://docs.google.com/document/d/1CYBb2u9KmhTcBVy-uUn_Lifu3osNnAgSkiD20LQhmko/edit#heading=h.74a2yp1p0du8

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment