HADOOP-19536. S3A : Add option for custom S3 tags while writing and deleting S3 objects #7834
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
steveloughran
changed the title
There was a problem hiding this comment.
great start! renamed the PR title so it matches the overall work.
Can you actually use a different branch name than trunk, ideally something with the JIRA in it. This is because github pr to check out a PR uses the branch name, and we all already hava a branch trunk.
Commented on soft-delete. it gets complicated fast in terms of trying to maintain fs metaphor (or even preventing object overwrite, probes for existence etc).
|
|
||
| ### Soft Delete Tags | ||
| ```properties | ||
| fs.s3a.soft.delete.enabled=true |
There was a problem hiding this comment.
is the idea that when enabled, each delete(path) call is remapped to tagging the object for deletion?
is this for recovery or for a performance benefit?
There was a problem hiding this comment.
- Yes the object will be tagged according to the tag given by the user or some default tag for deletion.
- It is for recovery. Users can archive some s3 objects on the basis of tags and recover that in future when they need.
|
|
||
| ## Soft Delete Feature | ||
|
|
||
| The soft delete feature allows you to tag objects instead of permanently deleting them, enabling data retention policies and recovery options. |
There was a problem hiding this comment.
or people use versioned buckets, obviously
| -rm s3a://my-bucket/file-to-archive.txt | ||
| ``` | ||
|
|
||
| ### Future Capabilities (Planned) |
There was a problem hiding this comment.
don't make these commitments as they get complex fast. e.g
- what if a path has nothing but soft deleted files underneath. Does it exist? can I do a non-recursive rm of a directory with nothing but soft delete entries underneath? we'd reject now as a LIST call would say stuff is there, and we don't do a HEAD on each file looking for a soft-delete marker.
- what if I rename a soft-deleted file? does it come back into existence?
- what if I create a file, the header is set to not create if a file is there, but there's a soft deleted entry?
Better to say
While tagged as soft delete, the files are still visible to filesystem operations
such as list and create.
There was a problem hiding this comment.
Got it, I will remove this section/rename this accordingly.
|
@ayush1300 why did you close this? |
|
🎊 +1 overall
This message was automatically generated. |
|
oh yes, I remember. I even asked for it. thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
How was this patch tested?
For code changes: