Implemented No-overwrite for uploads using cp command #9569
Conversation
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
a66905b to
69ea8ad
Compare
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
41b7423 to
8ad2ec2
Compare
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
e8f7134 to
7985693
Compare
kdaily
left a comment
kdaily
left a comment
There was a problem hiding this comment.
Not quite finished reviewing. Leaving a few high level/basic changes. Will continue a review of the test cases next.
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
35e8e12 to
68fb87d
Compare
aemous
left a comment
aemous
left a comment
There was a problem hiding this comment.
Looks good overall; requested some changes
| + [ | ||
| 'ChecksumType', | ||
| 'MpuObjectSize', | ||
| 'IfNoneMatch', |
There was a problem hiding this comment.
Since we're modifying the vendored s3transfer, we should likely be porting the same changes to boto/s3transfer.
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
2 times, most recently
from
427869f to
274113a
Compare
|
|
||
| class TestRequestParamsMapperNoOverwrite(unittest.TestCase): | ||
| def test_set_no_overwrite_param_when_flag_present(self): | ||
| # Test when no_overwrite flag is present |
There was a problem hiding this comment.
nit: Since the test name describes it well, I don't think the inline comments are needed. I would only add them if there's something not captured in the test name or needs to be called out. Same with below tests.
There was a problem hiding this comment.
In last two test methods, I had to explicitly mention setting the parameter hence comments are still present and for rest it is removed
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
2 times, most recently
from
bf4d8a1 to
0dcab51
Compare
aemous
left a comment
aemous
left a comment
There was a problem hiding this comment.
Looking good overall. Left a few minor comments.
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
c096558 to
e1c4d4a
Compare
aemous
left a comment
aemous
left a comment
There was a problem hiding this comment.
Very minor changes and we should be good to merge 👍
kdaily
left a comment
kdaily
left a comment
There was a problem hiding this comment.
Looks good overall too! Lets clean up the test case names, and remove unneeded docstrings/comments for them if the test name is already descriptive.
If possible, we should move the changes to s3transfer to its own commit.
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
3 times, most recently
from
50a1015 to
c6b558f
Compare
Updated s3Transfer manager generating original version Updated args list updated transfer manager for upload
Removed unwanted comments for test
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
c6b558f to
6345224
Compare
3 participants
Issue : Github Issue #2874
Description of changes:
Customers requested to prevent overwriting of objects. This is done by providing
no-overwriteheader usingifNoneMatchfeature of Amazon S3 which only allows to upload objects that are not present on the bucket. Theno-overwriteheader provides support to multi-part upload as well as works well with othercpcommand headers. Customer can be implemented usingaws s3 cp <source> <destination> --no-overwritewhich allows successful uploads when object with the same name is not present on the destination. However, if the user tries to upload the object which is already present at the target bucket using command line, awarningabout skipping the file will be generated and that object is not allowed to upload to S3 bucket.Testing:
Functional testing is performed to validate the object upload process. This included both scenarios uploading an object when object with same key is already present in the bucket as well as those when the object with same key is not present on bucket. These tests were conducted for both single-part uploads as well as multipart uploads.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.