Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

putObject: Streaming uploads with size -1 should avoid erroring out f… #506

Merged
merged 1 commit into from
Sep 4, 2016

Conversation

harshavardhana
Copy link
Member

@harshavardhana harshavardhana commented Sep 3, 2016

…or missing parts.

size == -1 should be treated specially for putObjectStream() missing parts
shouldn't error out since with '-1' number of parts are dynamic only when
the stream ends we do know that the we have finished the upload.

Fixes - minio/mc#1819

@krishnasrinivas
Copy link
Contributor

size == -1 should be treated specially for putObjectStream() and avoid fetching
partsInfo for previously uploaded parts since '-1' has no guarantee that the
data has no changed. Let the client upload again and not resume for unknown sizes.

but we check if part is already uploaded by checking ETag right? so if a stream is uploaded with size -1 with same data we will lose that optimization, right?

@harshavardhana
Copy link
Member Author

but we check if part is already uploaded by checking ETag right? so if a stream is uploaded with size -1 with same data we will lose that optimization, right?

We can keep the optimization and just that we don't need to error out for '-1' size since the total parts are dynamic in that scenario.

@harshavardhana harshavardhana changed the title putObject: Streaming uploads with size -1 should avoid fetching previ… putObject: Streaming uploads with size -1 should avoid erroring out f… Sep 3, 2016
@@ -190,6 +191,10 @@ func (c Client) putObjectMultipartStream(bucketName, objectName string, reader i
for i := 1; i <= totalPartsCount; i++ {
Copy link
Contributor

@krishnasrinivas krishnasrinivas Sep 3, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will also cause data corruption if previous incomplete upload had more parts than the current upload. you can use:
for i := 1; i <= partNumber; i++ { to avoid that situation and also the if size < 0 { check below will not be needed if you use partNumber.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah i see.

…parts.

partNumber is the final value indicates the total number of parts
uploaded. Use it to generate the final complete multipart list.

Fixes - minio/mc#1819
@krishnasrinivas krishnasrinivas merged commit 40505f5 into minio:master Sep 4, 2016
@harshavardhana harshavardhana deleted the fix-size branch September 4, 2016 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants