Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

service/s3/s3manager: Should be able to track upload progress #279

Closed
avitex opened this issue Jun 12, 2015 · 14 comments
Closed

service/s3/s3manager: Should be able to track upload progress #279

avitex opened this issue Jun 12, 2015 · 14 comments
Labels
feature-request A feature should be added or improved.

Comments

@avitex
Copy link

avitex commented Jun 12, 2015

I think a useful feature would be the ability to watch, for example, the progress of an upload to a s3 bucket.
@avitex avitex changed the title Watching s3 upload progress Feature Request: Watching s3 upload progress Jun 12, 2015
@jasdel jasdel added enhancement response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Jun 12, 2015
@jasdel
Copy link
Contributor

jasdel commented Jun 12, 2015

Hi @avitex What method of watching for S3 uploads were you thinking? Were you looking to monitor uploads via the s3manager.Uploader or just S3's PutObject?

@mischief
Copy link

you can already do this if you make your own io.Reader that tracks bytes read.

@avitex
Copy link
Author

avitex commented Jun 18, 2015

@jasdel Uploads via the s3manger.Uploader.
@mischief I had already tried this, but seemed to 'upload' instantly, perhaps my code was bugged?

@jasdel
Copy link
Contributor

jasdel commented Jun 24, 2015

@avitex an io.Reader will only allow you to monitor when a part is prepared for upload to S3, not the upload progress it self. I think this feature request would require the s3manager.Upload to expose some functionality to track the when individual parts have completed their upload, or some kind of overall progress counter.

@jasdel jasdel added feature-request A feature should be added or improved. and removed response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. enhancement labels Aug 5, 2015
@benhinchley
Copy link

I have just been looking around about how to do this nicely, If I find anything I will post it here, but this would definitely be a nice feature to have

@ghost
Copy link

ghost commented Jun 7, 2016

Is there any update on this, we have tried to implement such a feature but its impossible with the current upload manager.

@jasdel
Copy link
Contributor

jasdel commented Jun 7, 2016

Hi @minervadata, thanks for your support of this issue. This issue is still in our backlog, and would like to implement this feature, but have not started work on it yet. We're also always more than glad to review PRs if anyone is looking to add this feature to the s3 manager. In addition we can discuss the design of the feature here too, to help the implementation.

@flavioribeiro flavioribeiro mentioned this issue Jul 11, 2016
Open
@jasdel jasdel changed the title Feature Request: Watching s3 upload progress service/s3/s3manager: Should be able to track upload progress Apr 12, 2017
@krishnakhandagale
Copy link

Is upload progress feature implemented yet?

@jasdel
Copy link
Contributor

jasdel commented Aug 17, 2017

Hi @krishnakhandagale thanks for voting for support for this feature. This work is in our backlog and hand not be started yet. We're glad to review PRs that look at adding progress this issue though. It would be good to have a design chat prior to a PR to hash out how progress could be exposed.

@oleynikd
Copy link

oleynikd commented Oct 5, 2017

No workaround for this?

@maccam912
Copy link

In my fork I ended up just adding in a progress bar (https://github.com/cheggaaa/pb) and as each part in a multipart upload is completed it adds that progess to the bar. It's not perfect, but it works for me. Maybe a good starting point?

It just adds another channel for each uploader to send back the number of bytes uploaded and each time something is put in that channel the progress bar updates.

https://github.com/maccam912/aws-sdk-go/blob/master/service/s3/s3manager/upload.go#L712

https://github.com/maccam912/aws-sdk-go/blob/master/service/s3/s3manager/upload.go#L673

@xinst
Copy link
Contributor

xinst commented Mar 28, 2018

I have just create a pull request, and add an example for upload with progress
#1868

@diehlaws
Copy link
Contributor

Example added in 50ba1df - closing issue.

@dshearer
Copy link

This would be quite useful for me too.

skotambkar pushed a commit to skotambkar/aws-sdk-go that referenced this issue May 20, 2021
@jasdel
aws: Add default HTTP client instead of http.DefaultClient/Transport (a…
a357131 
…ws#315)

Adds a new BuildableHTTPClient type to the SDK's aws package. The type
uses the builder pattern with immutable changes. Modifications to the
buildable client create copies of the client.

Adds a HTTPClient interface to the aws package that the SDK will use as
an abstraction over the specific HTTP client implementation. The SDK
will default to the BuildableHTTPClient, but a *http.Client can be also
provided for custom configuration.

When the SDK's aws.Config.HTTPClient value is a BuildableHTTPClient the
SDK will be able to use API client specific request timeout options.

Fix aws#279
Fix aws#269
skotambkar pushed a commit to skotambkar/aws-sdk-go that referenced this issue May 20, 2021
@jasdel
Release v0.10.0 (2019-07-24) (aws#344)
5208fda 
Services
===
* Synced the V2 SDK with latest AWS service API definitions.
* Fixes [aws#341](aws/aws-sdk-go-v2#341)
* Fixes [aws#342](aws/aws-sdk-go-v2#342)

SDK Breaking Changes
===
* `aws`: Add default HTTP client instead of http.DefaultClient/Transport ([aws#315](aws/aws-sdk-go-v2#315))
  * Adds a new BuildableHTTPClient type to the SDK's aws package. The type uses the builder pattern with immutable changes. Modifications to the buildable client create copies of the client.  Adds a HTTPClient interface to the aws package that the SDK will use as an abstraction over the specific HTTP client implementation. The SDK will default to the BuildableHTTPClient, but a *http.Client can be also provided for custom configuration.  When the SDK's aws.Config.HTTPClient value is a BuildableHTTPClient the SDK will be able to use API client specific request timeout options.
  * Fixes [aws#279](aws/aws-sdk-go-v2#279)
  * Fixes [aws#269](aws/aws-sdk-go-v2#269)

SDK Enhancements
===
* `service/s3/s3manager`: Update S3 Upload Multipart location ([aws#324](aws/aws-sdk-go-v2#324))
  * Updates the Location returned value of S3 Upload's Multipart UploadOutput type to be consistent with single part upload URL. This update also brings the multipart upload Location inline with the S3 object URLs created by the SDK.
  * Fixes [aws#323](aws/aws-sdk-go-v2#323)
  * V2 Port [aws#2453](aws#2453)

SDK Bugs
===
* `private/model`: Handles empty map vs unset map behavior in send request ([aws#337](aws/aws-sdk-go-v2#337))
  * Updated shape marshal model to handle the empty map vs nil map behavior. Adding a test case to assert behavior when a user sends an empty map vs nil map.
  * Fix [aws#332](aws/aws-sdk-go-v2#332)
* `service/rds`: Fix presign URL for same region ([aws#331](aws/aws-sdk-go-v2#331))
  * Fixes RDS no-autopresign URL for same region issue for aws-sdk-go-v2. Solves the issue by making sure that the presigned URLs are not created, when the source and destination regions are the same. Added and updated the tests accordingly.
  * Fix [aws#271](aws/aws-sdk-go-v2#271)
* `private/protocola/json/jsonutil`: Fix Unmarshal map[string]bool ([aws#320](aws/aws-sdk-go-v2#320))
  * Fixes the JSON unmarshaling of maps of bools. The unmarshal case was missing the condition for bool value, in addition the bool pointer.
  * Fix [aws#319](aws/aws-sdk-go-v2#319)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-request A feature should be added or improved.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@maccam912 @jasdel @mischief @oleynikd @avitex @dshearer @benhinchley @krishnakhandagale @xinst @diehlaws