Presigned Multipart Upload Support

[timodwhit]

Describe the issue with documentation

Is it possible to use a backend service to generate a signature for the S3Client? All of the current example show that the S3Client is constructed with a stored secret key or using Cognito.

I am looking to use the PHP SDK to create and return an S3 client without any secrets. It would contain the policy data, public key and v4 signature, similar to how the POST functionality works. I would like to use this client to create a multipart upload.

Is this possible?

[AllanZhengYP]

Hi @timodwhit, is the presigned post what you are looking for?

[timodwhit]

@AllanZhengYP Is it possible to do a multipart upload with the presigned post? My understanding is that if I go that route, then I have to write all the parts for initiating the multipart, breaking the file into parts and then tracking the upload of the parts.

Ideally, I would be able to generate the signature for the s3client and then pass that to lib-storage for the upload functionality.

[AllanZhengYP]

@timodwhit

With the presigned post, you can generate the S3 post fields with policy, keys, etc. on the backend. So, on the frontend, you can post to upload the data to S3 without an SDK client. I can find how to do it in the README.

Is there any feature you rely on with lib-storage? Normally presigned post should be able to meet your need.

[timodwhit]

Thanks @AllanZhengYP, this is currently what we are doing with the JS found here: https://git.drupalcode.org/project/s3fs_cors/-/blob/8.x-1.x/s3fs_cors.js#L197 and the php-sdk to generate the fields for the post.

The hope was twofold:

1. simplify the post with the lib-storage upload
2. More importantly, allow for more than 5gb upload (and all the other benefits, over the presigned-post method)

I know that the presigned method allows for multipart uploads but it sounds like the breaking the file into parts, handling the failures, signaling the multipart upload etc is all left to the user, whereas the lib-storage upload sounds like it handles all those parts, is that correct?

[AllanZhengYP]

Hi @timodwhit,

Your analysis is very to-the-point. The lib-storage does offer better usability than any of presigned url solution. Unfortunately V3 SDK doesn't support this feature. We need to do some research to tell whether this can be supported on client side at all.
If you know there are any existing solutions, we are happy to learn.

I labelled this issue as a feature request so it will be weighed for the next features we are going to do in JS v3.

[timodwhit]

@AllanZhengYP Here is an example post that describes the effort using Uppy.js: https://community.transloadit.com/t/uppy-aws-s3-pre-signed-url-nodejs-complete-example-including-metadata-and-tags/15137

Here is a support example: https://github.com/transloadit/uppy/tree/master/examples/aws-presigned-url

[github-actions]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.