Another way to generate presigned urls

[Fak3]

Openspending-conductor uses another way to generate presigned urls https://github.com/openspending/os-conductor/blob/fc242f1/conductor/blueprints/datastore/controllers.py#L32

Previously in #187 we switched from s3.generate_presigned_url() to s3.generate_presigned_post() and I could not yet find a way to do streaming file uploads in the client. See issue openknowledge-archive/dpm-py#68

We can try to go the same way as openspending, if it will really allow to stream uploads and have md5 hash at the same time.

Acceptance criteria

• we are able to upload big files and have md hash at the same time

Tasks

• decide which approach better for us
  • os conductor way (use old version of boto)
  • use multipart_upload
• create tests
• refactor code

analyzes

The OS conductor implementation is using old boto library. In the new version they changed the api.
So we can not do

 s3key = connection.bucket.new_key(s3path)
 s3url = s3key.generate_url(
                    config.ACCESS_KEY_EXPIRES_IN, 'PUT',
                    headers=s3headers)

The only two options the new library has are:

generate_presigned_url()
generate_presigned_post()

also see discussions in the issue boto/boto3#934

For Chunked file upload for really big file think multipart_upload is the way to go.

[rufuspollock]

@subhankarb can you review this (but not implement yet).

[pwalsh]

Exactly (and I asked it to be done with that code from the beginning).

[rufuspollock]

@pwalsh I know 😉

[subhankarb]

@Fak3, @pwalsh
I went through the code before. The implementation here is using old boto library. In the new version they changed the api.
So we can not do

 s3key = connection.bucket.new_key(s3path)
 s3url = s3key.generate_url(
                    config.ACCESS_KEY_EXPIRES_IN, 'PUT',
                    headers=s3headers)

The only two options the new library has are:

generate_presigned_url()
generate_presigned_post()

I also discussed with them in the issue boto/boto3#934

For Chunked file upload for really big file I think multipart_upload is the way to go.

[rufuspollock]

@pwalsh @akariv any thoughts? Otherwise planning to close as WONTFIX.