Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using multiple existing s3 buckets as storage plugins. #2947

Closed
mazhigbee-pb opened this issue May 10, 2023 · 8 comments
Closed

Using multiple existing s3 buckets as storage plugins. #2947

mazhigbee-pb opened this issue May 10, 2023 · 8 comments
Labels
feature-request Request a new feature storage Issues related to the Storage category

Comments

@mazhigbee-pb
Copy link

Is your feature request related to a problem? Please describe.

I would like to use more than a single S3 bucket in my amplify project.

Describe the solution you'd like

Say I have two buckets, Foo and Bar.

It would be very nice if configuring another bucket was as easy as adding another AWSS3StoragePlugin. If the prefix resolver allowed me to customize the bucket information and security requirements.

I would expect to be able to add as many different buckets as I'd like.

{
  "storage": {
    "plugins": {
      "awsS3StoragePluginFoo": {
        "bucket": "FOO",
        "region": "[REGION]"
      },
      "awsS3StoragePluginBar": {
        "bucket": "BAR",
        "region": "[REGION]"
      }
    }
  }
}
let storagePluginForFooBucket = AWSS3StoragePlugin(configuration: .prefixResolver(FooPrefixResolver()))
let storagePluginForBarBucket = AWSS3StoragePlugin(configuration: .prefixResolver(BarPrefixResolver()))

try Amplify.add(plugin: storagePluginForFooBucket)
try Amplify.add(plugin: storagePluginForBarBucket)

Then when uploading to the bucket if you specify the key (or bucket-name( Amplify would look up or understand which prefix resolver to use when uploading to that bucket.

Amplify.Storage.uploadData(
        key: "Foo",
        data: data
    )

One thing of note: using Amplify.Storage.uploadData does not allow me to specify the "public-read" acl on the bucket. It would be nice if storage provided all the same options that the bucket has configured because Guest, Protected & Private doesn't match all the potentially configured security options.

Describe alternatives you've considered

As a workaround, I used the TransferManager/Utility to upload to various buckets. But this becomes hard to manage when the buckets have authentication requirements. Amplify.Storage.upload handles those requirements automatically but the TransferUtility requires me to include all that information.

AWSS3TransferUtility.default().uploadFile(fileUrl, bucket: bucketName, key: fileName, contentType: contentType...)

Is the feature request related to any of the existing Amplify categories?

Storage

Additional context

No response

@atierian atierian added storage Issues related to the Storage category feature-request Request a new feature labels May 17, 2023
@github-actions
Copy link
Contributor

This has been identified as a feature request. If this feature is important to you, we strongly encourage you to give a 👍 reaction on the request. This helps us prioritize new features most important to you. Thank you!

@atierian
Copy link
Member

Thanks for opening this feature request @mazhigbee-pb.

Can you please include some details on your use case that requires multiple buckets?

In case you weren't aware of this functionality, a custom defined prefix resolver may help you here.
Here's the Amplify iOS v1 documentation on the subject Storage > File Access Levels > Customization.

@mazhigbee-pb
Copy link
Author

@atierian Sorry for the late response!

We have two big features inside our app that have different backend environments, each with its own S3 buckets (inside the same account).

I couldn't figure out how to set up the prefix resolver so that it would work with both buckets. When I made changes in the prefix resolver to add the second bucket prefix whenever I made a request to storage via Amplify, the bucket name would never change to the second one. Both of these buckets have the same permission scope so changing the prefix based on the protection level doesn't solve the issue.

I tried having both buckets defined in my json config, but the resolver never seemed to pickup the second bucket when adding the plugin for storage. It would be great if there was a code example for this in the docs, or maybe this is just not something thats support by Amplify at this time?

@atierian
Copy link
Member

atierian commented Aug 9, 2023

Thanks for following up with more information about your use case.

I should have been more clear about the custom prefix resolver suggestion. This won't allow you to use multiple buckets as you've discovered; instead, it could allow you to use a single bucket with a folder structure to delineate the objects you store in two buckets today. For example:

|-- myexamplebucketfoo
|   ├─ a.json (myexamplebucketfoo/a.json)
|   └─ b.txt  (myexamplebucketfoo/b.txt)
|
└-- myexamplebucketbar
    ├─ c.json (myexamplebucketbar/c.json)
    └─ d.txt  (myexamplebucketbar/d.txt)

becomes:

└-- myexamplebucket
    ├─ foo
    |    ├─ a.json (myexamplebucket/foo/a.json)
    |    └─ b.txt  (myexamplebucket/foo/b.txt)
    |
    └─ bar
         ├─ c.json (myexamplebucket/bar/c.json)
         └─ d.txt (myexamplebucket/bar/d.txt)

@mazhigbee-pb
Copy link
Author

@atierian Any update on this? Sorry for never responding, but unfortunately it's a must that we use two separate buckets. So creating a nested folder structure isn't going to work for me.

@atierian
Copy link
Member

atierian commented Jul 2, 2024

Hey @mazhigbee-pb, thanks for checking in. We are actively working on multi-bucket support, but don't have a timeline to share yet. Stay tuned 😄

@ruisebas
Copy link
Member

Hi @mazhigbee-pb, support for multiple buckets has been released in version 2.40.0

You can learn how to configure it in our documentation for setting up Storage, and also on the specific section for each operation (e.g. for uploading into a specific bucket).

I'm closing this feature requests as completed. If you face any problems please open a new issue reporting it.
Thanks!

Copy link
Contributor

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request Request a new feature storage Issues related to the Storage category
Projects
None yet
Development

No branches or pull requests

3 participants