S3 bucket only returning 1000 files

[braddonakowski]

Describe the bug
We are having an issue where only 1000 files are being retrieved from our S3 bucket. We have around 20,000 images in our bucket. I came across this question on StackoverFlow, https://stackoverflow.com/questions/44070580/not-all-data-only-1000-elements-retrieved-from-s3-using-amazon-sdk, but am unable to find something similar for the flutter platform.

Platform
Amplify Flutter current supports iOS and Android. This issue is reproducible in (check all that apply):
[] Android
[x] iOS

Smartphone (please complete the following information):

• Device: [e.g. ipad air 3]
• OS: [e.g. iOS14.7]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[HuiSF]

Thanks for reporting this issue @braddonakowski .

I did some digging into amplify-android and amplify-ios (dependencies of amplify-flutter). My first impression is the Android code does the exact same thing as what mentioned in the Stackoverflow post you linked, to ensure the list function returns all items in a S3 bucket. However, the iOS implementation seems missing this piece of logic. I will confirm with amplify-ios maintainers.

[HuiSF]

Confirmed that amplify-ios doesn't handle this use case retrieving items from a S3 bucket more than the default limit.

I will create a feature request to amplify-ios repo...

[braddonakowski]

@HuiSF Thank you!

[braddonakowski]

Good morning @HuiSF,

What is an estimate on when this feature will be implemented?

Thank you,
Brad

[HuiSF]

Hello @braddonakowski thanks for asking! amplify-ios maintainers are actively looking into this feature request.
Any update should be posted in aws-amplify/amplify-swift#1286 . We will integrate once it's ready.

[braddonakowski]

Hi @HuiSF,

I noticed that this ticket has been open since June. How long do you think that it will take for this to be an implemented feature? This is preventing us from being able to complete an application due for deployment.

Are there any alternative ways for us to retrieve a list of all of the keys in the s3 bucket, using flutter on ios? Maybe any sort of http request to a rest API? Anything like that?

Thanks

[HuiSF]

Hi @braddonakowski ampify-ios maintainers are still working on this. Will keep you posted.

[cbs9md]

Hi @HuiSF. 2 things.

1. I wanted to know if there were any updates on your end concerning this issue.

2. The reason we need to list files in s3 is to check the existence of a file before downloading. The way we are currently doing this is listing the files, then checking if any of the files correspond with the key of the item we are trying to download. Was also wondering if there is a better way to accomplish this that would circumvent the need to list files in s3.

[HuiSF]

Hey @cbs9md thanks for following up on this issue. amplify-ios maintainers are working on the feature request.

I believe if an object doesn't exist in s3 bucket, downloadFile and getUrl API calls should return corresponding error, which you should be able to catch and react accordingly.

Maybe I missed some context, for the second point what kind of specific use case you want to achieve?

[rabi-palmtree]

2. The reason we need to list files in s3 is to check the existence of a file before downloading. The way we are currently doing this is listing the files, then checking if any of the files correspond with the key of the item we are trying to download. Was also wondering if there is a better way to accomplish this that would circumvent the need to list files in s3.

About 2, is it not easier to do use the available key and getUrl before downloadFile ?
GetUrlResult result = await Amplify.Storage.getUrl(key: fileKey, options: options);

[Jordan-Nelson]

This issue has been resolved in the developer-preview version of amplify flutter (v1.0.0). Pagination should be supported on all platforms in this version.

[Jordan-Nelson]

This issue has been addressed in v1.0.0 of Amplify Flutter, which is now stable. Please see the docs to learn more about pagination in the Storage category.

This release also includes web and desktop support for Auth, API, Analytics, and Storage. You can see the list of new features and bug fixes in the release notes, and see more details on how to migrate in the upgrade guide.