How to filter out ListObjects by a Pattern or containString

[hantsy]

We want to list objects in a S3 bucket like this,

client
    .listObjects(
        ListObjectsRequest {
            bucket = bucketName
        }
    )

But the ListObjectsRequest only accepts a prefix to match the file name, not provide pattern(regex) or containString like to filter out the expected files.

Using the result contents to filter is bad performance.

client
    .listObjects(
        ListObjectsRequest {
            bucket = bucketName
        }
    )
    .contents?.mapNotNull { it.key }?.filter { it.contains(...) }

Due to S3 Client listObjects can only list 1000 files at most, we can not get all expected files in this approach.

BTW, how to do a pagination here, for example, count with same condition and query the result by page(or offset) and limit.

[lauzadis]

Hi, S3 does not support listing objects with a specific regex pattern, only prefix.

To paginate, you can use listObjectsV2 which has a startAfter request field to specify the last key from the previous request.

[aajtodd]

There is also a paginator for listObjectsV2Paginated that will handle the paging logic for you. See pagination in the developer guide for more info.