Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3.list_objects_v2 doesn't contain a continuation token #316

Closed
laborg opened this issue Mar 16, 2021 · 1 comment · Fixed by #317
Closed

S3.list_objects_v2 doesn't contain a continuation token #316

laborg opened this issue Mar 16, 2021 · 1 comment · Fixed by #317

Comments

@laborg
Copy link
Contributor

laborg commented Mar 16, 2021

To paginate buckets I wanted to make use of the ContinuationToken but NextContinuationToken it isn't contained in the response from S3 although
"IsTruncated"="true"
is returned.

According to https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html

NextContinuationToken -> (string)
NextContinuationToken is sent when isTruncated is true, which means there are more keys in the bucket that can be listed. The next list requests to Amazon S3 can be continued with this NextContinuationToken . NextContinuationToken is obfuscated and is not a real key

Am I missing something obvious?
@laborg
Copy link
Contributor Author

laborg commented Mar 16, 2021

Okay, I think I now what happens:
list_objects_v2 just appends list-type=2 as a query parameter, which will be escaped by _clean_s3_uri to the non-digestable list-type$3D2. So the problem is that the version selector is not working.

@laborg laborg mentioned this issue Mar 16, 2021
Merged
bors bot added a commit that referenced this issue Mar 17, 2021
@bors @laborg
Merge #317
25398af 
317: S3: really call version 2 in `list_objects_v2` r=mattBrzezinski a=laborg

Fixes #316 by only using S3 escape rules for the path.

- With this I can properly use the continuation token for paginated results.
- I don't know enough about minio to know if the added tests are suitable to detect a regression. 
- URIs.jl has been added as an dependency, but I feel that this will sooner or later be the case anyway.

Co-authored-by: Gerhard Aigner <gerhard.aigner@gmail.com>
@bors bors bot closed this as completed in 2b7758f Mar 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

S3: really call version 2 in list_objects_v2
1 participant
@laborg