Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Object Store (AWS): Support dynamically resolving S3 bucket region #4188

Merged
merged 4 commits into from
May 10, 2023
Merged

Object Store (AWS): Support dynamically resolving S3 bucket region #4188

merged 4 commits into from
May 10, 2023

Conversation

mr-brobot
Copy link
Contributor

@mr-brobot mr-brobot commented May 9, 2023

Which issue does this PR close?

Closes #3213.

Rationale for this change

See #3213.

What changes are included in this PR?

Are there any user-facing changes?

Added a new aws::resolve_bucket_region function that uses the HeadBucket API to get the bucket region.

@github-actions github-actions bot added the object-store Object Store Interface label May 9, 2023
@mr-brobot mr-brobot changed the title feat(object_store): resolve aws region using bucket name Object store (AWS): Support dynamically resolving S3 bucket region May 9, 2023
@mr-brobot mr-brobot changed the title Object store (AWS): Support dynamically resolving S3 bucket region Object Store (AWS): Support dynamically resolving S3 bucket region May 9, 2023
@tustvold
Copy link
Contributor

tustvold commented May 9, 2023

Really nice find that it returns the header even on permission error 👍

Given this doesn't use anything on the builder, what do you think about making this a free function that takes a bucket name and &ClientOptions and returns the region? This would keep the builder signature slightly more consistent? Maybe something like resolve_bucket_region?

@mr-brobot
Copy link
Contributor Author

@tustvold I like that suggestion. Implemented as aws::resolve_bucket_region. ✅

I'd like your feedback on the tests too... They hit the S3 HeadBucket API using some bucket names that work today, but that might not be true forever (someone could create a bucket named "please-dont-exist").

This was to show it working in PR, but maybe better to convert to unit tests by creating a private head_bucket function with different implementations for test/live?

@tustvold
Copy link
Contributor

but that might not be true forever

I think this is fine, we can always update the tests should this change. I don't think it is worth the added complexity to mock it out, you would end up with more test logic than actual logic 😆

@tustvold
Copy link
Contributor

I took the liberty of pushing 5c5fc07 to clean the error handling up a little, prior to getting this in

@tustvold tustvold merged commit 016e7a0 into apache:master May 10, 2023
@tustvold
Copy link
Contributor

Thank you for working on this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
object-store Object Store Interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Object store (AWS): Support dynamically resolving S3 bucket
2 participants