Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto get the bucket region if the region is empty #37

Closed
Tracked by #3
BohuTANG opened this issue Feb 21, 2022 · 4 comments
Closed
Tracked by #3

Auto get the bucket region if the region is empty #37

BohuTANG opened this issue Feb 21, 2022 · 4 comments
Assignees
Labels
enhancement New feature or request

Comments

@BohuTANG
Copy link

For s3, there are 3 styles URL:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html

The last is S3://bucket-name/key-name, the bucket name does not include the AWS Region.

In OpenDAL:

opendal::services::s3::Backend::build()
.endpoint("https://bucket-name.s3.amazonaws.com")
.bucket("bucket-name")
.credential(credential)
.finish()

If we not set the region, it would be better to get it:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html

This is used by databendlabs/databend#3586:

copy into mytable
  from 's3://bucket-name/data/files'
  credentials=(aws_key_id='$AWS_ACCESS_KEY_ID' aws_secret_key='$AWS_SECRET_ACCESS_KEY')
  encryption=(master_key = 'eSxX0jzYfIamtnBKOEOwq80Au6NbSgPH5r4BDDwOaO8=')
  file_format = (type = csv field_delimiter = '|' skip_header = 1);
@BohuTANG BohuTANG added the enhancement New feature or request label Feb 21, 2022
@dantengsky
Copy link

dantengsky commented Feb 21, 2022

@BohuTANG
Copy link
Author

If we do it in the frontend(like Interpreter), we should import the s3 crate and call GetBucketLocation, this make the developer to know what's the dal working(This is my situation right now :D).

Looks better to do it in the dal, what's your thoughts? cc @Xuanwo

@Xuanwo
Copy link
Member

Xuanwo commented Feb 21, 2022

Yep, opendal will add support for this.

@Xuanwo Xuanwo self-assigned this Feb 21, 2022
@Xuanwo Xuanwo mentioned this issue Feb 21, 2022
6 tasks
@BohuTANG
Copy link
Author

Duplicated with #56

@Xuanwo Xuanwo moved this to 📋 Backlog in Xuanwo's Work Sep 15, 2022
@Xuanwo Xuanwo moved this from 📋 Backlog to 📦 Done in Xuanwo's Work Sep 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants