new feature: load endpoint_url from aws config file

[TennyZhuang]

Feature Description

aws config file allows users to customize the endpoint_url from config file or environment.

Problem and Solution

[default]
region = auto
endpoint_url = http://minio.local

The argument will override the endpoint used by AWS CLI. It's preferable for OpenDAL to respect this.

It can also be specified by S3, but we can support it later due to parsing complexity:

[default]
region = auto
services = minio

[services minio]
s3 =
  endpoint_url = http://minio.local

The environment variable AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3 can also override the argument.

Additional Context

OpenDAL appears to use the reqsign crate, owned by @Xuanwo, to parse the config file. Should we add the argument support in that crate?

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.

[TennyZhuang]

It seems that a reasonable approach is OpenDAL should read and parse the config file itself, since there are many configs unrelated to key signing.

[Xuanwo]

It seems that a reasonable approach is OpenDAL should read and parse the config file itself, since there are many configs unrelated to key signing.

I prefer incorporate all those scopes into reqsign. By implementing them in reqsign, users who need to interact with AWS services can reuse reqsign without relying on OpenDAL.

endpoint_url is good to be supported. Other S3-specific settings, such as multipart size, might not be applicable because they are intended for s3cmd, and opendal doesn't offer the same semantic to users.

[Xuanwo]

the reqsign crate, owned by @Xuanwo

This is not good for me. I'm working on donating resign to the ASF instead and will share the updates with the OpenDAL community once it happens.