Cannot access Alibaba Cloud OSS via AmazonS3

[jiacai2050]

Describe the bug

I use AmazonS3 to access OSS like this:

    AmazonS3Builder::new()
        .with_access_key_id(key_id)
        .with_secret_access_key(key_secret)
        .with_bucket_name(bucket)
        .with_url(endpoint)
        .build()

endpoint is https://oss-cn-hangzhou.aliyuncs.com, this build failed with following error:

Failed to setup analytic engine: OpenObjectStore { source: Generic { store: "S3", source: UrlNotRecognised { url: "https://oss-cn-hangzhou.aliyuncs.com" } } }

This is a demo for using Python to access OSS via S3 client in https://www.alibabacloud.com/help/en/object-storage-service/latest/use-aws-s3-sdk-to-access-oss#section-jmf-a67-hat

import boto3
from botocore.config import Config
endpoint = 'https://oss-cn-hangzhou.aliyuncs.com'
access_key_id = 'OSS_AccessKeyId'
secret_access_key = 'OSS_AccessKeySecret'

s3 = boto3.client(
    's3',
    aws_access_key_id=access_key_id,
    aws_secret_access_key=secret_access_key,
    endpoint_url=endpoint,
    config=Config(s3={"addressing_style": "virtual", "signature_version": 's3v4'}))

I think there is something wrong in current S3 client implementation.

To Reproduce

See above
Expected behavior

Additional context

Build successfully.

[tustvold]

As documented with_url only supports specific URL types - https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_url

Could you try using the lower-level methods such as https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_endpoint

[jiacai2050]

Thanks, with_endpoint works expected.

    AmazonS3Builder::new()
        .with_virtual_hosted_style_request(true)
        // region is not used when virtual_hosted_style is true,
        // but is required, so dummy is used here
        .with_region("dummy")
        .with_access_key_id(key_id)
        .with_secret_access_key(key_secret)
        .with_endpoint(endpoint)
        .with_bucket_name(bucket)
        .with_client_options(cli_opt)
        .with_retry(retry_config)
        .build()

There is only one issue left for me.

See snippet above, region is a required args, but it's not used when virtual_hosted_style is true, so I have to pass a dummy here.

[tustvold]

region is a required args, but it's not used when virtual_hosted_style is true, so I have to pass a dummy here.

The region code is a requirement for AWS request signing - https://docs.aws.amazon.com/general/latest/gr/signing-elements.html

It must be that Alibaba Cloud OSS is agnostic to what the actual region is set to, the docs don't appear to be very clear...

[jiacai2050]

It must be that Alibaba Cloud OSS is agnostic to what the actual region is set to, the docs don't appear to be very clear...

The region is set in endpoint.
https://www.alibabacloud.com/help/en/object-storage-service/latest/regions-and-endpoints#concept-zt4-cvy-5db

[tustvold]

The region as set on the builder is what gets passed to with_aws_sigv4 which in turn uses it to construct the signing elements. This means that what you set on the builder directly impacts the request's signature, irrespective of any endpoint set.

From what you have indicated, it would appear Alibaba Cloud OSS currently is not enforcing that the signature matches the region of the endpoint, but I'm somewhat loathe to assume this will always be the case?

[tustvold]

I'm closing this as I don't believe this is tracking anything anymore. Feel free to reopen if I'm mistaken