S3 Bucket Location does not work when using Builder for client building

[margussipria]

I have bucked in eu-west-1, and want to resolve bucket location by api, but It fails when using non deprecated client builder.

DNS queries are changed, when for Frankfurt s3 you get something like this s3-w.eu-central-1.amazonaws.com for Ireland datacenter it is only s3-3-w.amazonaws.com...

So I wanted to implement api request that fetches location (using amazon cli I get expected responce)

{
    "LocationConstraint": "eu-west-1"
}

Response, but when using Java Client's builder, region has to be defined, If you don't have region defined, you get this exception: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region., but when I define default us-east-1 for builder, I get exception The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: XYZ)

Again, I could parse exception message, but that could also change.

For now one could use deprecated client new AmazonS3Client(getCredentialsProvider(bucket), getProxyConfiguration) and it does still work.

[kiiadi]

@margussipria there are a couple of work arounds you can use path-style addressing ie:

AmazonS3 client = AmazonS3Client.builder().withRegion("us-east-1").withPathStyleAccessEnabled(true).build();
client.getBucketLocation("your-eu-west-1-bucket");

Alternatively you can catch the AmazonS3Exception and get the Region from the additionalDetails on the exception :

try {
   client.getBucketLocation("your-eu-west-1-bucket");
} catch (AmazonS3Exception e) {
   e.getAdditionalDetails().get("Region");
}

I agree neither are very pretty - we'll see if we can add one of these hooks to the SDK itself (need to be mindful of backwards compatibility).

[anishkej]

I agree. A fix to make this seamless would be great.

Thanks for the suggestions on how to work around this - we resorted to catching the exception, and getting the region from the message.

[millems]

One of the reasons this may be working on the old client (I'd have to see more detailed code samples to be sure) is because it may actually performing two separate calls. The first to the us-east-1 endpoint, and the second to eu-west-1 based on the exception message.

If you're okay with the two calls and you just want the same behavior as the deprecated client constructor, you can try enabling global bucket access on the builder to see if your problem is fixed:

AmazonS3 client = AmazonS3Client.builder().withRegion("us-east-1").withForceGlobalBucketAccessEnabled(true).build();
client.getBucketLocation("your-eu-west-1-bucket");

[rjstanford]

client.getBucketLocation("your-eu-west-1-bucket");

When I tried to use this approach, I kept getting "US" returned, which despite being in the enum and pointing to US_Standard isn't accepted as a Region when you try to set it on builder.withRegion(). Do you know of any other way to get a usable region back?

Until then we're doing the double-call method described above.

[jansyk13]

@rjstanford @kiiadi IMO correct call would be client.headBucket(new HeadBucketRequest(bucket)).getBucketRegion()
Btw it requires s3:ListBucket permission, see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html for more info.

[debora-ito]

@margussipria @anishkej @rjstanford @jansyk13

The SDK team has reviewed the feature request list for V1, and they decided to not implement this in V1 since they're concentrating efforts on V2 new features. One of the reasons is it would not be possible to change this without making breaking changes.

Please feel free to reach out if you have further questions.