Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry list requests for InvalidRegion error responses #1074

Closed
wants to merge 1 commit into from

Conversation

Praveenrajmani
Copy link

S3 responds with AuthorizationHeaderMalformed | InvalidRegion for the requests which have wrong region set. minio-go already retries such requests with the correct region set in the errResponse.Region.
The retry code was buggy which will not do if the bucketName is empty. Few list requests will not have bucketName set. So this fix covers all of it.

To Test:

1 - Vendorize this patch in minio/mc to test the fix
2 - [Hack] Configure mc to use mc config host add s3-west-2-ip http://52.218.204.152 <Access-Key> <Secret-Key> with 52.218.204.152 being the ip for s3.us-west-2.amazonaws.com
3 - Create a bucket under that region mc mb -region us-west-2 s3-west-2-ip/bucketfive
4 - All the ls requests uses the default us-east-1. Which will throw AuthorizationHeaderMalformed in our case since our bucket is in us-west-2 region. Now, if we do mc --debug ls s3-west-2-ip/ we can see the request being retried with the correct region provided in the errResp.Region.

Update metadata.bucketLocation with the correct region from the error response.

Fixes minio/mc#2570
@harshavardhana
Copy link
Member

harshavardhana commented Feb 12, 2019

The region is only associated with a bucket @Praveenrajmani - this fix is not correct. Also using IP instead of S3 is a hypothetical case. No one uses it and it is not recommended. S3 is a global DNS and can point to many IPs.

@Praveenrajmani
Copy link
Author

Closing this PR as per the following discussion with Harsha

  • ListBuckets API is irrespective to regions. Region is meaningless here. It should list all the buckets from all the regions. The ListBuckets() API should honour all the regions.
  • The GetBucketLocation() API will give us the correct region in X-Amz-Bucket-Region. This region can be trusted. If it denies, it gives the correct region in errResponse.Region XML resp.
  • Regions are always bucket-level concepts.
  • We cannot have ip-endpoints ( Say, 52.218.204.152 for s3.us-west-2.amazonaws.com ) s3 endpoints are dns-styled and it may point to many IPs. To put it simple, The Host header cannot be ip in this case.

And mc works in the same way. For eg,

1 - mc config host add s3-west-2 http://s3.us-west-2.amazonaws.com <Access-Key> <Secret-Key>
2 - mc mb -region us-west-2 s3-west-2/bucketsix
3 - Now, if we do mc --debug ls s3-west-2/bucketsix, We can see the requests carrying the actual region us-west-2 returned by the GetBucketLocation API (with the default us-east-1 set in the GBL request ).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants