Move s3 path style into s3.begin/s3.end blocks #1601
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It looks like the path style directive is scoped to each s3.begin/s3.end block, which is why the previous declaration failed for multiple buckets. Technically it would succeed if all the buckets were supposed to use virtual paths (the default), but in the path-style case, only the first bucket inherited the value. Other buckets reverted to the default.
This commit scopes the path style to each origin export to guarantee each of the request URLs generated by XRootD follow the same bucket convention.
The easiest way to test this is probably by starting an Origin with full debug logging at the origin/xrootd level and point it at two public AWS buckets. Here's origin config I used:
From there, curl one object from each namespace. Two objects for testing are
You can then search through the origin log for the generated URLs. Prior to this commit, you'd find that the object request for the first listed bucket (in this case noaa-wod-pds) generates the URL
https://s3.us-east-1.amazonaws.com/noaa-wod-pds/MD5SUMS
while the object request for any subsequent buckets reverts to virtual-style, e.g.https://genome-browser.s3.us-east-1.amazonaws.com/htdocs/.welcome.msg
After the change, both of these URLs should have the form
https://s3.us-east-1.amazonaws.com/<bucket>/<object>
Closes #1561