fix S3 provider priorities #10004
Merged
fix S3 provider priorities #10004
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java
Outdated
Show resolved
Hide resolved
src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java
Outdated
Show resolved
Hide resolved
src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Oliver Bertuch <poikilotherm@users.noreply.github.com>
|
It would be nice to review and merge our new S3 tests... ... before reviewing, testing, and merging this PR. |
pdurbin
changed the title
pdurbin
approved these changes
Nov 28, 2023
Do you still think this is the case? @pdurbin |
|
It's not a hard requirement or anything. |
landreev
reviewed
Dec 5, 2023
| // much but potentially make the failure (in the unlikely case a bucket doesn't | ||
| // exist/just disappeared) happen slightly earlier (here versus at the first | ||
| // file/metadata access). | ||
|
|
|
Looking good. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it: This PR addresses priority issues that existed between the 3 types of credential providers used by S3 stores. Before - the global role-based creds, if set, would override the per store profile or instance credentials. Now they do not, and static creds won't be ignored if/when a default profile exists in the Unix account running Payara.
The PR also includes removal of a redundant check that the bucket exists that would run for every file access.
Which issue(s) this PR closes:
Closes #10003
Special notes for your reviewer: New priority rules are listed in a source comment. As this is a bug fix, it looks to me like the current guides say the right thing about priorities. So I don't see anything to change in the docs (without doing a big rewrite of the whole S3 setup section!). Similarly, since this is a bug fix for a reasonably rare case (role-based access was broken from 4.20 to 5.13, and you have to be using two different S3 providers), I don't know that anything is needed in the release notes.
Suggestions on how to test this:
Regression testing that S3 setup works. Depending on how far you want to go, you could check all the combinations of EC2 role based access is/isn't set, the S3 store has a .profile set or not, (the profile actually exists in the .aws/credentials file or not), and the static microprofile config is/isn't set and verify that setting a profile or static creds wins over the role based method, profile wins over static when both are set, setting nothing results in the default profile in the .aws/credentials file being used, etc. (I have tested at QDR that this fix works in the case where role-based creds exist and we have an S3 store that has to use a non-default profile instead.)
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?:
Additional documentation: