[tiered-storage] Add support for AWS instance and role creds

[addisonj]

Motivation

This commit makes changes to the tiered storage support for S3
to allow for support of ec2 metadata instance credentials as well as
additional config options for assuming a role to get credentials.

Currently, because ec2 instance credentials require a session token,
the existing implementation does not let you use credentials provided
by the ec2 metadata API.

Also, the usage of roles can be very helpful, this commit also adds support
for roles.

Modifications

This works by changing the way we provide credentials to use the
funtional Supplier interface and for using the AWS specific
SessionCredentials object for when we detect that the
CredentialProvider is providing credentials that have a session token.

The creation of the AWSCredentialProvider is moved into the TieredStorageConfigurationData object
and two new configuration options are exposed, which are used to provide details of the role name and roleSessionName.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Added a test that ensures we properly get session credentials

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes, aws-sts sdk for support in assuming a role
• The public API: no
• The schema: no
• The default values of configurations: no, but does add new configuration
• The wire protocol: no
• The rest endpoints: no
• The admin cli options: no
• Anything that affects deployment: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? docs, updated reference and the cookbook
• If a feature is not applicable for documentation, explain why?
• If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

[addisonj]

run integration tests

[addisonj]

the integration test passes locally, the failure was in the offloader test though, so it is possible there is some race condition this slightly introduces?

Running them again to see if that is the case...

[addisonj]

I wasn't fully understanding how the integration tests work by needing to bake a new container first, so I wasn't actually testing this changeset! I found the issue, which was that the try/catch that the unit tests relied on were causing the tests to be massively slower. Added a new commit, which just changes this to ensure we have credentials on boot and then changes the tests to include some static credentials.

This could technically be a breaking change if someone in production is using s3/s3-compatible storage without credentials, but that seems somewhat unlikely and is easy worked around by provided some fake static credentials, and this seems like the cleaner choice rather than a warning and not having offloading work as expected.

[addisonj]

okay, was also able to get this deployed to a dev cluster and was able to validate it working with ec2 instance credentials, so assuming I got all the tests fixed, I think this should be good to go

[addisonj]

run java8 tests

[merlimat]

👍