[SPARK-10969] [Streaming] [Kinesis] Allow specifying separate credentials for Kinesis and DynamoDB #11215
Conversation
…redentials to KinesisClientLibConfiguration
…leanup tables created in separate accounts
|
Can one of the admins verify this patch? |
| |Credentialpool tests verify the ability to uses Kinesis stream, Dynamo DB and CloudWatch Metrics in | ||
| |separate AWS accounts. This requires a separate set of AWS credentials to be stored in system with the profile | ||
| |names "dynamoDB" and "cloudWatch" along with the default credentials. The test assumes that the Kinesis stream | ||
| |in the In absence of this only the underlying API will be tested. |
There was a problem hiding this comment.
This sentence looks like it got mangled during editing somehow.
|
I like the idea of being able to provide separate credentials for services, however it seems to me like a more-common use case is still going to be providing a single set of credentials for all services. It would be nice if the way to do so was still as straightforward as it is currently. |
|
@soulcutter Thank you very much for your review and sorry for the delay! |
|
Welp I think you're right :)
|
|
Solved with https://issues.apache.org/jira/browse/SPARK-19911 / #17250, see this comment. |
3 participants
KinesisUtils.createStream()allows specifying only one set of AWS credentials that will be used by Amazon Kinesis Client Library (KCL) for accessing Kinesis, DynamoDB and CloudWatch.Motivation
In a scenario where one needs to read from a Kinesis Stream owned by a different AWS account the user usually has minimal rights (i.e. only read from the stream). In this case creating the DynamoDB table in KCL will fail. Instead the DynamoDB table should be created in your own AWS account.
Proposal
This PR adds a new serializable class
AWSCredentialPoolcontaining credentials for Kinesis, DynamoDB and CloudWatch that is passed as a parameter toKinesisUtils.createStream(). These credentials are then passed to the constructor ofKinesisClientLibConfiguration.@tdas @cfregly please review if this PR is OK.
See SPARK-10969
This contribution is @ajayrammohan and my original work and we license the work to the project under the project's open source license.