Client library and dynamoDB configuration

[FlorianRuen]

Hello there,

I'm developping a consumer that take data from a Kinesis Data Stream, process them. But to do that, I need to save the position in the stream, so I need to use a DynamoDB Table to do this.

For the past, I don't set any dynamoDB configuration when creating my Kinesis Client Lib, but I saw the result on my bill, beacause the default read and write capacity is 10 (it's very important for a default value !!)

To optimize this bill, I want to set it onDemand because the max read/write I use is 1 or 2 (it depends), but I didn't see any configuration to set the dynamoDB Configuration to onDemand (we can set the InitialLeaseTableWriteCapacity or ReadCapacity)

Anybody facing the same problem ? Maybe a solution exist or I need to do this every time from the console ?

Thanks for help,

Regards,
Florian

[bcarpio]

Yes, we want to set the dynamodb config in terraform and then pass the name to the client library vs having it set for us by the client.

[FlorianRuen]

There is a parameter which is .withDynamoDbConfiguration() but we can't set many parameters inside (only change the read and write capacity, but we can't change the onDemand to fixed capacity)

[FlorianRuen]

Up, still no answers on this subject ? I can see it's recurrent subject in issue section

[svroonland]

@FlorianRuen have you tried BillingMode.PAY_PER_REQUEST?

eg

new Scheduler(
 configsBuilder.checkpointConfig(),
 configsBuilder.coordinatorConfig(),
 configsBuilder.leaseManagementConfig().billingMode(BillingMode.PAY_PER_REQUEST),
 configsBuilder.lifecycleConfig(),
 configsBuilder.metricsConfig(),
 configsBuilder.processorConfig(),         
...

[FlorianRuen]

@svroonland I didn't use Scheduler in my project

`
            val kinesisClientLibConfiguration = KinesisClientLibConfiguration(
                    streamName,
                    streamName,
                    awsService.generateCredentials(),
                    workerId
            )

            kinesisClientLibConfiguration
                    .withRegionName(kinesisProperties.regionName)
                    .withInitialPositionInStream(INITIAL_POSITION_IN_STREAM)
                    .withInitialLeaseTableReadCapacity(1)
                    .withInitialLeaseTableWriteCapacity(1)
`

And if I create a custom Dynamo DB CLient I can"t pass it to Kinesis Client Lib Configuration

[vdesaintpern]

It looks like it's part of 1.13.1 - can you confirm it's what you're looking for ?

https://github.com/awslabs/amazon-kinesis-client/pull/656/files/bd281853a05dfd8a443f7840e7692117eea6541b#diff-e8bd8b124decd36c394cd3622aa97861eb141417600ab337016ea24afb65447a

#582

[FlorianRuen]

Just to complete the case, we had e-mails exchange with @vdesaintpern, and it's not related to the link in his message.

To explain the link between the 1.13.1 (mentionned by @vdesaintpern) and my issue :

• I'm not using KCL directly, but i'm using Kinesis Data Stream. But to store the current position, Data Stream use KCL. So in my code, I'm just creating a Data Stream client, and I can't add the .withBillingMode(BillingMode.PAY_PER_REQUEST) because this is not available on Data Stream client configuration

• Today I've a workaround using

.withInitialLeaseTableReadCapacity(1) .withInitialLeaseTableWriteCapacity(1)

It's more efficient that the default value for write / read shard (which is around 5 or 10, I don't remember, but just to store data stream position, the associate cost is very high)

In conclusion, this case is still opened, because I don't have a solution for this, but just a workaround