Migration to sdk2

[posac]

Hi,

Do you have plan to migrate to aws sdk2 ? :)

Michal

[aggarwal]

We currently don't have any plans to migrate to AWS SDK 2.0 in the next 6 months. SDK 2.0 can run alongside SDK 1.x in a JVM. What is your use case to migrate DynamoDB Streams adapter to 2.0?

[camerondavison]

I would say that the KCL v2.0 has a nicer interface, and if you are using it for other kinesis activity it would be nice to not have to learn both apis.

[adrian-baker]

KCL 1.x is quite buggy, maybe 2.x is less so?

[aggarwal]

KCL 1.x is quite buggy, maybe 2.x is less so?

DynamoDB continues to work on improvements to KCL 1.x and DDB Streams Kinesis adapter, including bug fixes. Kinesis also supports both 1.x and 2.x and continues to work on bug fixes for both versions.

[anoopgarlapati]

As noted in #22 (comment), I have tried migrating KCL to 2.x and during my migration I have encountered that there is no equivalent for com.amazonaws.services.dynamodbv2.streamsadapter.model.RecordAdapter which can transform a KinesisClientRecord object from KCL 2.x. I need this to get the old and new images from the stream.
This was reported by another user in kinesis-client repo awslabs/amazon-kinesis-client#714 and was directed to ask here.
It seems like there is a gap in using KCL 2.x with 1.x of dynamodb-streams-kinesis-adapter so is there a workaround for this?

[aggarwal]

DynamoDB Streams does not support KCL 2.x today, and building this support is a larger effort than implementing 2-3 classes. There is no recommended workaround for this at this time, and we recommend using KCL 1.13.3 with version 1.5.1 of the adapter as of now. We continue to support customers who are using this combination.

[etspaceman]

What is the timeline for updating this library to use V2?

[kyeotic]

Is there any update on this? It seems like this is supposed to be the official path, but support does not exist.

The docs for the V2 DynamoDB Streams say

Amazon DynamoDB Streams provides API actions for accessing streams and processing stream records. To learn more about application development with Streams, see Capturing Table Activity with DynamoDB Streams in the Amazon DynamoDB Developer Guide.

If you follow that link you end up here, where it says

Using the Amazon Kinesis Adapter is the recommended way to consume streams from Amazon DynamoDB.

So the V2 SDK docs lead you back here, where support for v2 is not supported. While a timeline for v2 support would be nice, it seems like the docs should at least warn potential upgraders of the missing support.

[artembilan]

Also stuck on this.
I wonder if after this feature on DynamoDB we don't this adapter any more...

[olbat]

@artembilan I think that the feature you mentioned is not always a good alternative to native DynamoDB Streams

The integration of DynamoDB with Kinesis data streams to implement change data capture is absolutely not equivalent to the native implementation of DynamoDB Streams. These are two key differences:

Change data capture for DynamoDB Streams:

DynamoDB Streams helps ensure the following:

• Each stream record appears exactly once in the stream.
• For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

Kinesis Data Streams to capture changes to DynamoDB:

The Kinesis data stream records might appear in a different sequence than the item changes occurred. The same item notifications might also appear more than once in the stream. You can check the ApproximateCreationDateTime attribute to identify the approximate order that the item modifications occurred, and to identify duplicate records. ApproximateCreationDateTime indicates the time of the modification in milliseconds.

Therefore, depending the the use-case, one might not easily be able to trade one for the other (e.g. if the application needs the guarantee that messages appear in order of modification in the change stream).

So imo, support of native DynamoDB Streams in KCL v2 & the stream adapters looks like a rather important feature.

@aggarwal, do we have any update about a timeline for the support of this feature? Thanks in advance 👍

[JHTosas]

AWS SDK v1 will enter maintenance mode in July 31st, 2024, and reach end-of-life on Dec 31st, 2025. I really hope the migration is done before that.
https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/