What is the cost for using DynamoDB Stream Kinesis Adapter + Kinesis Client Library to read DDB Stream

[RuiZhang2]

Quote from DDB Pricing for On-Demand Capacity:
DynamoDB charges for reading data from DynamoDB Streams in read request units. Each GetRecords API call is billed as a streams read request unit and returns up to 1 MB of data from DynamoDB Streams. Streams read request units are unique from read requests on your DynamoDB table. You are not charged for GetRecords API calls invoked by AWS Lambda as part of DynamoDB triggers. You also are not charged for GetRecords API calls invoked by DynamoDB global tables.

It seems like 1 GetRecords API = 1 read request unit. Since after switching to adapter and KCL library, there is no GetRecords API call at all. How can we know the number of read units we make? Can you share more details how to calculate the cost on README?

[pingleig]

The adapter still call GetRecords under the hood (see code snippet at the end). There is CloudWatch metrics on GetRecords calls, in console you can find it by click table name and the monitoring tab.

If you want to avoid calling GetRecord you should use https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html in this case, you will use KCL directly without this adapter because you are calling Kinesis API directly. Changes in DynamoDB table are sent to Kinesis by DynamoDB service, though the order guarantee in DDB stream is not preserved in Kinesis stream and the cost model is different.

dynamodb-streams-kinesis-adapter/src/main/java/com/amazonaws/services/dynamodbv2/streamsadapter/AmazonDynamoDBStreamsAdapterClient.java

Lines 436 to 467 in 528d2be

	/**
	* @param getRecordsRequest Container for the necessary parameters to execute the GetRecords service method on DynamoDB Streams.
	* @return The response from the GetRecords service method, adapted for use with the AmazonKinesis model.
	*/
	@Override
	public GetRecordsResult getRecords(GetRecordsRequest getRecordsRequest) {
	if (getRecordsRequest.getLimit() != null && getRecordsRequest.getLimit() > GET_RECORDS_LIMIT) {
	getRecordsRequest.setLimit(GET_RECORDS_LIMIT);
	}
	GetRecordsRequestAdapter requestAdapter = new GetRecordsRequestAdapter(getRecordsRequest);
	requestCache.addEntry(getRecordsRequest, requestAdapter);
	try {
	com.amazonaws.services.dynamodbv2.model.GetRecordsResult result = internalClient.getRecords(requestAdapter);
	List<Record> records = result.getRecords();
	if (records != null && !records.isEmpty()) {
	final int lastIndex = result.getRecords().size() - 1;
	final StreamRecord lastStreamRecord = result.getRecords().get(lastIndex).getDynamodb();
	final double lastApproximateCreationTimestamp = lastStreamRecord.getApproximateCreationDateTime().getTime();
	final double millisBehindLatest = Math.max(System.currentTimeMillis() - lastApproximateCreationTimestamp, 0);
	IMetricsScope scope = MetricsHelper.getMetricsScope();
	scope.addData(MILLIS_BEHIND_LATEST_METRIC, millisBehindLatest, StandardUnit.Milliseconds, MetricsLevel.SUMMARY);
	}
	if (result != null && result.getNextShardIterator() == null && result.getSdkResponseMetadata() != null) {
	LOG.info("RequestId for getRecords which resulted in ShardEnd: " + result.getSdkResponseMetadata().getRequestId());
	}
	return new GetRecordsResultAdapter(result, generateRecordBytes);
	} catch (AmazonServiceException e) {
	throw AmazonServiceExceptionTransformer.transformDynamoDBStreamsToKinesisGetRecords(e, skipRecordsBehavior);
	}
	}

[amdhing]

The adapter makes GetRecords calls. You can identify the number of GetRecords calls made by your consumer applications towards a DynamoDB Stream by checking the SampleCount statistic for SuccessfulRequestLatency CloudWatch metric associated for the DynamoDB Stream.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html#SuccessfulRequestLatency

It may not be possible to identify calls made on a per consumer application level in-case multiple distinct consumer applications are consuming from the same stream.