assertAllParentShardsAreClosed check interacts poorly with high shard count dynamodb streams

[zerth]

This assertion during shard sync can prevent worker initialization for workers processing dynamodb streams associated with tables having large numbers of partitions:

amazon-kinesis-client/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardSyncer.java

Line 117 in 1ec0b65

assertAllParentShardsAreClosed(shardIdToChildShardIdsMap, shardIdToShardMap);

Example error message:

Sep 13, 2017 4:40:31 PM com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask call
SEVERE: Caught exception while sync'ing Kinesis shards and leases
com.amazonaws.services.kinesis.clientlibrary.exceptions.internal.KinesisClientLibIOException: Parent shardId shardId-00000001505307450567-xxxxxxxx is not closed. This can happen due to a race condition between describeStream and a reshard operation.
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.assertAllParentShardsAreClosed(ShardSyncer.java:161)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.syncShardLeases(ShardSyncer.java:117)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.checkAndCreateLeasesForNewShards(ShardSyncer.java:88)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask.call(ShardSyncTask.java:68)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:427)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:356)
    at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.call(MultiLangDaemon.java:111)
    at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.call(MultiLangDaemon.java:58)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

This is with KCL 1.7.5 and dynamodb-streams-kinesis-adapter 1.2.1.

I believe this is caused by pagination of the dynamodb stream description for very large tables taking more than several seconds, by which time a new child shard is likely to have been created (referencing a still-open parent which was seen earlier in the paginated response and thus triggering this assertion).

This commit attempts to fix this against 1.7.5: AdRoll@09dcc99

The attempted fix adds a configuration parameter controlling whether the assertion is made, and also prevents the creation of new leases for such children with still-open parents during shard sync.

[zerth]

PR was merged.