bug: discover is querying more records than expected

[pnadolny13]

The limit kwarg thats used was intended to only query a subset of the table but it looks like the tap logic continues to iterate for more batches instead of breaking.

The maximum number of items to evaluate (not necessarily the number of matching items). If DynamoDB processes the number of items up to the limit while processing the results, it stops the operation and returns the matching values up to that point, and a key in LastEvaluatedKey to apply in a subsequent operation, so that you can pick up where you left off. Also, if the processed dataset size exceeds 1 MB before DynamoDB reaches this limit, it stops the operation and returns the matching values up to the limit, and a key in LastEvaluatedKey to apply in a subsequent operation to continue the operation. For more information, see Working with Queries in the Amazon DynamoDB Developer Guide.

Potential Solutions:

• Add an argument to the get_items_iter method

  tap-dynamodb/tap_dynamodb/dynamo.py

  Line 40 in 3d7ccda

  self, table_name: str, scan_kwargs: dict = {"ConsistentRead": True}

  to break after a certain amount of batches. This way we could send in a limit 100 and batch count = 1 to only return 100 rows.
• Its possible theres a cleaner way to pass these arguments into the method, consider other solutions.