Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing bug where ShardFilter parameter for ListShards was being passe… #788

Merged
merged 2 commits into from
Feb 19, 2021
Merged

Fixing bug where ShardFilter parameter for ListShards was being passe… #788

merged 2 commits into from
Feb 19, 2021

Conversation

joshua-kim
Copy link
Contributor

@joshua-kim joshua-kim commented Feb 18, 2021

Issue #, if available:

Description of changes:

This resulted in a bug where all calls for ListShards when initializing
the lease table would fail, since Kinesis only requires the NextToken
parameter when making paginated calls.

Testing:

Created a large stream which was enough to cause pagination; tried to initiate a shard sync with an empty lease table (breaks as expected):

software.amazon.awssdk.services.kinesis.model.InvalidArgumentException: NextToken and ShardFilter cannot be provided together. (Service: Kinesis, Status Code: 400, Request ID: d62dd0fc-5122-ac86-8e85-5f676ac531f5, Extended Request ID: rINtS4L3nKaKtVM5VL2QrXKNc8+QCXjz/xM/8tN/gq/PJU/w+TuRDb8PdqbdrSkeP/qBAVFWLFWQuuLfrXnA0ypsFmqMVgpa)

After the change, we see that the lease table is created as expected with all shards.

New logging:


2021-02-18 18:18:20,605 [ShardRecordProcessor-0090] INFO  s.a.k.leases.KinesisShardDetector [NONE] - Stream paginated-stream: listing shards with list shards request ListShardsRequest(StreamName=paginated-stream) 
...
2021-02-18 18:19:57,949 [KinesisTester-0000] INFO  s.a.k.leases.KinesisShardDetector [NONE] - Stream paginated-stream: listing shards with list shards request ListShardsRequest(StreamName=paginated-stream, ShardFilter=ShardFilter(Type=AT_LATEST)) 

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…d in for paginated calls.

This resulted in a bug where all calls for ListShards when initializing
the lease table would fail, since Kinesis only requires the NextToken
parameter when making paginated calls.
if (StringUtils.isEmpty(nextToken)) {
request = request.streamName(streamIdentifier.streamName());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to validate filter is not empty and fail directly with a message if it is ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to, if a filter is null then this is equivalent to an ordinary listShards call since it uses same API but ShardFilter parameter is optional.

@yatins47
Copy link
Contributor

LGTM

@joshua-kim joshua-kim merged commit 5d68e65 into awslabs:master Feb 19, 2021
@joshua-kim joshua-kim deleted the listshards-shardfilter-pagination-bug branch February 19, 2021 02:26
@joshua-kim joshua-kim added this to the v2.3.4 milestone Feb 19, 2021
@joshua-kim joshua-kim added bug v2.x Issues related to the 2.x version labels Feb 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug v2.x Issues related to the 2.x version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants