Use an explicit lock for shutdown instead of the general lock #501
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the Scheduler loses its lease for a shard it will attempt to
shutdown the ShardConsumer processing that shard. When shutting down
the ShardConsumer acquires a lock on
this
and makes the necessarystate changes.
This becomes an issue if the ShardConsumer is currently processing a
batch of records as processing of the records is done under the
general
this
lock.When these two things combine the Scheduler can become stuck waiting
on the record processing to complete.
To fix this the ShardConsumer will now use a specific lock on shutdown
state changes to prevent the Scheduler from becoming blocked.
Allow the shutdown state change future to acquire the lock
When the ShardConsumer is being shutdown we create a future for the
state change originally the future needed to acquire the lock before
attempting to create the future task. This changes it to acquire the
lock while running on another thread, and complete the shutdown then.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.