Allow for a Graceful Shutdown of the Worker #109
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a new method to the worker requestShutdown that allows the worker to gracefully shutdown all record processors. The graceful shutdown gives the record processors a last chance to checkpoint before they're terminated. To use these new features the record processor must implement IShutdownnotificationaware. Some cleanup to try and make the state transitions of the record processor more clear.
Handle some possible race conditions during the shutdown process. Added more comments clarifying the how shutdown works, and the race conditions it can face.
Added some additional comments clarifying the shutdown check miss.
Updated the log message to include the the count awaiting final shutdown.
pfifer
added a commit
to pfifer/amazon-kinesis-client
that referenced
this pull request
Nov 3, 2016
* General
* Allow disabling shard synchronization at startup.
* Applications can disable shard synchronization at startup. Disabling shard synchronization can application startup times for very large streams.
* [PR awslabs#102](awslabs#102)
* Applications can now request a graceful shutdown, and record processors that implement the IShutdownNotificationAware will be given a chance to checkpoint before being shutdown.
* This adds a [new interface](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/interfaces/v2/IShutdownNotificationAware.java), and a [new method on Worker](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java#L539).
* [PR awslabs#109](awslabs#109)
* Solves [Issue awslabs#79](awslabs#79)
* MultiLangDaemon
* Applications can now use credential provides that accept string parameters.
* [PR awslabs#99](awslabs#99)
* Applications can now use different credentials for each service.
* [PR awslabs#111](awslabs#111)
pfifer
added a commit
that referenced
this pull request
Nov 3, 2016
* General
* Allow disabling shard synchronization at startup.
* Applications can disable shard synchronization at startup. Disabling shard synchronization can application startup times for very large streams.
* [PR #102](#102)
* Applications can now request a graceful shutdown, and record processors that implement the IShutdownNotificationAware will be given a chance to checkpoint before being shutdown.
* This adds a [new interface](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/interfaces/v2/IShutdownNotificationAware.java), and a [new method on Worker](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java#L539).
* [PR #109](#109)
* Solves [Issue #79](#79)
* MultiLangDaemon
* Applications can now use credential provides that accept string parameters.
* [PR #99](#99)
* Applications can now use different credentials for each service.
* [PR #111](#111)
|
What is the difference between the shutdown and shutdownRequested? What is the use case for both of them? Can you provide an example for both? Do we need to call requestShutdown then shutdown or does requestShutdown should shutdown the worker as well? Thanks. @pfifer |
|
@DavidDeCoding The In general I would recommend using |
|
Thanks @pfifer. I thought so too. |
jari-kujansuu
added a commit
to jari-kujansuu/spark
that referenced
this pull request
Mar 6, 2020
…heckpoint - instance doesn't hold the lease for this shard" errors. Spark's KinesisRecordProcessor to implement Kinesis IShutdownNotificationAware interface to be notified before processor will be shutdown so that it can update checkpoint before shutdown. See more details from: - awslabs/amazon-kinesis-client#79 - awslabs/amazon-kinesis-client#109
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a new method to the worker requestShutdown that allows the worker to
gracefully shutdown all record processors. The graceful shutdown gives
the record processors a last chance to checkpoint before they're
terminated.
To use these new features the record processor must implement
IShutdownnotificationaware.
Some cleanup to try and make the state transitions of the record
processor more clear.