-
Notifications
You must be signed in to change notification settings - Fork 467
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Initial FAQ, with hopefully more contributions in the future. (#1202)
- Loading branch information
Showing
1 changed file
with
29 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# Frequently Asked Questions (FAQ) | ||
|
||
## Stream Modality | ||
|
||
Questions related to stream modality (e.g., [MultiStreamTracker][multi-stream-tracker]). | ||
|
||
### What is the impact of transitioning my KCL app from single-stream to multi-stream? | ||
|
||
This answer assumes the [StreamTracker][stream-tracker] implementation is being changed. | ||
From KCL's perspective, there is no modality change by decreasing a multi-stream tracker from `N` streams to `1` stream. | ||
|
||
The DDB `leaseKey`, used to persist metadata including lease checkpoint, has a modality-dependent format: | ||
|
||
| Modality | `leaseKey` Format | | ||
| --- | --- | | ||
| single-stream | `<shardId>` | | ||
| multi-stream | `<accountId>:<streamName>:<streamCreationTimestamp>:<shardId>` | | ||
|
||
Transitioning an app -- either from single- to multi-, or vice versa -- creates a backwards-incompatible expectation on the `leaseKey`. | ||
As a result, a KCL app will be blind to any `leaseKey`, and its checkpoint, that does not match the expected format. | ||
|
||
## Resources | ||
|
||
For additional information, please consider reading: | ||
* https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html | ||
* https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html | ||
|
||
[multi-stream-tracker]: /amazon-kinesis-client/src/main/java/software/amazon/kinesis/processor/MultiStreamTracker.java | ||
[stream-tracker]: /amazon-kinesis-client/src/main/java/software/amazon/kinesis/processor/StreamTracker.java |