PIP-34 New subscription type Key_Shared

[codelipenghui]

There are many aspects to be covered for PIP-34, this is a task list tracker for this PIP

Tasks

• Core implementation PIP-34 Key_Shared subscription core implementation. #4079
• Broker level flag to enable or disable this feature PIP-34 Add flag to enable or disable Key_Shared subscription. #4120
• Integration tests Add messaging integration tests for different subscriptions. #4352
• Non persistent dispatcher Key_Shared subscription on non-persistent topic #4462
• Documentation PIP-34 Key_Shared subscription core implementation. #4079
• Batch message Introduce batch message container framework and support key based batching container #4435
• Redelivery ordering guarantee FIX Key_Shared dispatch ordering of message redelivery. #4406
• Reader support for sticky key range [[java-client] Add support for key hash range reading. #5928]

Future Enhancements

• Policy specify without message key Support specify dispatch policy for the non-key message in Key_Shared subscription mode #6553
• dispatch messages to new consumer after exist consumer ack all messages dispatched Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. #6554
• send hash ring change events to consumers Send Key hash range change events to consumers in Key_Shared subscription. #6555
• Expose key shared consumer selector range in API [Expose key shared consumer selector range in API #5752]

[sijie]

@codelipenghui I added an issue #4169 for the requirements of supporting sticky consumers in Key_Shared subscription.

[sijie]

@codelipenghui I added one item to the key_shared subscription - adding the support for a sticky-range reader. The requirement is coming from Flink connector. We need to read a sub-stream (a range) from a given message id. This should be pretty straightforward to add since a reader is an exclusive non-durable subscription. So we can just seek to the provided message id and filter the messages based on the key/value.

@yjshen can chime in for more requirements for Flink connector.

[david-streamlio]

Is there an ETA for this PIP?

[sijie]

@david-streamlio the core functionalities for this PIP are already done. It has been used in quite bunch of productions. the left items are more or fewer enhancements.

@codelipenghui I think we should consider moving the items that are not done into a separate issue for tracking, close this issue and mark BP-38 as done.

[codelipenghui]

@sijie I have added new issues for the items that are not done. So close this issue now.

[sijie]

@codelipenghui thank you