feat(consumer): support multiple balance strategies #2339
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As shown below, Kafka supports specifying multiple rebalance protocol from the very begining. (source)
However, Sarama only supports one group protocol right now, by specifying
Consumer.Group.Rebalance.Strategy
. This prevents users from enjoying some benefits in Kafka. For example, if users wanna change the default onerange
to a more advanced one likesticky
, when the new consumers withsticky
protocol try to join in the consumer group, they will be rejected by the coordinator with errorErrInconsistentGroupProtocol(kafka server: The provider group protocol type is incompatible with the other members)
and thus cannot consume any messages. Only when all the instances are upgraded withsticky
protocol can they resume consume message again.In contrast, the client supporting multiple group protocols enables rolling upgrades without downtime -- The upgraded member includes both the new protocol and the protocol, and the coordinator will choose a single protocol which all members support. Once all members have upgraded, the coordinator will choose whichever protocol is listed first in the
GroupProtocols
array.In this PR, I introduce a new config
Consumer.Group.Rebalance.GroupStrategies
and mark the old oneConsumer.Group.Rebalance.Strategy
deprecated. To make it backward compatible:Consumer.Group.Rebalance.GroupStrategies
is[]{BalanceStrategyRange}
Consumer.Group.Rebalance.Strategy
takes precedence ofConsumer.Group.Rebalance.GroupStrategies
Thus,
Consumer.Group.Rebalance.Strategy
isn't explicitly configured previously, then the new default valuerange
inConsumer.Group.Rebalance.GroupStrategies
will be used.Consumer.Group.Rebalance.Strategy
is configured by users previously,Consumer.Group.Rebalance.Strategy
is also chosen as the group protocol.I believe it's a desirable feature in sarama, since other Go Kafka Libraries, like segmentio/kafka-go, already support it.