Large-scale unrecoverable failure of consumer groups.

[edoger]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
1.36.0	2.8.12	1.14-1.19

Configuration

What configuration values are you using for Sarama and Kafka?

k := sarama.NewConfig()
k.Consumer.Offsets.Initial = sarama.OffsetNewest

Logs

When filing an issue please provide logs from Sarama and Kafka if at all
possible. You can set sarama.Logger to a log.Logger to capture Sarama debug
output.

logs: CLICK ME

[2022-09-19T11:14:28.416+08:00] DBG Kafka: kafka: error while consuming *****/24: kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition
[2022-09-19T11:14:28.417+08:00] DBG Kafka: kafka: error while consuming *****/3: kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition
[2022-09-19T11:14:28.417+08:00] DBG Kafka: kafka: error while consuming *****/6: kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition
[2022-09-19T11:14:28.417+08:00] DBG Kafka: kafka: error while consuming *****/18: kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition
[2022-09-19T11:14:28.417+08:00] DBG Kafka: consumergroup/session/test-1663557258425244539-a9819ae8-4030-4529-a3c8-6a5d5067e684/944 heartbeat loop stopped
[2022-09-19T11:14:28.417+08:00] DBG Kafka: consumergroup/session/test-1663557258425244539-a9819ae8-4030-4529-a3c8-6a5d5067e684/944 released
[2022-09-19T11:14:28.417+08:00] DBG Kafka: Initializing new client
[2022-09-19T11:14:28.417+08:00] DBG Kafka: client/metadata fetching metadata for all topics from broker *****:9092
[2022-09-19T11:14:28.417+08:00] DBG Kafka: Connected to broker at *****:9092 (unregistered)
[2022-09-19T11:14:28.418+08:00] DBG Kafka: client/brokers registered new broker #476156 at *****:9092
[2022-09-19T11:14:28.418+08:00] DBG Kafka: client/brokers registered new broker #476157 at *****:9092
[2022-09-19T11:14:28.418+08:00] DBG Kafka: client/brokers registered new broker #476158 at *****:9092
[2022-09-19T11:14:28.418+08:00] DBG Kafka: Successfully initialized new client
[2022-09-19T11:14:28.418+08:00] DBG Kafka: client/metadata fetching metadata for [*****] from broker *****:9092
[2022-09-19T11:14:28.419+08:00] DBG Kafka: client/coordinator requesting coordinator for consumergroup ***** from *****:9092
[2022-09-19T11:14:28.419+08:00] DBG Kafka: client/coordinator coordinator for consumergroup ***** is #476157 (*****:9092)
[2022-09-19T11:14:28.420+08:00] DBG Kafka: Connected to broker at *****:9092 (registered as #476157)

Problem Description

#2252

Has this new feature been tested at scale? At present, our production environment has experienced large-scale consumption stagnation after upgrading to the latest version, and the failure cannot be recovered by restarting the program. The final analysis determined that it was caused by this new feature (not fully backward compatible, heartbeat loop stopped).

[geberl]

With k.Consumer.Offsets.Initial = sarama.OffsetNewest we see the same behavior in our staging environment, but thankfully not in prod.

Sarama	Kafka	Go
1.36.0	2.8.0	1.19

Same versions in both environments. We downgraded Sarama to v1.35.0 and the issue disappeared.

[edoger]

@geberl We found this problem when graying out to 1% nodes. It is recommended to downgrade to version 1.34.1. According to the feedback from our performance testing team, the performance of version 1.35.0 is lower than that of version 1.34.1.

[dnwe]

I pushed out the above PR to restore the default behaviour of invalid offset resetting and made a release 1.37.0 containing it (amongst other things)

[edoger]

@dnwe Thanks! We will do a full test with the new version, I think the problem is solved.

[geberl]

@dnwe thanks for the fix, I can't reproduce the issue any more on my side 🎉