Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ListOffsets loop of failed requests on leader epoch change until timeout happens #4620

Closed
4 of 7 tasks
emasab opened this issue Feb 20, 2024 · 7 comments
Closed
4 of 7 tasks

ListOffsets loop of failed requests on leader epoch change until timeout happens #4620

emasab opened this issue Feb 20, 2024 · 7 comments

Comments

@emasab
Copy link
Contributor

emasab commented Feb 20, 2024 •
edited
Loading

Description

ListOffsets requests done for partitions with no committed offsets can be retried indefinitely if that partition leader epoch has changed, because the buffer is retried without recreating it with the new CurrentLeaderEpoch received from the Metadata refresh call.

How to reproduce

Start consuming partitions that have no committed offset, or seek to the latest offset. A partition leader change should happen that changes the current leader epoch to a value higher than the cached one. The ListOffsets request give a FENCED_LEADER_EPOCH and then it refreshes Metadata, but starts retrying the buffer with the same CurrentLeaderEpoch, leading to a loop of failed requests.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

  • librdkafka version (release number or git tag): 2.1.0+
  • Apache Kafka version: any
  • librdkafka client configuration: any
  • Operating system: any
  • Provide logs (with debug=.. as necessary) from librdkafka
  • Provide broker log excerpts
  • Critical issue
@emasab emasab mentioned this issue Feb 20, 2024
Closed
@emasab emasab added the bug label Feb 20, 2024
@davidblewett
Copy link

@emasab we have several corroborating reports of this bug with v2.3.0. Is the change in #4616 reasonable?

@emasab
Copy link
Contributor Author

emasab commented Mar 5, 2024 •
edited
Loading

@davidblewett no, the two issues seem unrelated. We're fixing this one though

@emasab
Copy link
Contributor Author

emasab commented Apr 10, 2024 •
edited
Loading

While for the other one we're evaluating a fix, this one isn't occurring, as a metadata refresh is requested and the ListOffsets request is migrated to the new leader after a leader epoch update and a new ListOffset request is made.

Some extracted logs when it's returning "Not leader for partition"

%7|1712742491.308|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-query -> offset-wait
%7|1712742491.308|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: ListOffsetsRequest (v7, opv 3) for 1 topic(s) and 1 partition(s)
%7|1712742491.309|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 17)
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received ListOffsetsRequestV7 from 127.0.0.1:39396
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset -1 (leader epoch 2) for END: Broker: Not leader for partition
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending ListOffsetsResponseV7 to 127.0.0.1:39396
%7|1712742491.309|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Received ListOffsetsResponse (v7, 41 bytes, CorrId 17, rtt 0.19ms)
%7|1712742491.309|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: Offset reply for topic test [0] (v3 vs v3)
%7|1712742491.309|REQERR|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: ListOffsetsRequest failed: Broker: Not leader for partition: explicit actions Refresh
%7|1712742491.309|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: OffsetRequest failed: Broker: Not leader for partition (Refresh)
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Requesting metadata for 1/1 topics: ListOffsetsRequest failed: Broker: Not leader for partition
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: Request metadata for 1 topic(s): ListOffsetsRequest failed: Broker: Not leader for partition
%7|1712742491.309|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: Offset reply error for topic test [0] (v3, Broker: Not leader for partition): Refresh
%7|1712742491.309|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: failed to query logical offset END: Broker: Not leader for partition: (re)starting offset query timer for offset END (leader epoch -1)
%7|1712742491.309|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-wait -> offset-query
%7|1712742491.309|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Sent MetadataRequest (v9, 55 bytes @ 0, CorrId 18)
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received MetadataRequestV9 from 127.0.0.1:39396
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending MetadataResponseV9 to 127.0.0.1:39396
%7|1712742491.309|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Received MetadataResponse (v9, 283 bytes, CorrId 18, rtt 0.20ms)
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: ===== Received metadata (for 1 requested topics): ListOffsetsRequest failed: Broker: Not leader for partition =====
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: ClusterId: mockCluster1fd2f8000378, ControllerId: 0
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: 3 brokers, 1 topics
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2:   Broker #0/3: 127.0.0.1:46459 NodeId 1
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2:   Broker #1/3: 127.0.0.1:36427 NodeId 2
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2:   Broker #2/3: 127.0.0.1:34863 NodeId 3
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2:   Topic test with 4 partitions
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 0 Leader 3 Epoch 2
%7|1712742491.310|BROKER|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: leader 1 epoch 1 -> leader 3 epoch 2
%7|1712742491.310|TOPICUPD|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Topic test [0]: migrating from broker 2 to 3 (leader is 3): leader updated
%7|1712742491.310|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: delegate to broker 127.0.0.1:34863/3 (rktp 0x7f4bd0001850, term 0, ref 6)
%7|1712742491.310|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: no longer delegated to broker 127.0.0.1:36427/2
%7|1712742491.310|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: delegating to broker 127.0.0.1:34863/3 for partition with 0 messages (0 bytes) queued
%7|1712742491.310|BRKMIGR|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Migrating topic test [0] 0x7f4bd0001850 from 127.0.0.1:36427/2 to 127.0.0.1:34863/3 (sending PARTITION_LEAVE to 127.0.0.1:36427/2)
%7|1712742491.310|VALIDATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: skipping offset validation in fetch state offset-query
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 1 Leader 1 Epoch 0
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 2 Leader 2 Epoch 0
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 3 Leader 2 Epoch 0
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: 1/1 requested topic(s) seen in metadata
%7|1712742491.310|TOPBRK|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Topic test [0]: leaving broker (0 messages in xmitq, next broker 127.0.0.1:34863/3, rktp 0x7f4bd0001850)
%7|1712742491.310|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: Broadcasting state change
%7|1712742491.310|TOPBRK|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Topic test [0]: joining broker (rktp 0x7f4bd0001850, 0 message(s) queued)
%7|1712742491.310|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
�[36m[<MAIN>                      /  6.001s] 1 test(s) running:�[0m 0104_fetch_from_follower_mock
%7|1712742491.810|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Topic test [0]: timed offset query for offset END (leader epoch -1) in state offset-query
%7|1712742491.810|OFFREQ|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:34863/3: Partition test [0]: querying for logical offset END (opv 3)
%7|1712742491.810|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-query -> offset-wait
%7|1712742491.810|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker changed state INIT -> TRY_CONNECT
%7|1712742491.810|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
%7|1712742491.810|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: broker in state TRY_CONNECT connecting
%7|1712742491.810|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker changed state TRY_CONNECT -> CONNECT
%7|1712742491.810|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
%7|1712742491.810|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Connecting to ipv4#127.0.0.1:34863 (plaintext) with socket 29
%7|1712742491.810|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Connected to ipv4#127.0.0.1:34863
%7|1712742491.810|CONNECTED|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Connected (#1)
%7|1712742491.810|FEATURE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1712742491.810|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1712742491.810|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
%7|1712742491.810|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Sent ApiVersionRequest (v3, 79 bytes @ 0, CorrId 1)
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: New connection from 127.0.0.1:56786
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ApiVersionRequestV3 from 127.0.0.1:56786
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ApiVersionResponseV3 to 127.0.0.1:56786
%7|1712742491.810|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Received ApiVersionResponse (v3, 13 bytes, CorrId 1, rtt 0.11ms)
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
%7|1712742491.810|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Sent ApiVersionRequest (v0, 43 bytes @ 0, CorrId 2)
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ApiVersionRequestV0 from 127.0.0.1:56786
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ApiVersionResponseV0 to 127.0.0.1:56786
%7|1712742491.810|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Received ApiVersionResponse (v0, 114 bytes, CorrId 2, rtt 0.09ms)
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker API support:
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey Produce (0) Versions 0..7
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey Fetch (1) Versions 0..11
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey ListOffsets (2) Versions 0..7
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey Metadata (3) Versions 0..9
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey OffsetCommit (8) Versions 0..8
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey OffsetFetch (9) Versions 0..6
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey FindCoordinator (10) Versions 0..3
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey JoinGroup (11) Versions 0..6
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey Heartbeat (12) Versions 0..5
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey LeaveGroup (13) Versions 0..4
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey SyncGroup (14) Versions 0..4
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey ApiVersion (18) Versions 0..2
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey InitProducerId (22) Versions 0..4
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey OffsetForLeaderEpoch (23) Versions 2..2
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey AddPartitionsToTxn (24) Versions 0..1
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey AddOffsetsToTxn (25) Versions 0..1
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey EndTxn (26) Versions 0..1
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey TxnOffsetCommit (28) Versions 0..3
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature MsgVer1: Produce (2..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature MsgVer1: Fetch (2..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature MsgVer1
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature MsgVer2: Produce (3..3) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature MsgVer2: Fetch (4..4) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature MsgVer2
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ApiVersion: ApiVersion (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature ApiVersion
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerGroupCoordinator: FindCoordinator (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature BrokerGroupCoordinator
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: FindCoordinator (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature BrokerBalancedConsumer
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ThrottleTime: Produce (1..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ThrottleTime: Fetch (1..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature ThrottleTime
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature Sasl: JoinGroup (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature Sasl
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature SaslHandshake: SaslHandshake (0..0) NOT supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Disabling feature SaslHandshake
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature LZ4: FindCoordinator (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature LZ4
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature OffsetTime: ListOffsets (1..1) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature OffsetTime
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature IdempotentProducer: InitProducerId (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature IdempotentProducer
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ZSTD: Produce (7..7) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ZSTD: Fetch (10..10) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature ZSTD
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature SaslAuthReq: SaslHandshake (1..1) NOT supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature SaslAuthReq: SaslAuthenticate (0..1) NOT supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Disabling feature SaslAuthReq
%7|1712742491.810|FEATURE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD
%7|1712742491.810|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker changed state APIVERSION_QUERY -> UP
%7|1712742491.810|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
%7|1712742491.810|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Skipping metadata refresh of 1 topic(s): connected: already being requested
%7|1712742491.810|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: ListOffsetsRequest (v7, opv 3) for 1 topic(s) and 1 partition(s)
%7|1712742491.810|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 3)
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ListOffsetsRequestV7 from 127.0.0.1:56786
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset 1000 (leader epoch 2) for END: Success
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ListOffsetsResponseV7 to 127.0.0.1:56786

Even when we allow to list offsets on the follower with ReplicaId -2 and the error becomes Leader epoch is older than broker epoch the retrying buffer is discarded because outdated and a new request is made to the new leader:

%7|1712743038.657|RETRY|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Moved 1 retry buffer(s) to output queue
%7|1712743038.657|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 20)
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received ListOffsetsRequestV7 from 127.0.0.1:33182
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset -1 (leader epoch 2) for END: Broker: Leader epoch is older than broker epoch
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending ListOffsetsResponseV7 to 127.0.0.1:33182
%7|1712743038.657|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Received ListOffsetsResponse (v7, 41 bytes, CorrId 20, rtt 0.28ms)
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply for topic test [0] (v3 vs v3)
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply error for topic test [0] (v3, Local: Outdated):

Full logs after allowing to ListOffsets on a follower

%7|1712743038.557|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-query -> offset-wait
%7|1712743038.557|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: ListOffsetsRequest (v7, opv 3) for 1 topic(s) and 1 partition(s)
%7|1712743038.557|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 18)
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received ListOffsetsRequestV7 from 127.0.0.1:33182
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset -1 (leader epoch 2) for END: Broker: Leader epoch is older than broker epoch
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending ListOffsetsResponseV7 to 127.0.0.1:33182
%7|1712743038.557|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Received ListOffsetsResponse (v7, 41 bytes, CorrId 18, rtt 0.17ms)
%7|1712743038.557|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply for topic test [0] (v3 vs v3)
%7|1712743038.557|REQERR|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: ListOffsetsRequest failed: Broker: Leader epoch is older than broker epoch: explicit actions Refresh,Retry
%7|1712743038.557|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: OffsetRequest failed: Broker: Leader epoch is older than broker epoch (Refresh,Retry)
%7|1712743038.557|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Requesting metadata for 1/1 topics: ListOffsetsRequest failed: Broker: Leader epoch is older than broker epoch
%7|1712743038.557|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Request metadata for 1 topic(s): ListOffsetsRequest failed: Broker: Leader epoch is older than broker epoch
%7|1712743038.557|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply error for topic test [0] (v3, Local: Operation in progress): Refresh,Retry
%7|1712743038.557|RETRY|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Retrying ListOffsetsRequest (v7, 75 bytes, retry 1/2, prev CorrId 18) in 100ms
%7|1712743038.557|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Sent MetadataRequest (v9, 55 bytes @ 0, CorrId 19)
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received MetadataRequestV9 from 127.0.0.1:33182
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending MetadataResponseV9 to 127.0.0.1:33182
%7|1712743038.558|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Received MetadataResponse (v9, 283 bytes, CorrId 19, rtt 0.11ms)
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: ===== Received metadata (for 1 requested topics): ListOffsetsRequest failed: Broker: Leader epoch is older than broker epoch =====
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: ClusterId: mockCluster1fd4e7000378, ControllerId: 0
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: 3 brokers, 1 topics
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2:   Broker #0/3: 127.0.0.1:42445 NodeId 1
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2:   Broker #1/3: 127.0.0.1:40441 NodeId 2
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2:   Broker #2/3: 127.0.0.1:40643 NodeId 3
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2:   Topic test with 4 partitions
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 0 Leader 3 Epoch 2
%7|1712743038.558|BROKER|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: leader 1 epoch 1 -> leader 3 epoch 2
%7|1712743038.558|TOPICUPD|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Topic test [0]: migrating from broker 2 to 3 (leader is 3): leader updated
%7|1712743038.558|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: delegate to broker 127.0.0.1:40643/3 (rktp 0x7f5390001850, term 0, ref 7)
%7|1712743038.558|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: no longer delegated to broker 127.0.0.1:40441/2
%7|1712743038.558|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: delegating to broker 127.0.0.1:40643/3 for partition with 0 messages (0 bytes) queued
%7|1712743038.558|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: migrating to new broker: (re)starting offset query timer for offset END (leader epoch -1)
%7|1712743038.558|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-wait -> offset-query
%7|1712743038.558|BRKMIGR|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Migrating topic test [0] 0x7f5390001850 from 127.0.0.1:40441/2 to 127.0.0.1:40643/3 (sending PARTITION_LEAVE to 127.0.0.1:40441/2)
%7|1712743038.558|VALIDATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: skipping offset validation in fetch state offset-query
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 1 Leader 1 Epoch 0
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 2 Leader 1 Epoch 0
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 3 Leader 2 Epoch 0
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: 1/1 requested topic(s) seen in metadata
%7|1712743038.558|TOPBRK|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Topic test [0]: leaving broker (0 messages in xmitq, next broker 127.0.0.1:40643/3, rktp 0x7f5390001850)
%7|1712743038.558|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: Broadcasting state change
%7|1712743038.558|TOPBRK|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Topic test [0]: joining broker (rktp 0x7f5390001850, 0 message(s) queued)
%7|1712743038.558|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743038.657|RETRY|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Moved 1 retry buffer(s) to output queue
%7|1712743038.657|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 20)
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received ListOffsetsRequestV7 from 127.0.0.1:33182
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset -1 (leader epoch 2) for END: Broker: Leader epoch is older than broker epoch
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending ListOffsetsResponseV7 to 127.0.0.1:33182
%7|1712743038.657|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Received ListOffsetsResponse (v7, 41 bytes, CorrId 20, rtt 0.28ms)
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply for topic test [0] (v3 vs v3)
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply error for topic test [0] (v3, Local: Outdated): 
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: outdated offset response: offset query timer already scheduled for offset END (leader epoch -1)
�[36m[<MAIN>                      /  8.001s] 1 test(s) running:�[0m 0104_fetch_from_follower_mock
%7|1712743039.058|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Topic test [0]: timed offset query for offset END (leader epoch -1) in state offset-query
%7|1712743039.058|OFFREQ|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40643/3: Partition test [0]: querying for logical offset END (opv 3)
%7|1712743039.058|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-query -> offset-wait
%7|1712743039.059|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker changed state INIT -> TRY_CONNECT
%7|1712743039.059|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743039.059|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: broker in state TRY_CONNECT connecting
%7|1712743039.059|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker changed state TRY_CONNECT -> CONNECT
%7|1712743039.059|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743039.059|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Connecting to ipv4#127.0.0.1:40643 (plaintext) with socket 29
%7|1712743039.059|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Connected to ipv4#127.0.0.1:40643
%7|1712743039.059|CONNECTED|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Connected (#1)
%7|1712743039.059|FEATURE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: New connection from 127.0.0.1:49350
%7|1712743039.059|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1712743039.059|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743039.059|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Sent ApiVersionRequest (v3, 79 bytes @ 0, CorrId 1)
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ApiVersionRequestV3 from 127.0.0.1:49350
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ApiVersionResponseV3 to 127.0.0.1:49350
%7|1712743039.059|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Received ApiVersionResponse (v3, 13 bytes, CorrId 1, rtt 0.17ms)
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
%7|1712743039.059|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Sent ApiVersionRequest (v0, 43 bytes @ 0, CorrId 2)
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ApiVersionRequestV0 from 127.0.0.1:49350
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ApiVersionResponseV0 to 127.0.0.1:49350
%7|1712743039.059|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Received ApiVersionResponse (v0, 114 bytes, CorrId 2, rtt 0.16ms)
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker API support:
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey Produce (0) Versions 0..7
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey Fetch (1) Versions 0..11
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey ListOffsets (2) Versions 0..7
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey Metadata (3) Versions 0..9
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey OffsetCommit (8) Versions 0..8
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey OffsetFetch (9) Versions 0..6
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey FindCoordinator (10) Versions 0..3
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey JoinGroup (11) Versions 0..6
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey Heartbeat (12) Versions 0..5
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey LeaveGroup (13) Versions 0..4
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey SyncGroup (14) Versions 0..4
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey ApiVersion (18) Versions 0..2
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey InitProducerId (22) Versions 0..4
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey OffsetForLeaderEpoch (23) Versions 2..2
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey AddPartitionsToTxn (24) Versions 0..1
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey AddOffsetsToTxn (25) Versions 0..1
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey EndTxn (26) Versions 0..1
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey TxnOffsetCommit (28) Versions 0..3
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature MsgVer1: Produce (2..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature MsgVer1: Fetch (2..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature MsgVer1
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature MsgVer2: Produce (3..3) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature MsgVer2: Fetch (4..4) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature MsgVer2
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ApiVersion: ApiVersion (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature ApiVersion
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerGroupCoordinator: FindCoordinator (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature BrokerGroupCoordinator
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: FindCoordinator (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature BrokerBalancedConsumer
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ThrottleTime: Produce (1..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ThrottleTime: Fetch (1..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature ThrottleTime
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature Sasl: JoinGroup (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature Sasl
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature SaslHandshake: SaslHandshake (0..0) NOT supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Disabling feature SaslHandshake
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature LZ4: FindCoordinator (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature LZ4
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature OffsetTime: ListOffsets (1..1) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature OffsetTime
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature IdempotentProducer: InitProducerId (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature IdempotentProducer
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ZSTD: Produce (7..7) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ZSTD: Fetch (10..10) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature ZSTD
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature SaslAuthReq: SaslHandshake (1..1) NOT supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature SaslAuthReq: SaslAuthenticate (0..1) NOT supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Disabling feature SaslAuthReq
%7|1712743039.059|FEATURE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD
%7|1712743039.059|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker changed state APIVERSION_QUERY -> UP
%7|1712743039.059|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743039.059|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Skipping metadata refresh of 1 topic(s): connected: already being requested
%7|1712743039.059|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: ListOffsetsRequest (v7, opv 3) for 1 topic(s) and 1 partition(s)
%7|1712743039.059|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 3)
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ListOffsetsRequestV7 from 127.0.0.1:49350
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset 1000 (leader epoch 2) for END: Success

@emasab emasab closed this as not planned Won't fix, can't repro, duplicate, stale Apr 10, 2024
@emasab emasab removed the bug label Apr 10, 2024
@nkostoulas
Copy link

@emasab does this only affect consumers using fetch from follower?

For the record we're seeing something similar with regular consumers. A single broker is restarted which causes partition leader changes, with the other 2 brokers that are not restarted witnessing FENCED_LEADER_EPOCH errors. Then our consumers that consume from those partitions are stuck not consuming anything. The only client errors we see are TRANSPORT ones though (from when the broker went down) but also some weird logs:

rdkafka: [2024-08-12T14:17:50Z][some-consumer#consumer-20][PARTCNT][5][thrd:main]: Topic <topic-name> count changed from 10 to 0

<topic-name> [0]: desired partition is no longer available (Local: Unknown partition)

We use confluent-kafka-go/librdkafka 2.5.0.

@emasab
Copy link
Contributor Author

emasab commented Sep 30, 2024 •
edited
Loading

This bug was closed because what was reported was the suspected behaviour, but then from the tests it's shown that it's prevented actually. In the description it's related to the leader epoch change.
There's a bug that causes a similar loop but happens in a different condition, that is when using Fetch From Follower and is fixed in #4616 (continued in #4754 )

@nkostoulas
Copy link

nkostoulas commented Sep 30, 2024 •
edited
Loading

Our (Go) consumers are getting stuck and we are not using Fetch From Follower. We've had to downgrade to 2.3.0.

@emasab
Copy link
Contributor Author

emasab commented Sep 30, 2024 •
edited
Loading

This issue was not based on an actual incident, so please create a different issue with debug logs, while using latest version, so we can check what happened and give it a correct description or link it to a different known bug.

From your description you could be hitting this one, but only with logs we can be sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@davidblewett @emasab @nkostoulas