Using Fetch v12 even if broker supports only V4-11

[palmtop]

Read the FAQ first: https://github.com/confluentinc/librdkafka/wiki/FAQ

Do NOT create issues for questions, use the discussion forum: https://github.com/confluentinc/librdkafka/discussions

Description

I was trying to connect to a mapr kafka broker with version 2.6.0 and the client fetched no messages and displayed timeouts approximately after 60 sec. When downgrading to 2.4.0 it worked without any problem.

From the Wireshark captures it can be seen that there was a sucessfull apiversions request and response, where the broker stated that it is supporting V4-11 for the Fetch.

After this negotiation the client was sending an Fetch V12 message, whihc resulted in an exception in the brokers log, and no answer. The client timed out after a while.

After downgrading to 2.4.0 the problem disappeared.

---

Details:

The client configuration includes only the bootstrap.server, group.id and:
'auto.offset.reset': 'earliest', # Start reading at the earliest message
'enable.auto.commit': True, # Auto-commit the message offset

Broker log:

2024-10-30 17:51:22,221 WARN [mapr-kafka-worker-3-22] DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: Invalid version for API key FETCH: 12
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Invalid version for API key FETCH: 12
at org.apache.kafka.common.protocol.ApiKeys.schemaFor(ApiKeys.java:343)
at org.apache.kafka.common.protocol.ApiKeys.requestSchema(ApiKeys.java:313)
at org.apache.kafka.common.protocol.ApiKeys.parseRequest(ApiKeys.java:321)
at com.mapr.data.kafka.rpc.netty.RpcMsgDecoder.decode(RpcMsgDecoder.java:78)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
... 17 more

How to reproduce

Try to connect to and mapr 7.8 and the issue will persist.

Or

Generate a apiversions answer with Fetch enabled till v11 and look into which version is the client trying.

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/confluentinc/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 2.6.1
• [Not applicable] Apache Kafka version: -
• librdkafka client configuration: See in description
• Operating system: RHEL 9.4
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts: See in description
• Critical issue

[emasab]

Closing as duplicate of #4870