Receive failed: Invalid response size 4020574 (0..4000000): increase receive.message.max.bytes

[BartekCipirski]

Description

librdkafka v2.4.0
kafka broker 3.4.1
Debian 12 x86_64
consumer config:
receive.message.max.bytes = 4194304 (4M)
fetch.max.bytes=4194304 - 512
auto.offset.reset = earliest

While running consumer from kafka input with configured auto.offset.reset = earliest I get error and my consumer can not receive messages anymore:
Receive failed: Invalid response size 4020574 (0..4000000): increase receive.message.max.bytes (after 5ms in state UP, 4 identical error(s) suppressed).

I use fluent bit kafka input with buffer_max_size 4M.

Source:
https://github.com/fluent/fluent-bit/blob/master/plugins/in_kafka/in_kafka.c

Message and fetch size configuration code:
https://github.com/fluent/fluent-bit/blob/master/plugins/in_kafka/in_kafka.c#L246

How to reproduce

Just make sure the producer generates messages faster than the consumer consumes

Issue is similar to already closed one:
#1616

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): v2.4.0
• [x ] Apache Kafka version: 3.4.1
• librdkafka client configuration: receive.message.max.bytes = 4194304 (4M) fetch.max.bytes=4194304 - 512 auto.offset.reset = earliest
• Operating system: Debian 12 x86_64
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue