Cannot produce message with size nearing message.max.bytes

[ozac]

Description

Using default message.max.bytes (1,000,000).
Trying to produce a message with size ranges from 999,918 to 1,000,000 gets into a loop.

Here is a sample output of rdkafka_example.exe:

Waiting for 1
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-PRODUCE: mybroker:9092/0: No more space in current message (0 messages)
Waiting for 1
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
Waiting for 1
LOG-7-PRODUCE: mybroker:9092/0: No more space in current message (0 messages)
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
Waiting for 1
LOG-7-TOPPAR: mybroker:9092/0: mytopic [0] 0+1 msgs
...

Does message.max.bytes refers to the user's message, or to low-level message which contains additional headers?

Producing a message with size 1,000,001 or larger, returns the expected error from broker:
Broker: Message size too large

Thanks!

How to reproduce

Checklist

Please provide the following information:

• librdkafka version (release number or git tag): 0.9.2-pre0 (https://github.com/edenhill/librdkafka/tree/0a54ed3001d73c0a30f6fcf0cff327b3b534f572)
• Apache Kafka version: 0.9.0.0
• librdkafka client configuration:
• Operating system: Windows
• Using the legacy Consumer
• Using the high-level KafkaConsumer
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[edenhill]

Thanks for your report and sorry for the slow response time.
I'll make a test-case for this to find out what is going on.

[ozac]

Hi,
I see that 0.9.5 is ready. Was this issue fixed?

Thanks

[ozac]

Any update?

Thanks

[edenhill]

Sorry, not yet. PRs are welcome!

[xzxxzx401]

Hi, we met the same problem at version v0.11.5.
We set message.max.bytes to 1024, then send a message with lenth 1023(user message, not including kafka headers)
then we find No more space in current MessageSet (0 message(s), 111 bytes) log.
What`s more, we also find in this situation, broker thread seems spinning, making CPU usage to 100% per thread.
It seems that this bug has not fixed?

[edenhill]

The problem is that the final size of the message can't be fully known at the time of produce() (where the message length is checked first):
message headers and framing variations depending on Kafka message version may lead to messages larger than the configured max size.

The workaround for now is to increase message.max.bytes on the producer to allow for slightly larger messages.

[xzxxzx401]

So can we just return this message when second check at making messageSets? just return an error(for example TOO LARGE) to dr_cb to avoid spin, instead of waiting TimeOut.

[edenhill]

Yeah, that sounds like it could work.

[yulifengli]

can you fix the spin loop,cpu 100% usage bug(poll (x,x,0)timeout (0) looping)

[fmstephe]

The workaround for now is to increase message.max.bytes on the producer to allow for slightly larger messages.

It's important to note that this isn't a work-around for the most serious part of this bug. If your message gets very close to the limit we enter an infinite loop.

This is very severe.

Raising the limit may allow messages to go through, but doesn't protect us from the possibility of an infinite loop.

[edenhill]

Feel free to try out the fix on the issue993 branch

It will go into v1.3.0 (q4)