Broke Kafka replication when requesting ack != -1 and a message larger than 1000000 bytes

[winbatch]

I'm not sure if this is a librdkafka issue or a kafka general one. I did the following steps:

• Changed Kafka broker to have max message size of 4000000
• Ran the rdkafka_performance binary and specified a size of 3000 bytes and an ack of 1. Ran successfully
• Ran the rdkafka_performance binary and specified a size of 2000000 bytes and an ack of 1. Ran successfully
• Ran the rdkafka_performance binary and specified a size of 2000000 and an ack of -1. The binary reports a failure and times out after 5000 ms.
  Any attempt to run the rdkafka_performance binary (regardless of msg size) using an ack of anything other than 1 fails thereafter with a timeout of 5000 ms. The replication appears to be broken hereafter.

Was able to reproduce this at will, using a new topic each time. I wasn't able to reproduce this when using a cluster max message size of 1000000 (the default). So I suspect there's some sort of issue when the msg is > 1000000 but less than the max msg size configured.

Was wondering if you could reproduce this and if so had any thoughts?

[edenhill]

Make sure to configure message.bytes.max equally on both brokers and producers, that means providing .. -X "message.max.bytes=NNNNNNNN" .. command line argument to rdkafka_performance.

In code you would use rd_kafka_conf_set(rk_conf, "message.bytes.max", "NNNNNNNN", ...)

[winbatch]

Ok, but why would all subsequent runs of the binary fail thereafter?

On Friday, January 3, 2014, Magnus Edenhill wrote:

Make sure to configure message.bytes.max equally on both brokers and
producers, that means providing .. -X "message.max.bytes=NNNNNNNN" ..command line argument to rdkafka_performance.

In code you would use rd_kafka_conf_set(rk_conf, "message.bytes.max",
"NNNNNNNN", ...)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/48#issuecomment-31510241
.

[edenhill]

It sounds to me that one or more of your brokers, but not all, have failed in some way.
Thats why required.acks = 1 works - only the receiving broker will ack it, but -1 doesnt - all ISRs will need to ack it.

Did you change message.bytes.max on ALL brokers in your cluster and restart them?
It would be fair to guess that mismatching max message sizes would frustrate a cluster while syncing messages.

[winbatch]

I'll have my colleague check the broker config. However, since new topics
on this cluster are fine until I run the procedure mentioned, I think
all brokers
are at least up and running- seems to be on a per topic basis.

On Friday, January 3, 2014, Magnus Edenhill wrote:

It sounds to me that one or more of your brokers, but not all, have failed
in some way.
Thats why required.acks = 1 works - only the receiving broker will ack it,
but -1 doesnt - all ISRs will need to ack it.

Did you change message.bytes.max on ALL brokers in your cluster and
restart them?
It would be fair to guess that mismatching max message sizes would
frustrate a cluster while syncing messages.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/48#issuecomment-31524132
.

[edenhill]

Any news on this?

[winbatch]

At this point I'd say just close it. Either I did something wrong with the
max msg size locally or there was something else with the broker. will
reopen if I encounter it again.

On Wed, Jan 29, 2014 at 1:00 AM, Magnus Edenhill
notifications@github.comwrote:

Any news on this?

Reply to this email directly or view it on GitHubhttps://github.com//issues/48#issuecomment-33558841
.