Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setting receive.message.max.bytes (Protocol parse failure) #116

Closed
winbatch opened this issue May 14, 2014 · 22 comments
Closed

Setting receive.message.max.bytes (Protocol parse failure) #116

winbatch opened this issue May 14, 2014 · 22 comments

Comments

@winbatch
Copy link

1400039259.017 RDKAFKA-4-PROTOERR: rdkafka#consumer-0: d146537-025.masked.com:5757/1052189117: Protocol parse failure at rd_kafka_messageset_handle:2637
1400039259.017 RDKAFKA-4-PROTOERR: rdkafka#consumer-0: d146537-025.masked.com:5757/1052189117: expected 18 bytes > 4 remaining bytes
@edenhill
Copy link
Contributor

Can you reproduce this by consuming from that same offset again?

@winbatch
Copy link
Author

I can. Though the expected bytes line seems to change (this time 18 bytes > 1 )

@winbatch
Copy link
Author

BTW, maybe in that error message you can write which partition + offset?

@edenhill
Copy link
Contributor

Awesome, can you reproduce with debug=msg,topic and provide me with the output?

@winbatch
Copy link
Author

Sent offline

On Wednesday, May 14, 2014, Magnus Edenhill notifications@github.com
wrote:

Awesome, can you reproduce with debug=msg,topic and provide me with the
output?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43069249
.

@edenhill
Copy link
Contributor

  • What is your message.max.bytes property set to on the broker and consumer?
  • What is fetch.message.max.bytes set to on the consumer? I would guess 1048576.
  • Is it possible that the topic contains messages larger than 1048576?

@winbatch
Copy link
Author

I didn't set either on the consumer, so must be default. Very possible
there are larger sizes in the broker. Will set those and see.

On Wednesday, May 14, 2014, Magnus Edenhill notifications@github.com
wrote:

  • What is your message.max.bytes property set to on the broker and
    consumer?
  • What is fetch.message.max.bytes set to on the consumer? I would
    guess 1048576.
  • Is it possible that the topic contains messages larger than 1048576?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43086757
.

@winbatch
Copy link
Author

I think that was it. Can you enhance to print the sizes as well as the
offset in the error?

On Wednesday, May 14, 2014, Dan Hoffman hoffmandan@gmail.com wrote:

I didn't set either on the consumer, so must be default. Very possible
there are larger sizes in the broker. Will set those and see.

On Wednesday, May 14, 2014, Magnus Edenhill <notifications@github.comjavascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

  • What is your message.max.bytes property set to on the broker and
    consumer?
  • What is fetch.message.max.bytes set to on the consumer? I would
    guess 1048576.
  • Is it possible that the topic contains messages larger than 1048576?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43086757
.

@winbatch
Copy link
Author

I take it back, still happened

On Wednesday, May 14, 2014, Dan Hoffman hoffmandan@gmail.com wrote:

I think that was it. Can you enhance to print the sizes as well as the
offset in the error?

On Wednesday, May 14, 2014, Dan Hoffman <hoffmandan@gmail.comjavascript:_e(%7B%7D,'cvml','hoffmandan@gmail.com');>
wrote:

I didn't set either on the consumer, so must be default. Very possible
there are larger sizes in the broker. Will set those and see.

On Wednesday, May 14, 2014, Magnus Edenhill notifications@github.com
wrote:

  • What is your message.max.bytes property set to on the broker and
    consumer?
  • What is fetch.message.max.bytes set to on the consumer? I would
    guess 1048576.
  • Is it possible that the topic contains messages larger than
    1048576?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43086757
.

@winbatch
Copy link
Author

1400078130.011 RDKAFKA-3-FAIL: rdkafka#consumer-0:
d146537-025.masked.com:5757/1052189117: Receive failed: Invalid message
size 104857637 (0..104857600): increase receive.message.max.bytes

1400078130.011 RDKAFKA-3-ERROR: rdkafka#consumer-0:
d146537-025.masked.com:5757/1052189117: Receive failed: Invalid message
size 104857637 (0..104857600): increase receive.message.max.bytes

1400078130.011 RDKAFKA-7-FETCH: rdkafka#consumer-0:
d146537-025.masked.com:5757/1052189117: Fetch reply: Local: Bad message
format

On Wednesday, May 14, 2014, Dan Hoffman hoffmandan@gmail.com wrote:

I take it back, still happened

On Wednesday, May 14, 2014, Dan Hoffman <hoffmandan@gmail.comjavascript:_e(%7B%7D,'cvml','hoffmandan@gmail.com');>
wrote:

I think that was it. Can you enhance to print the sizes as well as the
offset in the error?

On Wednesday, May 14, 2014, Dan Hoffman hoffmandan@gmail.com wrote:

I didn't set either on the consumer, so must be default. Very possible
there are larger sizes in the broker. Will set those and see.

On Wednesday, May 14, 2014, Magnus Edenhill notifications@github.com
wrote:

  • What is your message.max.bytes property set to on the broker and
    consumer?
  • What is fetch.message.max.bytes set to on the consumer? I would
    guess 1048576.
  • Is it possible that the topic contains messages larger than
    1048576?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43086757
.

@winbatch
Copy link
Author

Something is very weird. The max msg size on the broker is 10MB. I set
rdkafka on the consumer to 20MB and still have problems

On Wednesday, May 14, 2014, Dan Hoffman hoffmandan@gmail.com wrote:

1400078130.011 RDKAFKA-3-FAIL: rdkafka#consumer-0:
d146537-025.masked.com:5757/1052189117: Receive failed: Invalid message
size 104857637 (0..104857600): increase receive.message.max.bytes

1400078130.011 RDKAFKA-3-ERROR: rdkafka#consumer-0:
d146537-025.masked.com:5757/1052189117: Receive failed: Invalid message
size 104857637 (0..104857600): increase receive.message.max.bytes

1400078130.011 RDKAFKA-7-FETCH: rdkafka#consumer-0:
d146537-025.masked.com:5757/1052189117: Fetch reply: Local: Bad message
format

On Wednesday, May 14, 2014, Dan Hoffman <hoffmandan@gmail.comjavascript:_e(%7B%7D,'cvml','hoffmandan@gmail.com');>
wrote:

I take it back, still happened

On Wednesday, May 14, 2014, Dan Hoffman hoffmandan@gmail.com wrote:

I think that was it. Can you enhance to print the sizes as well as the
offset in the error?

On Wednesday, May 14, 2014, Dan Hoffman hoffmandan@gmail.com wrote:

I didn't set either on the consumer, so must be default. Very possible
there are larger sizes in the broker. Will set those and see.

On Wednesday, May 14, 2014, Magnus Edenhill notifications@github.com
wrote:

  • What is your message.max.bytes property set to on the broker and
    consumer?
  • What is fetch.message.max.bytes set to on the consumer? I would
    guess 1048576.
  • Is it possible that the topic contains messages larger than
    1048576?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43086757
.

@edenhill
Copy link
Contributor

  • message.max.bytes and fetch.message.max.bytes should be synchronized to the same value across all producers, consumers and brokers in the cluster.
  • receive.message.max.bytes needs to be fetch.message.max.bytes * number of partitions being consumed + framing overhead (add another 100k to be safe).

@edenhill
Copy link
Contributor

receive.message.max.bytes should and could be figured out by librdkafka all by itself, but it doesnt, thats why it needs to be configured properly. If you dont want to do the math (because the numbre of partitions is dynamic or whatever reason), you can just set it to some large number, like 4 gigs.

@winbatch
Copy link
Author

'receive.message .' Or 'fetch.message.'?

On Wednesday, May 14, 2014, Magnus Edenhill notifications@github.com
wrote:

receive.message.max.bytes should and could be figured out by librdkafka
all by itself, but it doesnt, thats why it needs to be configured properly.
If you dont want to do the math (because the numbre of partitions is
dynamic or whatever reason), you can just set it to some large number, like
4 gigs.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43095529
.

@edenhill
Copy link
Contributor

You are right. Clarified previous comment.

@winbatch
Copy link
Author

Ok. So set message and fetch.message to 10MB and receive to huge . Right?

On Wednesday, May 14, 2014, Magnus Edenhill notifications@github.com
wrote:

You are right. Clarified previous comment.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43096467
.

@edenhill
Copy link
Contributor

Yep!

@edenhill
Copy link
Contributor

Let me know if this solves the problem

@winbatch
Copy link
Author

It seems to be stable now. I had to put both max and fetch to be larger
than the broker max and also put receive to the max rdkafka allows
1000000000.

On Wednesday, May 14, 2014, Magnus Edenhill
<notifications@github.comjavascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

Let me know if this solves the problem

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-43103465
.

@edenhill edenhill changed the title Ever seen this error before? (Protocol parse failure) Setting receive.message.max.bytes (Protocol parse failure) May 15, 2014
edenhill added a commit that referenced this issue May 15, 2014
@edenhill
Better error messages when receive.message.max.bytes is too low (issue …
b1ff49c 
…#116)
@weekendyogi
Copy link

I am hitting this issue.

On all brokers
max.request.size=1000000
fetch.message.max.bytes=1000000

message.max.bytes=1000000
replica.fetch.max.bytes=1000000

On consumer
receive.message.max.bytes=1000000000
fetch.message.max.bytes=1000000
message.max.bytes=1000000

1527700852.798 RDKAFKA-7-BROKERFAIL: rdkafka#consumer-1: [thrd:ssl://knode1.scaletest.mgmt:8082/bootstrap]: ssl://knode1.scaletest.mgmt:8082/bootstrap: failed: err: Local: Bad message format: (errno: Success)
1527700852.798 RDKAFKA-3-FAIL: rdkafka#consumer-1: [thrd:ssl://knode1.scaletest.mgmt:8082/bootstrap]: ssl://knode1.scaletest.mgmt:8082/bootstrap: Receive failed: Invalid message size 1213486156 (0..1000000000): increase receive.message.max.bytes
1527700852.799 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:ssl://knode1.scaletest.mgmt:8082/bootstrap]: ssl://knode1.scaletest.mgmt:8082/bootstrap: Receive failed: Invalid message size 1213486156 (0..1000000000): increase receive.message.max.bytes

@edenhill
Copy link
Contributor

@weekendyogi what librdkafka version?

@weekendyogi
Copy link

weekendyogi commented May 30, 2018 via email •
edited
Loading

@MockingJayWong MockingJayWong mentioned this issue Jul 9, 2019
Closed
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@edenhill @winbatch @weekendyogi