Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mq_sink: unexpected behavior of max-message-bytes #2962

Closed
Tracked by #2971
liuzix opened this issue Sep 29, 2021 · 5 comments · Fixed by #3002
Closed
Tracked by #2971

mq_sink: unexpected behavior of max-message-bytes #2962

liuzix opened this issue Sep 29, 2021 · 5 comments · Fixed by #3002
Assignees
Labels

Comments

@liuzix
Copy link
Contributor

liuzix commented Sep 29, 2021

What did you do?

Assume the Kafka broker has 1M as message.max.bytes. I use TiCDC with Kafka sink and Open Protocol, where I set max-message-bytes=10K.

What did you expect to see?

If there is no row whose serialized size is more than 1M then everything works normally. If there is, I would need a detailed error message helping me identify the problematic row (and table).

The expected behavior is that: as long as CDC has max-message-bytes <= 1M and there is no row larger than 1M, everything should work.

What did you see instead?

Any message larger than 10K fails to be delivered.

The root cause is that (1) max-message-bytes is used both as an advisory to the Open Protocol encoder AND as a hard-limit for the kafka producer (sarama) (2) we have failed to take into account the producer overhead by the kafka client.

Problematic Scenarios

a.

|<------ CDC's limit = Broker's Limit = 10M ----->|<-overflow->|
| row A = 5M | row B = 4M | row C = 1000K| overhead = 100K|

The batch should have been reduced, but was not.

b.
|<--------------------------- Broker's Limit = 10M ------------------------>|
|<---- CDC's Limit = 5M --->|<--overflow-->|
|<------------- row A = 6M ----------------------->|<--still space here->|

The message should have been sent, but was not.

Versions of the cluster

v5.0.1

@liuzix liuzix added the type/bug The issue is confirmed as a bug. label Sep 29, 2021
@cx2671206
Copy link

cx2671206 commented Sep 29, 2021

@liuzix Want to know is there a difference between set max-message-bytes=10K and max-message-bytes=1MB in this case? Curious why you set it to 10K when kafka size limit is 1MB

If Kafka limit is 1MB, and max-message-bytes=10KB, what's the expected behavior for event with size larger than 10KB?

@liuzix liuzix added the bug-from-user Bugs found by users. label Sep 29, 2021
@liuzix
Copy link
Contributor Author

liuzix commented Sep 29, 2021

@liuzix Want to know is there a difference between set max-message-bytes=10K and max-message-bytes=1MB in this case? Curious why you set it to 10K when kafka size limit is 1MB

If Kafka limit is 1MB, and max-message-bytes=10KB, what's the expected behavior for event with size larger than 10KB?

If a single event has to take up say 5MB, we have to send it out in a batch containing only itself. Otherwise, data loss would occur.

@yiwu-arbug
Copy link

For the particular case we didn't find "Event does not fit into max-message-bytes. Adjust relevant configurations to avoid service interruptions." log line. @lance6716 confirmed we rule out case b, and the issue is likely caused by case a.

@lance6716
Copy link
Contributor

For the particular case we didn't find "Event does not fit into max-message-bytes. Adjust relevant configurations to avoid service interruptions." log line. @lance6716 confirmed we rule out case b, and the issue is likely caused by case a.

now for a new log, case b is also triggered.

@3AceShowHand
Copy link
Contributor

Consider we set max-message-size as 100, and now the message's size is 99, so the log will not trigger.

But when sending the message to the sarama producer, it will check the message's size, 99 + 36 = 135, 135 > 100, so reject it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants