KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees #20285

shashankhs11 · 2025-07-31T22:28:40Z

Fixes a bug in producer.

Cf - #20254 (comment)
and KAFKA-19479 for
more details

github-actions · 2025-08-08T03:37:10Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

junrao

@shashankhs11 : Thanks for the PR. Left a comment.

clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java

junrao

@shashankhs11 : Thanks for the updated PR. A few more comments.

clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java

clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java

clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java

...java/org/apache/kafka/streams/integration/AtLeastOnceDeliveryMessageLossIntegrationTest.java

clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java

junrao

@shashankhs11 : Thanks for the updated PR. A few more comments.

clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java

clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java

junrao

@shashankhs11 : Thanks for the updated PR. A few more comments.

clients/src/main/java/org/apache/kafka/clients/producer/internals/ProduceRequestResult.java

junrao

@shashankhs11 : Thanks for the updated PR. A few more comments.

clients/src/main/java/org/apache/kafka/clients/producer/internals/ProduceRequestResult.java

clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java

...java/org/apache/kafka/streams/integration/AtLeastOnceDeliveryMessageLossIntegrationTest.java

junrao · 2025-10-15T18:37:07Z

@shashankhs11 : Could you fix the build error? Also, could you run the same jmh test again to make sure there is no regression on the flush() time?

shashankhs11 · 2025-10-15T20:45:26Z

JMH Test Report

This is how I ran the test

I created a new local branch based off trunk with just the jmh test and ran the test.
Ran the test in this branch
2 different cases - numRecords = 5000 and numRecords = 10000

Result BEFORE fix

Case 1: `numRecords = 5000`

Case 2: `numRecords = 10000`

Result AFTER fix (current branch)

Case 1: `numRecords = 5000`

Case 2: `numRecords = 10000`

shashankhs11 · 2025-10-15T20:50:16Z

@junrao: fixed the build error and added jmh test report in a seperate comment above. There seems to be a regression when numRecords = 10000 -- Would this be acceptable?

Also left a comment in reply to this - #20285 (comment)

junrao · 2025-10-16T00:55:18Z

@shashankhs11 : The small regression from 0.004 to 0.005 ms/op is negligible and is fine.

shashankhs11 · 2025-10-16T02:03:40Z

CI check with Java 25 seemed to be failing. Rebased with latest trunk.

junrao

@shashankhs11 : Thanks for the updated PR. Just a minor comment. Also, could you rebase to pick up a flaky test fix #20713 ?

clients/src/main/java/org/apache/kafka/clients/producer/internals/ProduceRequestResult.java

shashankhs11 · 2025-10-16T17:43:38Z

Also, could you rebase to pick up a flaky test fix

@junrao: Done :)

junrao

@shashankhs11 : Thanks for the updated PR. LGTM. Could you update the description of the PR before merging?

shashankhs11 · 2025-10-16T21:26:59Z

Could you update the description of the PR before merging?

@junrao: Done! Changed the title as well as the description.

Thank you very much for your time and patience, Jun. I am very very grateful for the opportunity to have worked closely with you over the past couple of weeks. I got to learn a lot from this :)

junrao · 2025-10-16T22:25:50Z

@shashankhs11 : I kept the PR name to match the jira name, but adjusted the description in the commit message. Thanks for working on this issue!

mjsax · 2025-10-17T04:07:15Z

Thanks for reviewing this PR @junrao!

…ges when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees (#20285) Bug Fix in Producer where flush() does not wait for a batch to complete after splitting. Cf - #20254 (comment) and [KAFKA-19479](https://issues.apache.org/jira/browse/KAFKA-19479) for more details Reviewers: Jun Rao <junrao@gmail.com>

mjsax · 2025-10-17T05:15:45Z

Cherry-picked to 4.1, 4.0, and 3.9 branches.

…ges when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees (apache#20285) Bug Fix in Producer where flush() does not wait for a batch to complete after splitting. Cf - apache#20254 (comment) and [KAFKA-19479](https://issues.apache.org/jira/browse/KAFKA-19479) for more details Reviewers: Jun Rao <junrao@gmail.com>

github-actions bot added triage PRs from the community producer tests Test fixes (including flaky tests) clients labels Jul 31, 2025

shashankhs11 mentioned this pull request Jul 31, 2025

KAFKA-19479: Add test to verify data-loss bug was fixed inside the producer #20254

Merged

github-actions bot added the needs-attention label Aug 8, 2025

junrao reviewed Sep 29, 2025

View reviewed changes

clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java Outdated Show resolved Hide resolved

github-actions bot removed needs-attention triage PRs from the community labels Sep 30, 2025

mjsax added the ci-approved label Oct 4, 2025

shashankhs11 force-pushed the KAFKA-19479-more-tests branch from 467de10 to a51d19b Compare October 6, 2025 01:34

github-actions bot added the streams label Oct 6, 2025

junrao reviewed Oct 6, 2025

View reviewed changes

shashankhs11 requested a review from junrao October 7, 2025 02:12

junrao reviewed Oct 7, 2025

View reviewed changes

github-actions bot added the performance label Oct 8, 2025

shashankhs11 requested a review from junrao October 8, 2025 23:48

junrao reviewed Oct 9, 2025

View reviewed changes

shashankhs11 requested a review from junrao October 12, 2025 15:17

junrao reviewed Oct 13, 2025

View reviewed changes

shashankhs11 force-pushed the KAFKA-19479-more-tests branch from 0e9c522 to 6a4745c Compare October 16, 2025 02:02

junrao reviewed Oct 16, 2025

View reviewed changes

clients/src/main/java/org/apache/kafka/clients/producer/internals/ProduceRequestResult.java Outdated Show resolved Hide resolved

shashankhs11 added 2 commits October 16, 2025 10:40

add helper functions and result for tests

78121ab

test: added 3 passing tests

9f97d8e

shashankhs11 added 14 commits October 16, 2025 10:40

add perf test

3fd4553

jmh test

ac97b87

add unit test

97ee708

revert previous fix

0e1d2bc

add optimized fix with refactoring

0d381f5

fix typo

41b7769

new changes

788318e

rename to addDependent

467cdf4

fix typo in testname and change sleep time

b777871

fixed and added more comment

e6368ca

fix build error

590f1a5

fix build error in jmh

22fa090

remove test

59dbdd0

fix minor comment

896ecc1

shashankhs11 force-pushed the KAFKA-19479-more-tests branch from 6a4745c to 896ecc1 Compare October 16, 2025 17:41

junrao approved these changes Oct 16, 2025

View reviewed changes

shashankhs11 changed the title ~~KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees~~ KAFKA-19479: Bug Fix in Producer where flush() does not wait for its dependents Oct 16, 2025

junrao changed the title ~~KAFKA-19479: Bug Fix in Producer where flush() does not wait for its dependents~~ KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees Oct 16, 2025

junrao merged commit 6f4ce76 into apache:trunk Oct 16, 2025
24 of 25 checks passed

KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees #20285

KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees #20285

Uh oh!

Conversation

shashankhs11 commented Jul 31, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 8, 2025

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

junrao commented Oct 15, 2025

Uh oh!

shashankhs11 commented Oct 15, 2025

JMH Test Report

Result BEFORE fix

Case 1: numRecords = 5000

Case 2: numRecords = 10000

Result AFTER fix (current branch)

Case 1: numRecords = 5000

Case 2: numRecords = 10000

Uh oh!

shashankhs11 commented Oct 15, 2025

Uh oh!

junrao commented Oct 16, 2025

Uh oh!

shashankhs11 commented Oct 16, 2025

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shashankhs11 commented Oct 16, 2025

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

shashankhs11 commented Oct 16, 2025

Uh oh!

Uh oh!

junrao commented Oct 16, 2025

Uh oh!

mjsax commented Oct 17, 2025

Uh oh!

shashankhs11 commented Jul 31, 2025 •

edited by github-actions bot

Loading

Case 1: `numRecords = 5000`

Case 2: `numRecords = 10000`

Case 1: `numRecords = 5000`

Case 2: `numRecords = 10000`