Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Counting error in rdkafka_performance #1542

Closed
LionelCons opened this issue Nov 22, 2017 · 1 comment
Closed

Counting error in rdkafka_performance #1542

LionelCons opened this issue Nov 22, 2017 · 1 comment

Comments

@LionelCons
Copy link

I ran rdkafka_performance with:

$ rdkafka_performance -P -t performance-tests -c 60000000 -s 1000 -S 1 -a 0

It worked fine and at some point it has produced all the messages requested:

% 59895281 messages produced (59895281000 bytes), 59395313 delivered (offset 0, 0 failed) in 1032819ms: 57507 msgs/s and 57.51 MB/s, 1583935 produce failures, 499969 in queue, no compression
% 59943352 messages produced (59943352000 bytes), 59443359 delivered (offset 0, 0 failed) in 1033819ms: 57498 msgs/s and 57.50 MB/s, 1586516 produce failures, 499995 in queue, no compression
% 59999218 messages produced (59999218000 bytes), 59499222 delivered (offset 0, 0 failed) in 1034820ms: 57497 msgs/s and 57.50 MB/s, 1588434 produce failures, 499996 in queue, no compression
% 60000000 messages produced (60000000000 bytes), 59521253 delivered (offset 0, 0 failed) in 1035821ms: 57462 msgs/s and 57.46 MB/s, 1588447 produce failures, 478747 in queue, no compression
% 60000000 messages produced (60000000000 bytes), 59541214 delivered (offset 0, 0 failed) in 1036827ms: 57426 msgs/s and 57.43 MB/s, 1588447 produce failures, 458786 in queue, no compression
% 60000000 messages produced (60000000000 bytes), 59561058 delivered (offset 0, 0 failed) in 1037853ms: 57388 msgs/s and 57.39 MB/s, 1588447 produce failures, 438943 in queue, no compression

So far, so good.

However, near the end of the run, it reported:

% 60000000 messages produced (60000000000 bytes), 59960874 delivered (offset 0, 0 failed) in 1058937ms: 56623 msgs/s and 56.62 MB/s, 1588447 produce failures, 39128 in queue, no compression
% 60000000 messages produced (60000000000 bytes), 59979812 delivered (offset 0, 0 failed) in 1059938ms: 56587 msgs/s and 56.59 MB/s, 1588447 produce failures, 20198 in queue, no compression
% 60000000 messages produced (60000000000 bytes), 59997103 delivered (offset 0, 0 failed) in 1060944ms: 56550 msgs/s and 56.55 MB/s, 1588447 produce failures, 2901 in queue, no compression
% 1588447 backpressures for 60000000 produce calls: 2.647% backpressure rate
% 59999999 messages produced (59999999000 bytes), 60000000 delivered (offset 0, 0 failed) in 1061132ms: 56543 msgs/s and 56.54 MB/s, 1588447 produce failures, 0 in queue, no compression

As you can see, it went from "60000000 messages produced" down to "59999999 messages produced" but reported anyway "60000000 delivered"!

This is with librdkafka version 0.11.1 (0x000b01ff).

@edenhill
Copy link
Contributor

There just happens to be an event enqueued for the application (which is accounted for in outq_len()):
https://github.com/edenhill/librdkafka/blob/master/examples/rdkafka_performance.c#L1340

That line should probably be removed.

gnanasekarl added a commit to sububack/librdkafka that referenced this issue Sep 17, 2018
Removing the deduction of outq len from the total message produced count, which creates unnecessary mismatch between delivered and produced counters if in case there is some messages in outq.
Referenace: issue#1542
edenhill pushed a commit that referenced this issue Oct 18, 2018
* Counting error in rdkafka_performance #1542

Removing the deduction of outq len from the total message produced count, which creates unnecessary mismatch between delivered and produced counters if in case there is some messages in outq.
Referenace: issue#1542

* remove whitespaces

* Update rdkafka_performance.c
@edenhill edenhill closed this as completed Apr 9, 2019
sububack pushed a commit to sububack/librdkafka that referenced this issue Apr 11, 2019
Removing the deduction of outq len from the total message produced count, which creates unnecessary mismatch between delivered and produced counters if in case there is some messages in outq.
Referenace: issue#1542
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants