Min wall clock time per single kafkacat invokation

[ibalashov]

Hi,

It seems that it takes at least 1 second for kafkacat producer to complete.

time sh -c "date | ~/dev/kafkacat/kafkacat -P -t _temp -p 0 -c 1 -b 192.168.86.3,192.168.86.5"

real 0m1.068s
user 0m0.006s
sys 0m0.039s

Is there any timeout setting that would allow kafkacat to exit immediately upon producing all messages?

[edenhill]

You are absolutely right, there are a couple of internal timeouts in librdkafka that are 1s.
To find the best solution I'd like to ask you what would an acceptable exit-delay be for you?

[ibalashov]

Thanks for confirming this!
Hm.. Anything less than 100ms would serve my needs, but can't this delay be dropped altogether?

[edenhill]

This is now fixed in librdkafka commit confluentinc/librdkafka@8d9fdd2

$ time (echo "foo" | ./kafkacat -b 0 -P -t test -p 0)

real    0m0.052s
user    0m0.001s
sys 0m0.005s

Can you verify this on your end as well?

[ibalashov]

For some reason I still see the delay.
I tried rebuilding librdkafka and also getting one from https://github.com/edenhill/librdkafka/archive/master.tar.gz

Could it be something related to OS X?

[edenhill]

Huhm, are you sure kafkacat is using the updated librdkafka?
Is it linked dynamically or statically?
Try checking out a fresh kafkacat and run ./bootstrap.sh

[ibalashov]

Just did a fresh checkout and ./boostrap.sh on a linux installation elsewhere and now it works correctly.

Strangely, doing the same on OS X has no effect. Could you possibly spot any issues in the build log? https://gist.github.com/ibalashov/c74f8cb9217db3099362#file-kafkacat_buildlog_osx-txt

[edenhill]

Ah, SIGIO is ignored by default on OSX, but not Linux.
That's easily fixable.

But I'm not sure this is the right solution, a library typically shouldn't trigger signals and install signal handlers without the application's knowledge. Will have to think about it a bit more.

Is this stopping your development/deployment?

[ibalashov]

Glad to hear at least the cause is clear.
No worries, I needed this mainly for tests, and now I can run kafkacat from inside Linux VM if/when needed.
Thanks!

[edenhill]

I've made a better fix for this now (opt-in on the application side, kafkacat opts in) which should also work better on OSX.