errno: Interrupted system call

[potchin]

This may be related to #18 but its supposed to be fixed.

I am catting a text file into kafka with kafkacat (built today from bootstrap.sh), my messages are appearing in the topic but kafkacat is exiting with a return code of 1. The debug messages show...

%7|1475839646.429|MSGSET|rdkafka#producer-1| kafka.sp.mgmt:9092/0: MessageSet with 800 message(s) delivered
%7|1475839646.431|DESTROY|rdkafka#producer-1| Terminating instance
%7|1475839646.431|BROADCAST|rdkafka#producer-1| Broadcasting state change
%7|1475839646.529|TOPPAR|rdkafka#producer-1| kafka.sp.mgmt:9092/0: mytopic [1] 0+0 msgs
%7|1475839646.529|TOPPAR|rdkafka#producer-1| kafka.sp.mgmt:9092/0: mytopic [0] 0+0 msgs
%7|1475839646.529|DESTROY|rdkafka#producer-1| Destroy internal
%7|1475839646.529|DESTROY|rdkafka#producer-1| Remove all topics
%7|1475839646.529|PARTCNT|rdkafka#producer-1| Topic mytopic partition count changed from 2 to 0
%7|1475839646.529|BRKDELGT|rdkafka#producer-1| Broker kafka.sp.mgmt:9092/0 no longer leader for topic mytopic [0]
%7|1475839646.529|BRKDELGT|rdkafka#producer-1| No broker is leader for topic mytopic [0]
%7|1475839646.529|BRKMIGR|rdkafka#producer-1| Migrating topic mytopic [0] from kafka.sp.mgmt:9092/0 to (none)
%7|1475839646.529|BRKDELGT|rdkafka#producer-1| Broker kafka.sp.mgmt:9092/0 no longer leader for topic mytopic [1]
%7|1475839646.529|BRKDELGT|rdkafka#producer-1| No broker is leader for topic mytopic [1]
%7|1475839646.529|BRKMIGR|rdkafka#producer-1| Migrating topic mytopic [1] from kafka.sp.mgmt:9092/0 to (none)
%7|1475839646.529|TOPPARREMOVE|rdkafka#producer-1| Removing toppar mytopic [-1]
%7|1475839646.529|TOPIC|rdkafka#producer-1| mytopic: purging 0 messages
%7|1475839646.529|TOPPAR|rdkafka#producer-1| kafka.sp.mgmt:9092/0: mytopic [1] 0+0 msgs
%7|1475839646.529|TOPPAR|rdkafka#producer-1| kafka.sp.mgmt:9092/0: mytopic [0] 0+0 msgs
%7|1475839646.529|TOPBRK|rdkafka#producer-1| kafka.sp.mgmt:9092/0: Topic mytopic [0]: leaving broker (next leader (none))
%7|1475839646.529|BROADCAST|rdkafka#producer-1| Broadcasting state change
%7|1475839646.529|TOPPARREMOVE|rdkafka#producer-1| Removing toppar mytopic [0]
%7|1475839646.529|TOPBRK|rdkafka#producer-1| kafka.sp.mgmt:9092/0: Topic mytopic [1]: leaving broker (next leader (none))
%7|1475839646.529|BROADCAST|rdkafka#producer-1| Broadcasting state change
%7|1475839646.529|TOPPARREMOVE|rdkafka#producer-1| Removing toppar mytopic [1]
%7|1475839646.530|TERMINATE|rdkafka#producer-1| kafka.sp.mgmt:9092/0: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1475839646.530|BROKERFAIL|rdkafka#producer-1| kafka.sp.mgmt:9092/0: failed: err: Local: Broker handle destroyed: (errno: Interrupted system call)
%7|1475839646.530|STATE|rdkafka#producer-1| kafka.sp.mgmt:9092/0: Broker changed state UP -> DOWN
%7|1475839646.530|BROADCAST|rdkafka#producer-1| Broadcasting state change
%7|1475839646.530|BUFQ|rdkafka#producer-1| kafka.sp.mgmt:9092/0: Purging bufq with 0 buffers
%7|1475839646.530|BUFQ|rdkafka#producer-1| kafka.sp.mgmt:9092/0: Updating 0 buffers on connection reset
%7|1475839646.538|TERMINATE|rdkafka#producer-1| :0/internal: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1475839646.538|BROKERFAIL|rdkafka#producer-1| :0/internal: failed: err: Local: Broker handle destroyed: (errno: Success)
%7|1475839646.538|STATE|rdkafka#producer-1| :0/internal: Broker changed state UP -> DOWN
%7|1475839646.538|BROADCAST|rdkafka#producer-1| Broadcasting state change
%7|1475839646.538|BUFQ|rdkafka#producer-1| :0/internal: Purging bufq with 0 buffers
%7|1475839646.538|BUFQ|rdkafka#producer-1| :0/internal: Updating 0 buffers on connection reset

Any suggestions?

[potchin]

Just to add, I have checked and all lines from the file are correctly appearing in the topic.

[edenhill]

If you disable librdkafka debugging, is kafkacat saying anything?
Is this reproducible with any file or just this one?
How big is the file? (in bytes and lines)

[potchin]

Nope, nothing being output if I dont specify "-X debug=all".

To give some more info, I'm calling kafkacat from a shell script to process some historical data (few hundred files). It worked fine for while (returning 0) but with the newer files (which are larger) the error happens everytime.

The current file is ~24M and 300k lines long. Unfortunately it contains confidential data so I cant share it :(

[potchin]

Breaking point seems to be somewhere between 10MB and 14MB

[edenhill]

I think Ive reproduced it now.
You might be hitting queue.buffering.max.messages which causes produce() to return QUEUE_FULL, but this should simply block kafkacat until room is available, not return an error code.

[edenhill]

You can verify that this is the case by setting -X queue.buffering.max.messages=N to a value larger than the number of messages you will send.

[edenhill]

This should now be fixed on master, can you update and try to reproduce?

[potchin]

Awesome, its working perfectly now. Thanks for the quick turnaround :)