Producer core dump

[ottomata]

Am in the process of testing PR #42 as a fix for Issue #5, and ran into this:

const Kafka = require('node-rdkafka');
const producer = new Kafka.Producer({
    'metadata.broker.list': 'localhost:9092',
});
producer.connect(undefined, () => {
    producer.produce({
        topic: 'test',
        message: '{"f": 1}'
    });
});

nodejs: ../node_modules/nan/nan_object_wrap.h:33: static T* Nan::ObjectWrap::Unwrap(v8::Local<v8::Object>) [with T = NodeKafka::Topic]: Assertion `object->InternalFieldCount() > 0' failed.
Aborted (core dumped)

node-rdkafka compiled at commit 62e53c22cafe93b9697fc9ff6fc0daa7c2670da2.

[webmakersteve]

It looks like it didn't properly create the topic. I can test this later, but just in case this helps, can you try using a different topic name?

If that doesn't work, instead of using a string in that method, can you try using new producer.Topic('test', {}); and see if that makes the issue go away?

I don't normally recommend doing it that way as it interferes with graceful shutdown, but it may help isolate the bug. But like I said, I'll take a look at this later today otherwise :)

[ottomata]

Topic 'test' does exist. I also tried with another existent topic and a non-existent one, and get the same behavior. Switching to new Topic like this:

const Kafka = require('node-rdkafka');
const producer = new Kafka.Producer({
    'metadata.broker.list': 'localhost:9092',
});
producer.connect(undefined, () => {
    producer.produce({
        topic: new producer.Topic('test', {}),
        message: '{"f": 1}'
    });
});

Gives

/vagrant/es/kasocki/node_modules/node-rdkafka.latest/lib/producer.js:165
    this.emit('error', LibrdKafkaError.create(err));
         ^

TypeError: this.emit is not a function
    at new Producer.Topic (/vagrant/es/kasocki/node_modules/node-rdkafka.latest/lib/producer.js:165:10)
    at /vagrant/es/kasocki/scr/producer.js:8:16
    at next (/vagrant/es/kasocki/node_modules/node-rdkafka.latest/lib/client.js:132:7)
    at /vagrant/es/kasocki/node_modules/node-rdkafka.latest/lib/client.js:209:7
    at /vagrant/es/kasocki/node_modules/node-rdkafka.latest/lib/client.js:335:7`

[webmakersteve]

Sorry. Should be topic: producer.Topic('test', {}); And make sure to add an on('error') listener so we can see why the RdKafka::Topic creation is failing.

I'll update this to fail a little bit better than it does. It should throw when you try to produce to a failed topic, rather than core dump due to an assertion.

[ottomata]

topic: producer.Topic('test', {}) also assert and dumps:

nodejs: ../node_modules/nan/nan_object_wrap.h:33: static T* Nan::ObjectWrap::Unwrap(v8::Local<v8::Object>) [with T = NodeKafka::Topic]: Assertion `object->InternalFieldCount() > 0' failed.
Aborted (core dumped)

I had an on('error' handler, but it never was called. I just added it back, and it isn't called before the assert.

[webmakersteve]

d'oh. Today is definitely a Monday. This is a breaking change. it should be:

producer.produce('test', null, new Buffer('{"f": 1}'), null);

I am going to add deprecation messages across the board for the old usage before release.

[ottomata]

Oh hm. Ok. Can you keep the callback parameter in there too? We promisify() the producer/consumer, and having a node-style callback is needed to get a promisified produceAsync() method added.

[webmakersteve]

The deprecated version of the function will still honor the callback, even though it will just setImmediate to it with the error if one was thrown. This method is no longer async, as it only adds a message to the queue of the internal librdkafka producer buffer. It is already non-blocking, so the update avoids wasting time queueing up the work.

[webmakersteve]

f9070c6 you can see how it will work here

[ottomata]

Hm ok. Does this mean it will no longer be possible to know if a specific message is ACKed by Kafka via callback?

[webmakersteve]

The callback wouldn't tell you if the message was acked unfortunately: https://github.com/edenhill/librdkafka/blob/master/src-cpp/rdkafkacpp.h#L1635

If you require acks, you can use the delivery report as an authoritative indicator that your message has been written to the ISRs/lead broker. But acks do not block produce

The library has the ability to take an opaque to identify the message later in the delivery report, but it is not currently implemented on the node side. However, it isn't out of the realm of possibility if people got some use out of it.

I think generally, you need to trust librdkafka to retry until it gets the acks you specify you need for that message.

[webmakersteve]

Gracefully deprecated with #55