Performance and testing pass

[webmakersteve]

Lots of changes here and I'm not done. Just publishing this initially as a PR so it can get some visibility.

[webmakersteve]

1. Producer now has a closer mapping to the underlying RdKafka produce method. No longer does object enumeration. Should be about a 10-15% performance gain in producing especially with large messages.
2. Consumer now copies memory instead of creating a reference to the message in the underlying buffer. Should allow better free'ing, but does slow performance down based on how large the messages read are.
3. Do version checking to support flush and additional constants if they are in librdkafka
4. Export constants in C++ Land and enumerate the object on library initialization to make them more maintainable

[webmakersteve]

Updated TopicWritable to use the new produce method

The way that stream backpressure works now that the method is synchronous:
The stream will keep writing to the producer until it gets a ERR__QUEUE_FULL. When it does, it holds off on writing and does not call the callback. It then retries the message in 500ms and furthermore until it gets a fatal error or the queue accepts the message.

You will be able to increase the amount the queue takes through the librdkafka configuration parameter..

Other change of note: consumed message data is no longer message.message. It is message.payload to be more in line with the librdkafka message object.

Also, at this point, converted end to end tests to use mocha, as the big thing stopping it was the graceful disconnect issue which appears to be as gone as it will be.

NOTE: sometimes, shutting down can take up to 10-15 seconds. I believe these are outgoing sockets still timing out in the background. It generally hangs that long on error'd requests, but it should still exit out eventually.

[edenhill]

You might be interested in the new .._F_MSG_BLOCK flag:
https://github.com/edenhill/librdkafka/blob/master/src-cpp/rdkafkacpp.h#L1608

Re message.message: standard Kafka semantics is actually message.value, so you might want to go with that rather than .payload.

Re slow shutdown: debug=broker,cgrp now (master) prints what a broker or cgrp might be stalling on when trying to shut down, be it outstanding requests, pending commits, etc.

[webmakersteve]

@edenhill I did take note of that MSG_BLOCK flag but saw it was not included in the current release. Was going to put some precompiler directives to support the flag if it was available but it would change the underlying API so I opted not to until it's been released.

Re: message.message - Sure. Thanks for the suggestion! Makes sense to me.

Re: slow shutdown: I'll take a look today with debug config on and see what I can see. Thanks!

[webmakersteve]

New Changes

1. message.value is now the buffer consumed from kafka.
2. Produced message now supports NULL payload
3. Produces message now supports NULL key or empty string key

[webmakersteve]

@edenhill From what I can gather it's another GC related issue. If a created topic object is not cleaned up when delete is called on the producer, it blocks in the destructor and never gets free'd. Can be worked around depending on how you structure your JS so I don't think it's a big problem, but will probably look into other ways to do it.

[webmakersteve]

New Changes

1. message.key is now null when there is no key present instead of undefined.

[webmakersteve]

New Changes

1. Added missing consumer method subscription
2. Added missing consumer method position
3. Added missing consumer method committed (async)
4. Began refactor of topic readable to make it a first class citizen
5. subscribe is now sync and throws
6. unsubscribe is now sync and throws