Releases: confluentinc/librdkafka
v0.9.5
Maintenance release
Critical fixes
- q_concat: don't wakeup listeners if srcq is empty (fix by @orthrus in #1121): Fixes idle consumer CPU usage on Windows
- Prioritize commit_cb over messages on poll queue (closes #1088)
- Prioritize all relevant user-facing ops (callbacks) (#1088)
- Fix SaslHandshakeRequest timeout
Fixes
- Properly log SSL connection close (closes #1081)
- Don't log broker failures when an error_cb is registered (closes #1055)
- Fix use-after-free when log.queue and debug was used
- Log offset commit failures when there is no offset_commit_cb (closes #1043)
- Fix retrybuf memory leak on termination when broker is down
- Fix small memory leak in metadata_leader_query
- Use app_offset to calculate consumer_lag (closes #1112)
- Fix consumer_lag calculation (dont use cached hi_offset)
- C++: fix conf->get() allocation (closes #1118)
- Avoid use of partially destroyed topic object (#1125, fixed by @benli123)
- sasl win32: protection level and message size not sent (fixed by @zyzil )
- Properly auto-set metadata.max.age.ms when metadata.refresh.interval.ms is disabled (closes #1149)
- Fix memory alignment issues (#1150)
- consumer_example: fix crash on -X dump (closes #841)
- configure: auto add brew openssl pkg-config path
- producev() default partition should be UA, not 0 (#1153)
Enhancements
- Added global->topic config fallthru: Topic-level configuration properties can now be set on the global configuration object. The property will be applied on the default_topic_conf object (if no such object exists one is created automatically).
- Use SNI when connecting through SSL (by @vincentbernat )
- Windows performance improvements (use atomics instead of locks, avoid locking in some cases)
- Add lz4/lib sources for in-tree building when external lz4 is not available
- Add CRC checking support to consumer (#1056)
- Added op priority to queues (for #1088)
- Fail known unsupported requests locally (closes #1091)
- Improve reconnect delay handling (#1089)
- Improve OpenSSL config error propagation (#1119)
- C++: improved error reporting from Conf::set()
- Don't emit TIMED_OUT_QUEUE (revert)
v0.9.4
librdkafka v0.9.4 is a small feature release with a horde of enhancements and fixes.
NOTE: The librdkafka C++ API is unfortunately not ABI safe (the API stability is guaranteed though): C++ users will need to recompile their applications when upgrading librdkafka.
New features
- SASL: Windows SASL GSSAPI support using native Windows SSPI (#888)
- Added rd_kafka_offsets_for_times (KIP-79, #842)
- Added rd_kafka_producev() using va-args to construct message fields, including timestamps (#858, #707, #908, #345)
- C++: added produce(..., timestamp, ..)
- Added support for partition-specific queues (@lambdaknight)
- allows redirecting partitions to separate queues and separates application threads.
- True low-latency (on linux and osx) by fd-based queue wakeups
Critical fixes
- leader failover error detection and recovery for both producer and consumer
- cgrp: failover robustness improvements
- cgrp: certain failed Heartbeats could mask out future heartbeat requests
- buffer handle callbacks could previously be triggered on wrong thread
Enhancements
- Events: expose EVENT_OFFSET_COMMIT
- Experimental cmake support (thanks to @ruslo)
- Validate topics in subscribe() (mainly regex validation)
- Added set_log_queue() - allows application thread log handling (#355)
- Add support for Metadata v1..2 (part of KIP-4)
- New metadata request and caching framework:
- much fewer metadata requests
- use cached metadata when deemed possible
- KIP-4 support for lean metadata requests in clusters with many topics
- improved (faster, more reliable) leader lookups
- C++: added TopicPartition::create(, ..offset)
- C++: added const attribute to TopicPartition getters
- C++: added TopicPartition::destroy(vector) helper
- C++: added commitSync(.., OffsetCommitCb) variant
- C++: Enrichment of Conf getter for callbacks. (#883, @vin-d)
- Expose rd_kafka_topic_partition_list_sort()
- Added max.in.flight -> max.in.flight.requests.per.connection alias
- Win32: add lz4 support (using NuGet lz4 package)
- Add cgrp stats: rebalance_age, rebalance_cnt, assignment_size
Fixes
- increase maximum supported topic limit to 1M
- fix static linking on osx
- Fix infinte wait in .._commit_queue() if callback on temp queue was used
- Improved handling of assign() and outstanding rebalance_cb on termination
- commit_queue: avoid global offset_commit_cb if local specified
- Imrpoved Heartbeat error handling
- Improve OffsetCommit failure handling
- Query topics with missing or down leaders from topics_scan()
- Handle rebalance from consumer_close (#1002)
- rd_kafka_producev() would send 0 timestamp by default instead of current time
- queue.buffering.max.ms and fetch.error.backoff.ms now takes precedence over socket.blocking.max.ms (#966)
- flush() fix: msg_cnt not initialized
- rdkafka_performance: fix multi -p partitions allocation error and memory leak
- Refactor query_watermark_offsets to use new leader lookup code
- Proper error propagation on invalid topic names (#977)
- callbacks will no longer cut consumer_poll() timeouts short
- improved queue handling and serving
- rd_kafka_producev(): fix topic/rkt referencing
- Fix ApiVersionRequest timeout (10s)
- rd_kafka_q_concat: fix erroneous unlock() on enq failure
- Speed up termination by emitting TERMINATE ops
- cgrp: enforce metadata update check if matched subscribed topics is empty
- Proper suppression of connect failure logs (#847)
- Proper ATOMIC_OPn (#708)
- fix: use out of scope variable (#974, @HeChuanXUPT)
- C++: Fix undefined references to static const ints (#961, #962, @bbfgelman1)
- mklove: lib_check now attempts compilation after pkg-config passed (for false positive of SSL on OSX)
- mklove: fix ./configure --reconfigure
- stats: fix brokers.toppars key duplication
v0.9.3
Maintenance release
Enhancements
- Increase .._TOPICS_MAX to 1M
- Add cgrp stats: rebalance_age, rebalance_cnt, assignment_size
- win32: allow the use of LIBRDKAFKA_STATICLIB to enable static library (#905, @skarlsson)
- C++: Add support for setting "consume_cb" in Conf with a RdKafka::ConsumeCb and proper handling of that callback object by the underlying C library. (#741, @lambdaknight)
- Added socket.nagle.disable to disable the Nagle algorithm (#862)
- Added rd_kafka_topic_partition_list_sum()
- Added connect_cb and closesocket_cb to conf
Fixes
- Fix relative offset calculation (#859, @yunjing)
- cgrp: dont get stuck in wait-coord state on reoccuring query errors (issue #844)
- cgrp: fix a number of assign() problems (issues #871, #844, possibly #730)
- cgrp: increased robustness for wait_commit et.al (#785)
- cgrp: time out ops in wait_coord queue (#785)
- cgrp: do async termination by state to avoid recursive locks from timeout_scan
- Dont store initial offset in fetch_start0 (#742)
- Win32 compilation and linking fixes
- queue.buffering.max.kbytes wrapped on platforms with 32-bit size_t (#998)
- Timeout was not enforced in flush()
- librdkafka++: just link C library, not also its dependencies
- rdkafka_performance: fix multi -p partitions allocation error and memory leak
- Fix ApiVersionRequest timeout (10s)
- rd_kafka_q_concat: fix erroneous unlock() on enq failure
- C++: Added TopicPartition::create(, ..offset)
- C++: Fix undefined references to static const ints (#961, #962, @bbfgelman1)
- Dont start SASL for the internal broker (#968)
- Proper suppression of connect failure logs (#847)
- Proper ATOMIC_OPn (#708)
- fix: use out of scope variable (#974, @HeChuanXUPT)
- INTRODUCTION: fix wrong batch.num.messages default value (#924)
- configure: check for library headers too (fixes OpenSSL failures on OSX)
- mklove: lib_check now attempts compilation after pkg-config passed (for false positive of SSL on OSX)
- mklove: fix ./configure --reconfigure
- stats: fix brokers.toppars key duplication
- consumer: Correctly set timestamp for compressed messages (#858)
- Dont attempt SASL auth with broker-unsupported mechs (issue #907)
- consumer_poll() without a valid group.id would return a broken message_t
- Fix duplicate desired_link calls (crash or memory leak)
- Fix for async offset commit issue on AIX (#852, @asharma339)
- _builtin.. replacements for native toolchains on sun and aix (#865, @asharma339)
- Add snappy function prefix to avoid symbol collision with google snappy (#880, @hustlijian)
- toppars: Dont mix up UNKNOWN / DESIRED flags
- Use compiler specific atomic ops for sun studio compiler (#869, @asharma339)
- fix bug rd_kafka_timer_t microsecond interval overflow. (#870, @lvxv)
- Set HeartbeatRequest timeout to session.timeout.ms
- Allow max one HeartbeatRequest in transit
- DescribeGroups: Avoid zero-byte allocations (for AIX)
- fix func rd_kafka_new declare use_conf but not use cause conf->socket_cb segmentfault (#863) (@wadee)
0.9.2
Additions
- Event interface (see rd_kafka_event_type_t)
- VerifiableConsumer and VerifiableProducer clients for running official kafkatests
- Added support for LZ4 compression (KIP-57)
- Added rd_kafka_commit_queue()
- Added rd_kafka_queue_length()
- Added rd_kafka_flush()
- Added IO FD-based queue triggering for including librdkafka in IO-based poll loops (issue #749)
- Added enable.partition.eof to allow turning off PARTITION_EOF
- auto.offset.reset: added aliases: beginning and end
- Added experimental blocking produce call (.._F_BLOCK flag to produce())
Enhancements
- rd_kafka_version_str(): return a git-aware version string
- Retry metadata requests if returned response is empty
- Expedite MetadataRequests in front of other requests
- Improved
...metadata()
selection of broker to query - Added ERR__TIMED_OUT_QUEUE to differentiate between in-transit and in-queue timeouts
- rdkafka_performance: added -N to disable delivery reports
- rdkafka_performance: added -G for high-level KafkaConsumer
- Regex support on Win32: added fallback regex implementation (issue #552)
- Producer: Allow unlimited number of messages per batch/MessageSet (bump batch.num.messages to 10000)
- Producer: Copy small message payloads to transmit buffer (issue #788)
- .._conf_set(): revert to default value (if any) when passed value==NULL.
- Redesigned toppar op version barriers to be more robust
- Automatically enable api.version.request for broker.version.fallback>=0.10
- Expedite termination by broadcasting state change on destroy
- Improve ALL_BROKERS_DOWN event generation
- Fall back to mutex-protected variables when 32 or 64 bit atomics are not available (#693)
- Disable SSLv3
Consumer fixes
- Fix crash in committed() with short timeouts (issue #827)
- Dont attempt offset commits for empty non-null list
- cgrp: clear stored member_id on UNKNOWN_MEMBER_ID errors
- handle_JoinGroup: dont propagate UnknownMemberId to app
- cgrp term: force final unassign call even if waiting for rebalance
- Dont send Heartbeats when calling rebalance_cb(REVOKE)
- cgrp: ignore outdated Join & Sync responses when state has transitioned
- Dont propagate error for empty commits when internal commits are retried
- Stop offset_query_tmr when stopping fetcher
- OffsetCommit&OffsetFetch request: properly coalesce partitions for the same topic (issue #785)
- C++: added Consumer.seek() (#699)
- Fix lock ordering issue in fetcher
- Update subscription on topic changes (issues #802, #644, #576, #461)
- Fixed a number of consumer (group) corner-cases
- Resume GroupCoord queries on all errors (#768)
- Error out on unknown partition.assignment.strategy:s
- Maintain cgrp version barrier for relevant internal request ops (fixes #761)
- Dynamically increase the max fetch size when needed (#597, by @takano-akio)
- Proper timeout calculation for consume(), consumer_poll(), etc
- Properly handle empty assignments in SyncGroupResponse (#603)
- Fixed a number of hang-on-termination issues
- Consumers could stall in offset-wait state on leader change for up to socket.timeout.ms * retries (#843)
Producer fixes
- Send uncompressed if compression messageset is larger than uncompressed (issue #781)
- Proper ProducerResponse error checking before retries (issue #777)
- Time out messages on toppar xmit msgq too (fixes #776, #672)
- Dont create cgrp for producers if group.id is configured (issue #691)
- Fixed a number of hang-on-termination issues
General fixes
- Bring down the connection regardless of socket.max.fails if in-flight requests time out (#756)
- Count partially sent buffers as in-flight when timeout scanning (issue #756)
- Dont insert requests in front of partly sent SSL requests
- Reset partially sent buffer's offsets to 0 on disconnect (issue #756, #792)
- Dont prepend FLASH messages before partially sent buffers (possibly #756)
- Time out broker retry bufs (issue #776)
- Count topic_t app refcounts separately (issues #345, #821)
- Clean up OpenSSL global locks when all librdkafka instances are gone (issue #815)
- Fixed SSL broker_fail() arguments (issue #796)
- Improved purging of partition op & message queues to speed up termination
- Proper thread creation error propagation (issue #712)
- Forwarded queue's content was not properly moved to dest queue on fwd_set()
- Avoid endless loop when reconnect.backoff.jitter.ms is low (#779)
- Fix double free of topic.blacklist (issue #778)
- Fix SASL PLAIN auth (broken in 0.9.1)
- rdkafka_performance: could exit before all msgs sent when host was slow
- query_watermark_offsets() workaround for KAFKA-1588 (issue #700)
- Proper shrinking of rktparlist (issue #739 reported by @diazdiego86)
- query_watermark: Trigger leader query if leader not known (closes #694, fix by @JohanAR)
- Broadcast broker state changes (et.al) to expedite waiter wakeups (closes #695)
- Properly handle empty sasl.kerberos.kinit.cmd (seen in #688)
- rd_kafka_new: delete conf only on success (#836 by @julien-lecomte)
0.9.1
New major features
- Broker-based balanced High-Level KafkaConsumer (broker version >= 0.9)
- SASL support (client authentication) - requires Cyrus libsasl
- SSL support (client authentication, transport encryption) - requires OpenSSL
- Native Win32 support (Visual Studio)
- Broker-based quota support (KIP-13)
- Relative offsets support (KIP-31)
- Message timestamp support (KIP-32)
New enhancements
- Broker API version query support for brokers >=0.10.0 with fallbacks for older brokers. (KIP-35)
- api.version.request - enable/disable ApiVersionRequests
- api.version.fallback.ms - how long to use the fallback version after ApiVersionRequest failure
- broker.version.fallback - fallback version to adapt API usage to.
- More info: https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility
- Added broker connection max in-flight enforcement: max.in.flight.requests.per.connection
- Added config aliases "retries", "acks" (#157)
- request.required.acks/acks: added support for "all" (-1)
- Flag config properties now support explicit +addx,-dely and "all"
- Added rd_kafka_last_error() alternative to errno for legacy APIs (#571)
- Added rd_kafka_get_err_descs() to retrieve full list of error codes.
- Added rd_kafka_partitions_pause/_resume()
- Consumer: flush local fetchq for partition(s) and stop fetching
- Producer: dont send any ProduceRequests to broker for partition(s)
This is also used when a rebalance_cb is registered between the time of
rebalancing and until the application has called assign() to synchronized state.
Prevents app from receiving messages after a rebalance but before assign()
- Keep track of app_offset: last message+1 provided to application (issue #106). This is the "next offset" for the application.
- Added log.connection.close to turn off logging of broker-triggered disconnects (issue #437)
- Added rd_kafka_mem_free()
- Formal C and C++ API docs (doxygen)
- Added RdKafka::Message::topic_name().
Allows retreiving message topic (name) for auto consumed topics where we dont have a local Topic object - Add C++ Metadata API (@eugpermar and @incessantier)
- Changed topic.metadata.refresh.interval.ms to 5m, must be lower than server idle timeout
- Alias some new java client properties (bootstrap.servers, enable.auto.commit)
- Added topic-level "compression.codec" configuration
- Added "builtin.features" property to allow applications to query for built-in features (e.g., gzip, snappy, zlib)
- Add Consumer::OffsetTail(). (@bwelling)
- Removed enforce.isr.cnt - should be handled by broker now and request.required.acks=-1
- Added rd_kafka_topic_opaque()
- Added rd_kafka_*_conf_get() for querying conf property values on a conf object
- New high-level balanced KafkaConsumer (requires broker version >= 0.9)
- Add consumer queue support for C++. (@rthalley and @macisamuele)
- Fixed alignment errors on Sparc, et.al.
- Increased topic.metadata.refresh.interval.ms to the official default 10 minutes (#363)
- Added config property 'socket.blocking.max.ms' to control max socket poll time
- Added lazy producer message offset reporting
When 'produce.offset.report' is off, produced offsets will still be reported or the last message in each produced batch.
This means an application can still keep track of offsets (though a bit more roughly) without the cost of per-message offset reports. - Added rd_kafka_yield() to break out of callback loops
- Propagate UNKNOWN_TOPIC and UNKNOWN_PARTITION to consumers (issue #348)
- Handle broker name and nodeid updates (issue #343)
This also migrates bootstrap brokers to proper brokers if they can be exactly matched by name. - Added consume.callback.max.messages (issue #248)
- Added C++ Consumer::consume_callback as front-end of rd_kafka_consume_callback (kodekarl)
- Add a built in consistent hashing partitioner (@NZKoz)
- Add ability to specify a CRL for checking remote certificates (@vincentbernat)
Fixes
- Preserve configured partition.assignment.strategy order
- Final offset commits on consumer close() are now limited to session.timeout.ms
- rd_kafka_log_print should print to stderr, not stdout
- C++ unassign() assigned an empty list, which is not an unassign..
- Dont fail/retry unsent requests on broker failure, leave them in queue.
- Portability fixes (FreeBSD, Win32, MUSL libc, SunOS, AIX)
- C++: added virtual destructors to all classes that have a virtual function (@danny794)
- Consumer: fix race condition with start(logical_offset) quickly followed by seek(abs_offset)
- Maintain per-partition op-version in Fetch requests (issue #586)
- Richer CONFIGURATION.md format (supported values, value type)
- Fixed a number of possible hang-on-termination scenarios.
- Improved handling of failed broker requests (exposed by #529)
- Fixed cases where a timeout_ms of 0 would serve precisely timed events forever
- Fixed buffer retry counting
- Dont rely on partition leader broker to be set when querying metadata (issue #518)
- Ignore intermittent Topic Metadata failures while brokers are rebalancing. (issue #513)
- rd_kafka_topic_partition_available() returned 1 for leader-less partitions (issue #513). Regression since 0.8.6
- Backoff sleep on broker UP -> DOWN state transition was never called. This could trigger busy-looping when connecting to a broken broker.
- Added configurable reconnect delay with jitter: reconnect.backoff.jitter.ms (issue #403)
- Snappy endian swapping for older glibc 2.9 (issue #505, #407)
- Fix possible crash on buf retries when broker is unavailable
- Consumer errors are now rkmessage-based rather than error_cb
- Added RD_KAFKA_OFFSET_INVALID to indicate invalid offsets in various APIs
- Minimize default address caching time to 1s (issue #494)
- Added rd_kafka_errno() (issue #489)
- Proper generation and installing of pkg-config .pc files for both C and C++ libraries
- Report unsupported compression codecs (on consume) as CONSUMER_ERR ops
- RdKafka::Conf->set "socket_cb" and "open_cb" checked for wrong Conf type
- Deprecate rd_kafka_set_logger() in favour of rd_kafka_conf_set_log_cb()
- Dont reset partition count to zero on metadata topic error
A metadata partition count of 0 is returned by the broker with errors such as "No leader broker" for a topic, since these errors are
typically intermittent we hold on to the existing partition count. - Emit ALL_BROKERS_DOWN if none of the initial bootstrap brokers could be added.
- Consumer offset fixes (#352, #276, #227)
- Dont loose messages when partition goes away (and there is no UA partition)
Previously any messages in a partition's queue would be silently lost if a partition went away (and there was no UA partition).
These messages are now properly delivery reported with UNKNOWN_PARTITION. - Provide rkmessage->rkt topic in dr_msg_cb (#333)
- Fixed wait in rd_kafka_consume_callback_queue().
It returned immediately if no messages were available (timeout_ms was not used). - Fix occasional invalid json on statistics buffers (@alfred-landrum)
- dr_msg_cb: rkmessage->partition is now properly set (issue #233)
- Renamed position() -> committed() and added position() that returns next offset. According to the Java client.
- Workaround Java client and broker Offset metadata parsing bugs (issue #577)
- Added rd_kafka_query_watermark_offsets() and rd_kafka_get_watermark_offsets() (issue #503, #530)
- C++: Added commitSync and commitAsync APIs taking a list of offsets (issue #528)
- Added
enable.auto.offset.store
(issue #528) sasl.kerberos.kinit.cmd
now takes a command+args string template (issue #635)- Send consumer group user_data as empty instead of NULL to workarund broker 0.9.0.0 bug (issue #660)
0.8.6
Mainly a maintenance release with 15 enhancements and 25 fixes.
Enhancements
- NULL messages (and keys) are now supported (@secretmike)
- Switched to use zlib crc functions for increased performance (@tburmeister)
- Added 'group.id' config property to topic config (overrides global 'group.id')
- Include group.id in offset file (issue #199). Escape offset filenames.
- C++: Implemented Message::key() (@KodeKarl).
- Enabled OSX builds for travis-ci
- Added "consumer_lag", "lo_offset", "hi_offset", "stored_offset" to stats output.
- Added conf: delivery.report.only.error: only trigger dr_cb() for errored messages.
- Added "internal.termination.signal" property (default off)
- rdkafka_performance: added latency measurement mode (-l) and offset reporting (-O)
- rdkafka_performance: Added '-A ': write per-message latency to file
- rdkafka_performance: send NULL key when -s 0
- rdkafka_performance: print key
- rdkafka_performance: Make -c work in consumer mode to limit number of messages consumed
- rdkafka_performance: Print received message content when verbosity is >= 3
Fixes
- Offset storage fixes (issue #191)
- Extern C-ilize rdkafka.h (issue #216)
- Revert
request.required.acks
limit (-1..1) back to (-1..1000) (issue #208) - Avoid excessive topic leader queries when a broker goes down
- Renamed RdKafka::Producer::MSG_COPY and MSG_FREE to RK_MSG_COPY and RK_MSG_FREE (issue #198)
- rdkafka_performance: Better usage error reporting
- Avoid re-inclusion of rdkafkacpp.h (issue #195)
- rdkafka_performance: proper stats_fp closing
- Fix mismatches in broker_keep and broker_destroy that cause brokers to leak (#193) (@thelamb)
- Silence recurring broker resolve failures (#186)
- Added target dependencies for "make -j". (@KodeKarl)
- Dont log broker signal interrupts when terminating
- Fixed double-free of conf (defect added in ce8cd60)
- mklove update for mktemp (issue #176)
- Dont leak topic_conf memory when already added topic is passed to rd_kafka_topic_new()
- NULL conf pointer after free (@oldmantaiter)
- remove "executable" attribute from source files (@rgerhards)
- Dont require auto.commit.enable for stored offsetting (issue #171)
- Drop earlier (unwanted) messages from fetched messagesets
- Increased message.max.retries to allow long-time offline queueing.
- Library ordering (fixes Ubuntu ppc builds)
- Require linkage
- Make sure to store EOF offset on termination (issue #164)
- If RD_KAFKA_OFFSET_TAIL() rewinds past offset 0, cap it to 0.
- OSX requires linking required libs when creating dylib (issue #161)
- Use proper .dylib install_name on OSX
Thanks to @tburmeister, @thelamb, @KodeKarl, @rgerhards, @oldmantaiter, @secretmike
0.8.5
Enhancements and bugfixes:
- Dont link C++ library with C library - avoids symbol leakage
- Fixed endian.h check (for kFreeBSD, thanks to paravoid)
rd_kafka_topic_new()
was not completely thread-safe- Unlock timer lock when calling timer callback (and honour rk_terminate)
- Store EOF offset for empty partitions on termination (previously did not store any offset)
- Off-by-one error when calculating tv_nsec - by dalegaard
- Fix metadata cache purge when
topic.metadata.refresh.interval.ms=-1
(issue #156) - Allow 0 value for
fetch.error.backoff.ms
- Use a random broker for metadata requests
- Dont crazy-spin on
topic.metadata.refresh.interval.ms==-1
(issue #156) - Fixed
_BIG_ENDIAN
check on Solaris Sparc - thanks to shivashis - Now compiles on FreeBSD - thanks to rtyler
- Sparse metadata retrieval mode (consumes less bandwidth) (issue #138)
Config property:topic.metadata.refresh.sparse
= true or false - Added "protocol" debugging (sent and received message types)
0.8.4
The 0.8.4 release introduces the new C++ interface, a configure script for increased portability and a large number of enhancements and bug fixes.
New features and enhancement
- Native C++ interface - see src-cpp/rdkafkacpp.h
- ./configure support using the mklove framework: https://github.com/edenhill/mklove
- pkg-config rdkafka.pc file is now generated during build
- Broker-based offset storage using
OffsetCommit
andOffsetFetch
- Added Metadata API (issues #90 and #94)
- rdkafka_example: Added -L (list) mode to list metadata
- Added batch producer
rd_kafka_produce_batch()
for even more performant producing (issue #112) - Added
queued.max.messages.kbytes
(local consumer queue limit) (issue #81) - Added
enforce.isr.cnt
to locally enforce that this value is <= cluster ISR cnt (issue #91) - rdkafka_performance: Added
-G <pipe command>
to pipe stats output directly to command. - rdkafka_performance: Added
-r <msgs/s>
producer rate limiting - Added
broker.address.family
config property (limits broker connectivity to v4 or v6) - Added experimental msg_dr_cb() passing a complex message struct (NOT API SAFE just yet)
- Added rd_kafka_opaque() to return the opaque pointer set by set_opaque
- Added socket_cb and open_cb config properties (issue #77) (for CLOEXEC)
- Added RPM package spec file for building RPM package - see rpm/librdkafka.spec
- Added "msgs" to toppar stats output
- rdkafka_performance: added tabular output, and more..
- Solaris/SunOS support
- Bart Warmerdam added an example how to integrate with ZooKeeper in
examples/rdkafka_zookeeper_example.c
- Support for infinite message timeouts
message.timeout.ms=0
(thanks to Bart Warmerdam) - Added
socket.keepalive.enable
config property to enable SO_KEEPALIVE - Added
socket.max.fails
to disconnect from broker on request timeouts (defaults to 3) - Added
produce.offset.report
config property to propagate message offset of produced message back to application. - Added
RD_KAFKA_OFFSET_TAIL(cnt)
- start consumingcnt
messages from end. - New consumer queue re-routing to pull messages from multiple topics+partitions simultaneously on the same thread with one single call. (
rd_kafka_consume_start_queue()
)
Bug fixes
- All signals are now properly blocked by rdkafka threads - no need to set up SIGPIPE handlers
in the application anymore. - Fixed compilation errors and warnings on various platforms
- rdkafka_performance: use message.timeout.ms default value (issue #74)
- Added redundant but correctly spelled "committed_offset" field to stats (issue #80)
- Dont retry failed Fetch requests, wait for next intervalled Fetch (issue #84)
- rd_kafka_*conf_dump() did not properly return all types of values (namely string2int mappings)
- More consistent handling of unknown topics and partitions (issue #96)
- Handle transient partition_cnt=0 during topic auto creation (issue #99)
- Add pthread_attr_destroy() (thanks to Bart Warmerdam)
- Consuming from a cluster with more than 500 partitions would crash rdkafka, this is now fixed. (issue #111)
- Fixed MacOSX 10.9 compilation
- Use .dylib extension for shared library on MacOSX
- Fixed dead-lock race condition on metadata update (issue #131)
- Topics were not destroyed in example code (issue #130)
The OpenSuse build service is now used to ensure portability, please see
https://build.opensuse.org/package/show/home:edenhill/librdkafka
for build results and packages.
Sponsors
The fix for issue #131 (dead-lock race condition on metadata update) was sponsored by
the redBorder project (http://redBorder.org)
Big thanks goes to LonelyCoder (https://github.com/andoma) for his help in conceiving the C++ interface and throwing down Changs, sometimes simultaneously - kap kom kap!
0.8.3
New features and enhancements:
- Added
fetch.message.max.bytes
andreceive.message.max.bytes
properties. - Added rd_kafka_errno2err() for converting errno to proper rdkafka errors (issue #39)
- Trigger RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN error when all brokers are down (issue #64)
- Immediate failure on unknown topic (issue #39)
- rdkafka_example: Added raw output (-A) and quiet (-q) options, suitable for generic output to other programs.
- Added "rxcorriderrs" to stats output (unknown correlation ids)
Fixes:
- Receiving from multiple well fed partitions will no longer cause "Invalid message size .." errors.
- Topic refcount fix - rdkafka would not fully terminate under some circumstances (issue #72)
- rd_kafka_wait_destroyed() now also waits for all rk's to go away.
- Fix Cygwin compilation, again
- Proper errcode for broker transport failures, not just __FAIL (issue #64)
- Fixed out-of-order problem for produced messages under some circumstances
- Internal topic count was not properly decreased on topic destroy
- Plugged memory leak on exit: internal broker ops queue
- Added friendly hint when message.max.bytes is configured too low.
- Fixed duplicate broker names in broker fail logs
- message.send.max.retries did not have effect, no messages were retried (for issue #74)
- Avoid flooding the broker with topic-specific metadata requests when produce fails
0.8.2
New features:
- Added automatic consumer offset management (local file)
- The application no longer needs to install a SIGPIPE handler
- Added more configuration API functions (copy, destroy, dump, ...)
- examples: Added support for canonical "end","beginning","stored" to -o offset arg
- rdkafka_example: Added -X prop=val
- Now builds on MacOSX 10.9
- Now builds on RHEL 5
- Now builds on Cygwin
- Added -I (idle) switch to rdkafka_performance
Fixes:
- Proper librdkafka.a dependency for examples
- Added 'make check' target to check availability of all build artifacts
- Dont attempt to gzip compress 0 length message chunks (issue #60)
- The application no longer needs to install a SIGPIPE handler
- Compilation warning fixes on various platforms and gcc versions
- Allow producing and consuming empty messages (issue #60)
- rdkafka_performance: -X topic... properties did not work
- Fixed decoding of snappy-java compression framing (issue #55)
- Fixed flaky Gzip decompression (issue #54)
- Fail messages for unknown partitions immediately (issue #39)
- Clarify available compression support (none, gzip, snappy)
- Allow NULL 'conf' argument to rd_kafka_topic_new()
- Use 'void *' for produce payload type instead of 'char *'
- rdkafka_performance: Count immediate produce() failures as failed messages too
- Handle per topic+partition produce errors (issue #40)
- Print error string, not number, in metadata request failure log message
- Only generate statistics if statistics.interval.ms is non-zero
- Correctly calculate number of topics for filtered Metadata requests