-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
librdkafka failing to disable compression when broker does not support it. #2480
Comments
Great analysis! Can you reproduce this with |
./rdkafka_simple_producer broker.com:9092 topic%7|1567521398.772|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.1.0-57-gff697f-dirty (0x10100ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, GCC GXX PKGCONFIG INSTALL GNULD LDS LIBDL PLUGINS ZLIB SSL SASL_CYRUS HDRHISTOGRAM SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x10) |
I'm having an issue like this with kafka 10.2.1-r2 and librdkafka 1.1.0 (I suspect the offending code was added in fb6ae3d, so that's an 1.0.0 rc). I broke the program at src/rdkafka_msgset_writer.c:201
This means min_ApiVersion should be 2 , but line 177. pp upgrades to min_ApiVersion 3. The assert thus fails. I think kafka 10.2 with its api version actually speaks lz4, so the test is at odds with reality. |
Actually, I went ahead. This works for kafka 10.2. Can't say for which versions it will not work.
|
./configure --disable-zstd --disable-lz4 --disable-lz4-ext --disable-ssl Configuration summary: Disable LZ4, but still compile and link these files |
Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ
Description
LZ4 compression enabled in librdkafka causes assert when producing to a broker that does not support LZ4
When librdkafka checks if compression is enabled on broker before producing events, there is the following check in librdkafka/src/rdkafka_msgset_writer.c:
For kafka version 0.10.1.1 and librdkafka version 1.0.0 the following is returned for compression feature value and rkb_features:
msetw->msetw_rkb->rkb_features = 511
compr_req[msetw->msetw_compression].feature = 0x80 -- 128
That means that the check:
is resolved to false. Meaning it believes LZ4 compression is a feature the broker has enabled.
Now,
also resolves to false. This means that compression remains enabled.
for min/max api version supported returns min = 0, max =2.
For LZ4:
Which means that the min API version required is 3.
At the end of function:
rd_kafka_msgset_writer_select_MsgVersion there is an assert that happens because of the following check:
where msetw->msetw_ApiVersion is -1 becuase the min_ApiVersion is 3 (set by LZ4 compression)
I believe this check is wrong:
If the min API version required parameter (set current as 0) is changed to
compr_req[msetw->msetw_compression].ApiVersion, then the function would return a -1, disabling compression.
Right now, it returns 2, even though apiversion 2 does not support compression.
There might also be a bug with the broker features that are enabled. I can create a pull request for this if what is outlined above is actually the bug, but I am not sure what exactly the checks should resolve to!
How to reproduce
enable LZ4 compression on librdkafka version 1.0.0 and produce to a broker kafka version 0.10.1.1
IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
debug=..
as necessary) from librdkafkaThe text was updated successfully, but these errors were encountered: