Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add zstd support #55

Open
pikrzysztof opened this issue May 30, 2019 · 2 comments · May be fixed by #94
Open

Add zstd support #55

pikrzysztof opened this issue May 30, 2019 · 2 comments · May be fixed by #94

Comments

@pikrzysztof
Copy link

zstd is supported natively as of kafka 2.1. zstd bindings are available here. I'm not sure about the license though.

It would be great to have it supported by this library.

@zmstone
Copy link
Contributor

zmstone commented May 30, 2019

I have been a bit slow on following kafka changes.
Latest verified kafka version in this repo's tests is 1.1.

Although we have a snappy lib here,
I am reluctant pulling in another dependency to this repo,
since zstd as well as lz4 are optional.

I prefer to have compression backend configurable.

e.g. we can do:

zstd_decompress(BinData) ->
  {ok, Module} = application:get_env(kafka_protocol, zstd_compression_module),
  Module:decompress(BinData).
zstd_compress(IoData) ->
  {ok, Module} = application:get_env(kafka_protocol, zstd_compression_module),
  Module:compress(IoData).

Or to resolve at compile time, we can override erl_opts with
{d, 'KAFKA_PROTOCOL_ZSTD_MODULE', the_zstd_module} in addition:

-ifdef(KAFKA_PROTOCOL_ZSTD_MODULE).
zstd_decompress(BinData) ->
  ?KAFKA_PROTOCOL_ZSTD_MODULE:decompress(BinData).
zstd_compress(IoData) ->
  ?KAFKA_PROTOCOL_ZSTD_MODULE:compress(IoData).
-else.
zstd_decompress(_BinData) -> error(no_support).
zstd_compress(_IoData) -> error(no_support).
-endif

Then the dependency is put to user's project and we do not have to worry about licensing either.

@bucko909
Copy link

bucko909 commented Jun 3, 2019

I think either of those is likely acceptable. The second solution is probably the most performant as it doesn't have to round-trip ets.

@zmstone zmstone linked a pull request Jul 11, 2021 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants