Skip to content
This repository was archived by the owner on Aug 23, 2023. It is now read-only.

re-evaluate kafka compression #312

Closed
Dieterbe opened this issue Sep 7, 2016 · 2 comments
Closed

re-evaluate kafka compression #312

Dieterbe opened this issue Sep 7, 2016 · 2 comments
Labels
perf

Comments

@Dieterbe
Copy link
Contributor

Dieterbe commented Sep 7, 2016 •
edited
Loading

now that sarama is patched to support compressed offsets.
want to understand cpu vs mem vs disk io, effects on kafka/metrictank performance better.
can easily be done with raintank-docker and fakemetrics.
@Dieterbe Dieterbe changed the title re-evaluate compression re-evaluate kafka compression Sep 9, 2016
@woodsaj
Copy link
Member

woodsaj commented Jan 6, 2017

The compression was fixed in Sarama.v1.10.1, which we are now using.

We can benchmark this now.

@Dieterbe Dieterbe added the perf label Jun 28, 2017
@Dieterbe
Copy link
Contributor Author

Dieterbe commented May 2, 2018 •
edited
Loading

with our MetricPoint format for 0.9 i did some testing between comp=none and comp=snappy.
snappy was the clear winner. #876 (comment) and so we default to it now.
in the future, once our formats stabilize we can potentially do more testing of other compression algorithms.

@Dieterbe Dieterbe closed this as completed May 2, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
perf
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Dieterbe @woodsaj