Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce allocs by pooling gzip.Writer objects #3

Merged
merged 2 commits into from
Jul 18, 2018

Conversation

shanson7
Copy link
Contributor

Pretty simple change to pool for reuse. This came about because an instance of metric that was running had a pprof that looked like:

(pprof) top5
Showing nodes accounting for 21093.71GB, 64.73% of 32586.37GB total
Dropped 1236 nodes (cum <= 162.93GB)
Showing top 5 nodes out of 131
      flat  flat%   sum%        cum   cum%
10250.14GB 31.46% 31.46% 10888.51GB 33.41%  github.com/grafana/metrictank/vendor/github.com/klauspost/compress/flate.NewWriter /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/klauspost/compress/flate/deflate.go
 3774.92GB 11.58% 43.04%  5570.56GB 17.09%  github.com/grafana/metrictank/vendor/github.com/eapache/go-xerial-snappy.Decode /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/eapache/go-xerial-snappy/snappy.go
 3003.95GB  9.22% 52.26%  3003.95GB  9.22%  github.com/grafana/metrictank/vendor/github.com/Shopify/sarama.(*partitionConsumer).parseMessages /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/Shopify/sarama/consumer.go
 2269.06GB  6.96% 59.22%  9842.42GB 30.20%  github.com/grafana/metrictank/vendor/github.com/Shopify/sarama.(*MessageBlock).decode /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/Shopify/sarama/message_set.go
 1795.65GB  5.51% 64.73%  1795.65GB  5.51%  github.com/grafana/metrictank/vendor/github.com/golang/snappy.Decode /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/golang/snappy/decode.go

Showing that flate.NewWriter was by far the largest source of allocations in our process. I tracked that to this component. I have already released this change to our production cluster and have seen a significant reduction and flate.NewWriter has completely dropped from the pprof output.

Benchcmp results:

benchmark           old ns/op     new ns/op     delta
BenchmarkGzip-8     207762        66689         -67.90%

benchmark           old allocs     new allocs     delta
BenchmarkGzip-8     50             34             -32.00%

benchmark           old bytes     new bytes     delta
BenchmarkGzip-8     1005656       4229          -99.58%

Copy link

@replay replay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks

@woodsaj woodsaj merged commit 25a564a into raintank:master Jul 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants