Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use snappy compression for internal gRPC communication. #3368

Merged
merged 5 commits into from
Sep 16, 2019

Conversation

danielmai
Copy link
Contributor

@danielmai danielmai commented May 3, 2019

Closes #3358.


This change is Reviewable

@danielmai danielmai requested review from manishrjain and a team as code owners May 3, 2019 21:31
Copy link
Contributor

@codexnull codexnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 17 files at r1.
Reviewable status: 2 of 17 files reviewed, all discussions resolved (waiting on @manishrjain)

Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 16 of 17 files at r1, 1 of 1 files at r2.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @danielmai and @manishrjain)


conn/pool.go, line 153 at r2 (raw file):

			grpc.MaxCallRecvMsgSize(x.GrpcMaxSize),
			grpc.MaxCallSendMsgSize(x.GrpcMaxSize),
			grpc.UseCompressor((snappyCompressor{}).Name())),

You don't need the parentheses

Example: https://play.golang.org/p/T5DzEZhF_x8


conn/snappy.go, line 1 at r2 (raw file):

// Copyright 2017 The Cockroach Authors.

Is the copyright message right?


conn/snappy.go, line 25 at r2 (raw file):

)

// NB: The encoding.Compressor implementation needs to be goroutine

nit: what is NB supposed to stand for?

Copy link
Contributor Author

@danielmai danielmai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @manishrjain and @martinmr)


conn/pool.go, line 153 at r2 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…

You don't need the parentheses

Example: https://play.golang.org/p/T5DzEZhF_x8

Nice catch. But I don't feel that it's necessary to change the source from CockroachDB.


conn/snappy.go, line 1 at r2 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…

Is the copyright message right?

Yes. This file is from CockroachDB: https://github.com/cockroachdb/cockroach/blob/856ba9108f112f85d406bbe88d2208651859336e/pkg/rpc/snappy.go. It's the bare minimum needed to implement a gRPC compressor using snappy compression. Aside from the package name, everything else is the same.


conn/snappy.go, line 25 at r2 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…

nit: what is NB supposed to stand for?

I'm not too sure, but it's probably this: https://en.wikipedia.org/wiki/Nota_bene. Basically, "pay attention". Again, I didn't change it from the original source.

Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @manishrjain)

@campoy
Copy link
Contributor

campoy commented Sep 13, 2019

Hey @danielmai, what happened to this PR?

@danielmai
Copy link
Contributor Author

@campoy This PR in its current state would require all instances to use snappy compression for internal communication. Ideally, it'd be better if instances opted-in to use snappy compression for internal traffic if available to allow for rolling upgrades.

This PR was created on the hunch that there'd be improvements for resource usage or performance with compressed payloads, but there was no investigation to confirm this.

It'd be great if someone else could pick up where I left off here.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 16 of 17 files at r1, 1 of 1 files at r2.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved

@danielmai danielmai merged commit c6ecca1 into master Sep 16, 2019
@danielmai danielmai deleted the danielmai/internal-grpc-snappy branch September 16, 2019 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Enable gRPC compression for internal gRPC communication
5 participants