Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Upgrade to google.golang.org/grpc v1.66.2 #9401

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

aknuds1
Copy link
Contributor

@aknuds1 aknuds1 commented Sep 25, 2024

What this PR does

Upgrade google.golang.org/grpc from v1.65.0 to v1.66.2. The performance regression referenced from dskit's reversal to v1.65.0 is fixed in v1.66.2, and the potential functional regression also referenced in said reversal is Loki specific.

What we found out is that v1.66 fixes a serious performance bottleneck in conjunction with compression; the decompress function is optimized through memory pooling, whereas v1.65.0 would use io.ReadAll and cause a lot of allocations during decompression. As a result, we saw ingester CPU usage increase by ~40% when enabling gRPC compression. The increase is now much more modest, maybe ~7% based on observations.

Also upgrade github.com/googleapis/enterprise-certificate-proxy to v0.3.4, to fix a checksum issue in v0.3.3 (v0.3.4 was released to take care of this). With v0.3.3, building on OS X fails:

✗ go mod tidy && go mod vendor
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.3
verifying github.com/googleapis/enterprise-certificate-proxy@v0.3.3: checksum mismatch
	downloaded: h1:G6q7VHBoU74wQHXFsZSLMPl0rFw0ZDrlZ3rt6/aTBII=
	go.sum:     h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

Which issue(s) this PR fixes or relates to

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@aknuds1 aknuds1 marked this pull request as ready for review September 25, 2024 07:54
@aknuds1 aknuds1 requested review from stevesg, grafanabot and a team as code owners September 25, 2024 07:54
@aknuds1 aknuds1 marked this pull request as draft September 25, 2024 07:54
@aknuds1 aknuds1 marked this pull request as ready for review September 25, 2024 07:56
@aknuds1
Copy link
Contributor Author

aknuds1 commented Sep 25, 2024

Some tests are breaking for this upgrade, have to fix them.

@aknuds1 aknuds1 marked this pull request as draft September 25, 2024 08:17
@aknuds1 aknuds1 changed the title Upgrade to google.golang.org/grpc v1.66.2 WIP: Upgrade to google.golang.org/grpc v1.66.2 Sep 25, 2024
@aknuds1
Copy link
Contributor Author

aknuds1 commented Sep 25, 2024

Tests currently pass, but that is through undoing some memory unsafe optimizations in the mimirpb package. We want to investigate whether the optimizations can safely be retained.

Also upgrade github.com/googleapis/enterprise-certificate-proxy to v0.3.4, to fix a checksum issue in v0.3.3.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant