SearchTagValuesV2 use protobuf internally instead of json to reduce latency #3731
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mdisibio
requested review from
joe-elliott,
annanay25,
mapno,
yvrhdn,
zalegrala,
electron0zero,
ie-pham and
stoewer
as code owners
mapno
reviewed
May 30, 2024
… on content type detection
mapno
approved these changes
May 30, 2024
This was referenced Jun 3, 2024
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
The frontend communicates with the queriers using json, and this is becomes a bottleneck for large autocomplete lookups. This PR switches /api/v2/search/tag/.../values to use http/protobuf instead. Testing locally and in a larger cell on a large lookup, the savings are around 30% user-facing latency to Grafana. Timings: 15s -> 10s, 5s -> 3s. We like graphs, so here is one showing the second set:
The encoding is implemented optionally where the querier will reply with proto only if the request header is
Accept: application/protobuf. Else it defaults to json. This helps with compatibility during rollouts of old/new pods, and also curl-ability.A couple future steps:
(1) It might be worth swapping over the remaining apis with large payloads like search/tags and metrics. But didn't want to expand scope.
(2) Even better would be preserve the protobuf all the way to Grafana like we do for trace lookup, but there is work needed to connect the response/request through the new async pipeline in the frontend, so the final response can be built with the appropriate encoding.
Which issue(s) this PR fixes:
Fixes n/a
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]