Cache vertical shards in query frontend #5648
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fpetkovski
commented
Aug 26, 2022
pkg/queryfrontend/roundtrip_test.go
Outdated
|
|
||
| // Allow other requests to execute | ||
| lock.Unlock() | ||
| <-time.After(200 * time.Millisecond) |
There was a problem hiding this comment.
The query frontend fanout seems to cancel all requests as soon as one request fails. Because of that, it is hard to provoke partial failures since the first failure will cause all parallel requests to fail.
This like adds a delay to failures so that successful requests can succeed, but it can still lead to flakes. Not sure if there is a better option to test this scenario.
There was a problem hiding this comment.
Mhm, maybe we could check if numFailures == 0 { waitForAnotherRequestToCome(); } somehow? Perhaps a shared sync.Cond could be used?
There was a problem hiding this comment.
This is a neat trick, seems to be a much better solution 👍
There was a problem hiding this comment.
Hm, looks like it's not possible to add this coordination here because we need to release other threads when the sharding middleware has seen a successful response. If we release them here, failed requests can still complete before successful ones.
fpetkovski
force-pushed
the
vertical-sharding-caching
branch
from
f25e569
to
445e2d0
Compare
yeya24
previously approved these changes
Aug 26, 2022
GiedriusS
reviewed
Aug 29, 2022
pkg/queryfrontend/roundtrip_test.go
Outdated
|
|
||
| testutil.Equals(t, tc.expected, *res) | ||
|
|
||
| //if *res > tc.expected { |
pkg/queryfrontend/roundtrip_test.go
Outdated
|
|
||
| // Allow other requests to execute | ||
| lock.Unlock() | ||
| <-time.After(200 * time.Millisecond) |
There was a problem hiding this comment.
Mhm, maybe we could check if numFailures == 0 { waitForAnotherRequestToCome(); } somehow? Perhaps a shared sync.Cond could be used?
The vertical sharding middleware is currently executed after the caching middleware. Because of this, individual vertical shards are not getting cached when the succeed. Caching is only done when the entire requests including all shards complete successfully. This commit moves the vertical sharding middleware before the caching middleware. It also modifies caching keys to contain the total shards and the shard number so that each vertical shard gets an independent caching key. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
fpetkovski
force-pushed
the
vertical-sharding-caching
branch
from
348a8b3
to
cd96510
Compare
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
fpetkovski
force-pushed
the
vertical-sharding-caching
branch
from
cd96510
to
9962948
Compare
prajain12
pushed a commit
to prajain12/thanos
that referenced
this pull request
Sep 6, 2022
* Cache vertical shards in query frontend The vertical sharding middleware is currently executed after the caching middleware. Because of this, individual vertical shards are not getting cached when the succeed. Caching is only done when the entire requests including all shards complete successfully. This commit moves the vertical sharding middleware before the caching middleware. It also modifies caching keys to contain the total shards and the shard number so that each vertical shard gets an independent caching key. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Adjust cache key tests Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Remove source of flakiness by using sync.Cond Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Prakul Jain <prakul.jain@udaan.com>
prajain12
pushed a commit
to prajain12/thanos
that referenced
this pull request
Sep 6, 2022
* Cache vertical shards in query frontend The vertical sharding middleware is currently executed after the caching middleware. Because of this, individual vertical shards are not getting cached when the succeed. Caching is only done when the entire requests including all shards complete successfully. This commit moves the vertical sharding middleware before the caching middleware. It also modifies caching keys to contain the total shards and the shard number so that each vertical shard gets an independent caching key. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Adjust cache key tests Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Remove source of flakiness by using sync.Cond Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Prakul Jain <prakul.jain@udaan.com>
prajain12
pushed a commit
to prajain12/thanos
that referenced
this pull request
Sep 6, 2022
* Cache vertical shards in query frontend The vertical sharding middleware is currently executed after the caching middleware. Because of this, individual vertical shards are not getting cached when the succeed. Caching is only done when the entire requests including all shards complete successfully. This commit moves the vertical sharding middleware before the caching middleware. It also modifies caching keys to contain the total shards and the shard number so that each vertical shard gets an independent caching key. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Adjust cache key tests Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Remove source of flakiness by using sync.Cond Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Prakul Jain <prakul.jain@udaan.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The vertical sharding middleware is currently executed after the
caching middleware. Because of this, individual vertical shards are
not getting cached when the succeed. Caching is only done when the
entire requests including all shards complete successfully.
This commit moves the vertical sharding middleware before the caching
middleware. It also modifies caching keys to contain the total shards
and the shard number so that each vertical shard gets an independent
caching key.
Changes
Enable caching in query frontend for individual vertical shards.
Verification
Added unit tests and tests locally by purposefully causing failures in queriers.