Debounce shard pruning request #5374
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdettai
force-pushed
the
sqs-garbage-collection
branch
from
89d09bd to
b680d1e
Compare
rdettai
force-pushed
the
prune-debounce
branch
from
d8a89fc to
b08930c
Compare
rdettai
force-pushed
the
prune-debounce
branch
2 times, most recently
from
db09a8e to
cac6c90
Compare
rdettai
force-pushed
the
sqs-garbage-collection
branch
from
43b202f to
7c543ee
Compare
rdettai
force-pushed
the
prune-debounce
branch
from
cac6c90 to
5ee1bd7
Compare
rdettai
force-pushed
the
sqs-garbage-collection
branch
from
7c543ee to
d1885af
Compare
rdettai
force-pushed
the
prune-debounce
branch
from
5ee1bd7 to
3229965
Compare
fulmicoton
reviewed
Sep 17, 2024
fulmicoton
reviewed
Sep 17, 2024
fulmicoton
reviewed
Sep 18, 2024
fulmicoton
reviewed
Sep 18, 2024
|
|
||
| let debouncer = self | ||
| .prune_shards_debouncers | ||
| .entry(( |
There was a problem hiding this comment.
one bummer is that this map is evergrowing.
fulmicoton
reviewed
Sep 18, 2024
fulmicoton
reviewed
Sep 18, 2024
fulmicoton
reviewed
Sep 18, 2024
rdettai
force-pushed
the
prune-debounce
branch
2 times, most recently
from
243b1d7 to
d703a8b
Compare
rdettai
commented
Sep 18, 2024
There was a problem hiding this comment.
No real change here, just simplifying the trait bounds for readability
fulmicoton
reviewed
Sep 19, 2024
rdettai
force-pushed
the
prune-debounce
branch
from
a96677b to
4da6215
Compare
fulmicoton
approved these changes
Sep 19, 2024
rdettai
force-pushed
the
sqs-garbage-collection
branch
from
fbb8916 to
a600c3d
Compare
rdettai
force-pushed
the
prune-debounce
branch
from
44290ef to
5835f81
Compare
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Instead of running a separate process to clean up the shard table, we initiate the cleanup requests from the files sources. To avoid an increasing number of requests when the number of pipelines grows the requests are debounced in the control plane.
How was this PR tested?
Integration tests