storage: ensure stats are reported on idle replicas #30346
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a replica becomes idle (i.e. all its dataflows advance to the empty frontier), there might remain pending storage statistics that haven't been reported yet because the reporting interval hasn't been reached. When the worker thread then gets parked for an indefinite amount of time, those statistics also don't get reported for an indefinite amount of time. This commit ensures that pending storage statistics always get reported by imposing a limit on the time a worker thread is allowed to park, ensuring it wakes up again when the stats collection interval has elapsed.
teskje
force-pushed
the
storage-ensure-statistics-reporting
branch
from
3292954
to
67a3f14
Compare
teskje
changed the title
teskje
commented
Nov 7, 2024
|
|
||
| let mut last_stats_time: Option<Instant> = None; | ||
| // The last time we reported statistics. | ||
| let mut last_stats_time = Instant::now(); |
There was a problem hiding this comment.
This is a slight change to the previous behavior. I figured it's probably not important that we collect statistics immediately after the first worker step. If that's wrong I can change it back into an Option, just makes the code a bit more noisy.
petrosagg
approved these changes
Nov 10, 2024
|
TFTR! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a replica becomes idle (i.e. all its dataflows advance to the empty frontier), there might remain pending storage statistics that haven't been reported yet because the reporting interval hasn't been reached. When the worker thread then gets parked for an indefinite amount of time, those statistics also don't get reported for an indefinite amount of time.
This PR ensures that pending storage statistics always get reported by imposing a limit on the time a worker thread is allowed to park, ensuring it wakes up again when the stats collection interval has elapsed.
Relevant Slack thread.
Motivation
Part of https://github.com/MaterializeInc/database-issues/issues/7020
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.