fix(swingset): better snapshot scheduling, do BOYD before each #7558
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
warner
force-pushed
the
7504-boyd-before-snapshot
branch
from
d80a8e0
to
96c70a5
Compare
|
oops, looks like I need to fix the rest of the swingset tests first, my apologies |
warner
force-pushed
the
7548-increase-snapshot-initial
branch
from
c51df0b
to
1afcfa6
Compare
warner
force-pushed
the
7548-increase-snapshot-initial
branch
from
1afcfa6
to
5e723fa
Compare
warner
force-pushed
the
7504-boyd-before-snapshot
branch
from
0562ec5
to
b022799
Compare
warner
force-pushed
the
7504-boyd-before-snapshot
branch
from
b022799
to
0bb9d02
Compare
This changes the snapshot scheduling logic to be more consistent. We still use `snapshotInitial` to trigger a snapshot shortly after worker initialization, and `snapshotInterval` to trigger periodic ones after that. However the previous code compared `snapshotInitial` to the absolute deliveryNum, which meant it only applied to the first incarnation, and would not attempt to take a snapshot shortly after upgrade, leaving the kernel vulnerable to replaying the long `startVat` delivery for a larger window than we intended. And `snapshotInterval` was compared against the difference between the latest transcript and the latest snapshot, which changed with the addition of the load-worker pseudo-entry. The new code uses `snapshotInitial` whenever there is not an existing snapshot (so the first span of *all* incarnations), and compares it against the length of the current span (so it includes all the pseudo-events). `snapshotInterval` is also compared against the length of the current span. The result is simpler and more predictable set of rules: * in the first span of each incarnation, trigger a snapshot once we have at least `snapshotInterval` entries * in all other spans, trigger once we have at least `snapshotInterval` In addition, when triggering a snapshot, we perform a BringOutYourDead delivery before asking the worker to save a snapshot. This gives us one last chance to shake out any garbage (making the snapshot as small as possible), and reduces the variation we might see forced GC that happens during snapshot write (any FinalizationRegistry callbacks should get run during the BOYD, not the save-snapshot). closes #7553 closes #7504
warner
force-pushed
the
7504-boyd-before-snapshot
branch
from
0bb9d02
to
ec16fa2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changes the snapshot scheduling logic to be more consistent. We still use
snapshotInitialto trigger a snapshot shortly after worker initialization, andsnapshotIntervalto trigger periodic ones after that.However the previous code compared
snapshotInitialto the absolute deliveryNum, which meant it only applied to the first incarnation, and would not attempt to take a snapshot shortly after upgrade, leaving the kernel vulnerable to replaying the longstartVatdelivery for a larger window than we intended. AndsnapshotIntervalwas compared against the difference between the latest transcript and the latest snapshot, which changed with the addition of the load-worker pseudo-entry.The new code uses
snapshotInitialwhenever there is not an existing snapshot (so the first span of all incarnations), and compares it against the length of the current span (so it includes all the pseudo-events).snapshotIntervalis also compared against the length of the current span.The result is simpler and more predictable set of rules:
snapshotIntervalentriessnapshotIntervalIn addition, when triggering a snapshot, we perform a BringOutYourDead delivery before asking the worker to save a snapshot. This gives us one last chance to shake out any garbage (making the snapshot as small as possible), and reduces the variation we might see forced GC that happens during snapshot write (any FinalizationRegistry callbacks should get run during the BOYD, not the save-snapshot).
closes #7553
closes #7504