feat: Support state-sync #7225

mhofman · 2023-03-23T05:10:44Z

Review commit-by-commit

closes: #5542
closes: #3769
closes: #6563
closes: #5934

Description

This PR builds on #7115, #7026 and agoric-labs/cosmos-sdk#297 to implement state-sync in the cosmos and cosmic-swingset layers.

It takes over the cosmos state-sync process which is normally started in a go-routine without synchronization. Because the swingstore DB is not capable of doing a read at arbitrary historical savepoints, we need to introduce a synchronization with block commits.

When we reach a snapshot height, we first initiate a swingstore export before proceeding with the normal cosmos export process. We also add a synchronization step to stall the subsequent block commit until the swingstore export has been reported as having started. Once the swingstore export is in progress we can rely on its read transaction to proceed processing blocks, even if the export process spans multiple blocks.

On the cosmic-swingset side, we add a tool that leverages the new swingstore export logic to save the exported artifacts to disk. We also use the streaming export of swingstore data during block execution to save the main export data in vstorage under a new path. Conversely on the import side, we add a tool that reads a disk based export to produce an exporter that complies with the new swingstore import logic. These tools are usable independent of the state-sync logic. For state-sync they are driven by "blocking messages" sent by the golang side.

On the cosmos side, we implement the ExtensionSnapshotter interface. During snapshot taking, the extension running in the snapshot goroutine will request the path to the exported data from javascript and block until it is available. Then it will create a proto message for each artifact present in the exported data. On the restore side, we recreate a disk-based swingstore export by exporting the swingstore "export data" from vstorage (which was previously restored by state-sync), and the artifacts from the extension state-sync payloads. We then send a message to JavaScript to import this data into the expected swingstore location.

Security Considerations

The artifact verification on restore will be performed by the swingstore import logic. The "export data" is the root of trust and must be from a trusted source. Since it is extracted from the verified cosmos DB, it is considered trusted.

Scaling Considerations

The export process is performed as much as possible in parallel of normal execution. The artifact export is spawned in a sub process, and the streaming export data is buffered and sent to golang after the kernel has finished processing it current I/O, which should put it at idle time while it awaits for a syscall or delivery result.

Documentation Considerations

Nothing special besides normal cosmos state-sync instructions.

Testing Considerations

Tested manually extensively, including introducing synthetic delays in the snapshot process to test synchronization, or edge conditions such as overlapping snapshots.

Updated the deployment integration test to have the validators produce state-sync artifacts and have the loadgen follower start from a state-sync snapshot.

We should increase coverage of the edge cases through unit tests. See #7349

mhofman · 2023-03-23T05:12:17Z

I created this PR as draft since I'm still working on tests, but please start reviewing as the implementation is fully ready. I will not rebase this PR until approved.

datadog-full-agoric · 2023-03-23T05:20:47Z

Datadog Report

Branch report: mhofman/5542-export-state-sync
Commit report: b469d2d

❌ agoric-sdk: 2 Failed (0 Known Flaky), 0 New Flaky, 3469 Passed, 53 Skipped, 22m 25.99s Wall Time

❌ Failed Tests (2)

state › in-memory transcriptStore read/write - agoric.swing-store - Details

Expand for error

 ---
     name: AssertionError
     assertion: deepEqual
     values:
       'Difference (- actual, + expected):': >2-
           [
             [
         -     [
 
         -       'transcript.empty.current',
 ...

state › persistent transcriptStore read/write - agoric.swing-store - Details

Expand for error

 ---
     name: AssertionError
     assertion: deepEqual
     values:
       'Difference (- actual, + expected):': >2-
           [
             [
         -     [
 
         -       'transcript.empty.current',
 ...

codecov · 2023-03-23T05:22:16Z

Codecov Report

Merging #7225 (b86ae62) into master (f4de2ed) will decrease coverage by 0.40%.
The diff coverage is 26.32%.

❗ Current head b86ae62 differs from pull request most recent head c692911. Consider uploading reports for the commit c692911 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7225      +/-   ##
==========================================
- Coverage   71.01%   70.61%   -0.40%     
==========================================
  Files         450      461      +11     
  Lines       86398    89298    +2900     
  Branches        3        3              
==========================================
+ Hits        61358    63062    +1704     
- Misses      24974    26169    +1195     
- Partials       66       67       +1

Impacted Files	Coverage Δ
golang/cosmos/app/app.go	`0.45% <0.00%> (-0.02%)`	⬇️
golang/cosmos/daemon/cmd/root.go	`61.83% <0.00%> (+0.59%)`	⬆️
golang/cosmos/x/swingset/abci.go	`0.00% <0.00%> (ø)`
golang/cosmos/x/swingset/keeper/keeper.go	`11.87% <0.00%> (-0.10%)`	⬇️
golang/cosmos/x/swingset/keeper/snapshotter.go	`0.00% <0.00%> (ø)`
...ges/cosmic-swingset/src/helpers/bufferedStorage.js	`75.25% <ø> (ø)`
...kages/cosmic-swingset/src/helpers/is-entrypoint.js	`100.00% <ø> (ø)`
...smic-swingset/src/helpers/json-stable-stringify.js	`71.27% <ø> (ø)`
packages/cosmic-swingset/src/helpers/make-queue.js	`70.45% <ø> (ø)`
golang/cosmos/x/swingset/types/swingset.pb.go	`0.53% <0.53%> (+<0.01%)`	⬆️
... and 10 more

... and 24 files with indirect coverage changes

mhofman · 2023-03-24T02:12:01Z

@michaelfig I'd like you to be the primary reviewer
@JimLarson you always give great feedback on golang style and I expect you'll find plenty of things for me to correct. I also want to make sure you don't see any problem in the cosmos sdk integration I had to do.
@warner (or @gibson042), this is more of an FYI, don't feel obligated to review.

michaelfig · 2023-03-26T21:50:44Z

@michaelfig I'd like you to be the primary reviewer

Gladly! Can you please rebase your fixup!s now that the tests are passing if you want me to review commit-by-commit?

mhofman · 2023-03-26T23:13:32Z

The fixups are pretty minimal, but if you prefer, I'll do a clean rebase.

FYI @warner @JimLarson

michaelfig

I made a first pass. Looking good so far, but I'd like a few random things addressed, and will wait for other reviewers before approving.

golang/cosmos/app/app.go

golang/cosmos/x/swingset/keeper/keeper.go

golang/cosmos/x/swingset/keeper/snapshotter.go

packages/cosmic-swingset/src/helpers/swingstore-exporter.js

packages/cosmic-swingset/src/helpers/swingstore-importer.js

michaelfig · 2023-03-26T23:56:18Z

packages/cosmic-swingset/src/import-kernel-db.js

+import fsPromisesPower from 'fs/promises';
+import pathPower from 'path';
+
+import { Fail } from '@agoric/assert';


Suggested change

import { Fail } from '@agoric/assert';

const { Fail } = assert;

Is that the recommend style? Why not use the import form? I'm personally not a major fan of relying on globals that are not part of the JS standard library

The recommended style has vacillated over time. globalThis.assert is supplied by the SES shim, in the same way that globalThis.harden is. @agoric/assert is just a thin wrapper around that global.

Personally, I think the module import is more legible, but the direct manipulation of the global costs less overhead. @erights may have more of an opinion.

FWIW, this issue isn't decided:

migrate to @endo/errors package #5672

Yeah, we never did settle this. The fact that @agoric/assert, as indicated by the @agoric/ part, is in agoric-sdk only, and so not available in the endo repository, is for me the main reason why I continue to prefer the assignment form. It works everywhere we use Hardened JS.

If this package were migrated to be @endo/assert such that everything above SES could use it, then the points @turadg makes in that thread on the advantages of the import style would, for me, rise to the remaining important difference, causing me to switch to the import style.

packages/cosmic-swingset/src/import-kernel-db.js

michaelfig · 2023-03-27T00:13:00Z

The fixups are pretty minimal, but if you prefer, I'll do a clean rebase.

No need to on my account. I was able to see what I needed to do a review.

mhofman · 2023-03-27T18:34:52Z

I just force pushed the branch to remove the 5 commits related to debug/testing features that were sitting atop the initial PR, but below the initial fixups. Here is what was removed (and were never planned to be merged anyway): e68df6d..b5745e2

mhofman · 2023-03-28T07:56:16Z

I think I've addressed all actionable review feedback so far, or left comments where I didn't make any changes.

FYI, given the extent of fixup commits, I'm maintaining a parallel branch where all those are squashed into their respective target: 5542-export-state-sync-fixedup

PTAL @michaelfig

golang/cosmos/x/swingset/keeper/snapshotter.go

michaelfig · 2023-03-28T16:36:59Z

packages/cosmic-swingset/src/import-kernel-db.js

+import fsPromisesPower from 'fs/promises';
+import pathPower from 'path';
+
+import { Fail } from '@agoric/assert';


The recommended style has vacillated over time. globalThis.assert is supplied by the SES shim, in the same way that globalThis.harden is. @agoric/assert is just a thin wrapper around that global.

Personally, I think the module import is more legible, but the direct manipulation of the global costs less overhead. @erights may have more of an opinion.

michaelfig · 2023-03-28T16:57:08Z

I think I've addressed all actionable review feedback so far, or left comments where I didn't make any changes.

I'm pretty happy about where this is going, and I'm fine with you autosquashing before you mark the PR ready for review. I would encourage @JimLarson and @gibson042 not to begin review until the PR is out of draft.

mhofman · 2023-03-28T18:13:39Z

Will extract a few tangential commits out in a separate PR, and re-request review once ready.

mhofman · 2023-03-31T15:44:09Z

@gibson042 @JimLarson I've just rebased this on master after extracting a couple housekeeping changes in separate PRs. While it's still missing the unit tests for the state-sync logic, I believe this is ready for review at this point, and will take out of draft to make it explicit.

mhofman · 2023-03-31T15:44:50Z

Reminder, please review commit-by-commit!

The activityhash is now part of the swingstore export

This reverts commit fedb533.

mhofman · 2023-04-19T01:43:23Z

@Mergifyio requeue

mergify · 2023-04-19T01:43:26Z

requeue

☑️ This pull request is already queued

mhofman · 2023-04-19T02:33:40Z

@Mergifyio refresh

mergify · 2023-04-19T02:33:44Z

refresh

✅ Pull request refreshed

mhofman requested review from warner, michaelfig and JimLarson March 23, 2023 05:10

mhofman added the force:integration Force integration tests to run on PR label Mar 23, 2023

michaelfig reviewed Mar 27, 2023

View reviewed changes

mhofman force-pushed the mhofman/5542-export-state-sync branch from e68df6d to b5745e2 Compare March 27, 2023 18:31

michaelfig reviewed Mar 28, 2023

View reviewed changes

mhofman removed request for warner and JimLarson March 28, 2023 18:12

This was referenced Mar 28, 2023

refactor: move around some helpers before state-sync #7258

Merged

Various swing-store fixes #7259

Merged

mhofman force-pushed the mhofman/5542-export-state-sync branch from c692911 to a2587ba Compare March 31, 2023 15:42

mhofman requested review from gibson042 and JimLarson March 31, 2023 15:44

mhofman marked this pull request as ready for review March 31, 2023 15:44

This was referenced Apr 6, 2023

Add unit tests for state-sync #7349

Closed

state sync spike to put swing-store data in IAVL tree on periodic basis #5934

Closed

mhofman added 17 commits April 19, 2023 01:00

feat(cosmic-swingset): remove unnecessary explicit activityhash

5dcc44d

The activityhash is now part of the swingstore export

feat(vstorage): Export sub tree

726e188

feat(cosmic-swingset): basic snapshot wiring

b1072d8

Revert "fix: disable state-sync by default, until we've implemented it"

30340fb

This reverts commit fedb533.

fix(cosmic-swingset): shutdown controller on exit

c5410dd

feat(cosmic-swingset): add kernel-db exporter

df304d5

feat(cosmic-swingset): wire snapshot taking in chain-main

22bc1d5

feat(cosmic-swingset): execute export in a subprocess

05e2b15

feat(cosmic-swingset): explicit verbose option for export db

a2dabd1

feat(cosmic-proto): add state-sync artifacts proto

4cc25f5

feat(cosmos): wire swingset SnapshotExtension

7b0d99e

feat(cosmic-swingset): add kernel-db importer

00fab12

feat(cosmos): wire swingset RestoreExtension

c94e49d

feat(cosmic-swingset): wire snapshot restoring in chain-main

9d053b7

feat(deployment): Enable state-sync on validator nodes

2faa1fb

feat(ci): Use state-sync in deployment loadgen test

27b97c1

feat(cosmos): Export KVData as artifact

283fded

mhofman force-pushed the mhofman/5542-export-state-sync branch from 3c3a9d3 to 283fded Compare April 19, 2023 01:00

mergify bot merged commit e658480 into master Apr 19, 2023

mergify bot deleted the mhofman/5542-export-state-sync branch April 19, 2023 02:37

mhofman mentioned this pull request Apr 20, 2023

feat: tweak state-sync logging #7468

Merged

warner mentioned this pull request May 12, 2023

tools to export/import swing-store to/from genesis block #6527

Closed

mhofman mentioned this pull request May 26, 2023

Non traumatic major XS upgrades #7855

Open

mhofman mentioned this pull request Jul 10, 2023

Migrate swing-store shadow copy to x/swingset cosmos store #8031

Closed

mhofman mentioned this pull request Aug 19, 2023

Persistent cosmic-swingset state #8222

Open

mhofman mentioned this pull request Mar 18, 2024

Implement agd snapshots export #9100

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support state-sync #7225

feat: Support state-sync #7225

mhofman commented Mar 23, 2023 •

edited

Loading

mhofman commented Mar 23, 2023

datadog-full-agoric bot commented Mar 23, 2023 •

edited

Loading

codecov bot commented Mar 23, 2023 •

edited

Loading

mhofman commented Mar 24, 2023

michaelfig commented Mar 26, 2023

mhofman commented Mar 26, 2023

michaelfig left a comment

michaelfig Mar 26, 2023

mhofman Mar 27, 2023

michaelfig Mar 28, 2023

dckc Apr 13, 2023 •

edited

Loading

erights Apr 18, 2023

michaelfig commented Mar 27, 2023

mhofman commented Mar 27, 2023

mhofman commented Mar 28, 2023

michaelfig Mar 28, 2023

michaelfig commented Mar 28, 2023

mhofman commented Mar 28, 2023

mhofman commented Mar 31, 2023

mhofman commented Mar 31, 2023

mhofman commented Apr 19, 2023

mergify bot commented Apr 19, 2023

mhofman commented Apr 19, 2023

mergify bot commented Apr 19, 2023

	import { Fail } from '@agoric/assert';
	const { Fail } = assert;

feat: Support state-sync #7225

feat: Support state-sync #7225

Conversation

mhofman commented Mar 23, 2023 • edited Loading

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

mhofman commented Mar 23, 2023

datadog-full-agoric bot commented Mar 23, 2023 • edited Loading

Datadog Report

❌ Failed Tests (2)

codecov bot commented Mar 23, 2023 • edited Loading

Codecov Report

mhofman commented Mar 24, 2023

michaelfig commented Mar 26, 2023

mhofman commented Mar 26, 2023

michaelfig left a comment

Choose a reason for hiding this comment

michaelfig Mar 26, 2023

Choose a reason for hiding this comment

mhofman Mar 27, 2023

Choose a reason for hiding this comment

michaelfig Mar 28, 2023

Choose a reason for hiding this comment

dckc Apr 13, 2023 • edited Loading

Choose a reason for hiding this comment

erights Apr 18, 2023

Choose a reason for hiding this comment

michaelfig commented Mar 27, 2023

mhofman commented Mar 27, 2023

mhofman commented Mar 28, 2023

michaelfig Mar 28, 2023

Choose a reason for hiding this comment

michaelfig commented Mar 28, 2023

mhofman commented Mar 28, 2023

mhofman commented Mar 31, 2023

mhofman commented Mar 31, 2023

mhofman commented Apr 19, 2023

mergify bot commented Apr 19, 2023

☑️ This pull request is already queued

mhofman commented Apr 19, 2023

mergify bot commented Apr 19, 2023

✅ Pull request refreshed

mhofman commented Mar 23, 2023 •

edited

Loading

datadog-full-agoric bot commented Mar 23, 2023 •

edited

Loading

codecov bot commented Mar 23, 2023 •

edited

Loading

dckc Apr 13, 2023 •

edited

Loading