feat!: Add hooks to allow app modules to add things to state-sync

[yihuang]

Description

Closes: #7340

• Support registering multiple snapshotters in snapshot manager.
• Append the extension snapshotters to existing snapshot stream.

TODO: testing.

• existing tests are fixed

---

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

• included the correct type prefix in the PR title
• added ! to the type prefix if API or client breaking change
• targeted the correct branch (see PR Targeting)
• provided a link to the relevant issue or specification
• followed the guidelines for building modules
• included the necessary unit and integration tests
• added a changelog entry to CHANGELOG.md
• included comments for documenting Go code
• updated the relevant documentation or specification
• reviewed "Files changed" and left comments if necessary
• confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

• confirmed the correct type prefix in the PR title
• confirmed ! in the type prefix if API or client breaking change
• confirmed all author checklist items have been addressed
• reviewed state machine logic
• reviewed API design and naming
• reviewed documentation is accurate
• reviewed tests and test coverage
• manually tested (if applicable)

[yihuang]

it's moved to snapshot package to avoid cyclic import.

[yihuang]

Added a field for extension snapshotters.
multistore snapshotter is a special one, it doesn't need a name, it is always placed at the beginning of the binary stream.

[yihuang]

should be called when setup app.

[yihuang]

• removal of format parameter: Snapshotter chooses a format autonomously, not pass in from the caller.

[yihuang]

Snapshotter announces the format it uses to snapshot with method SnapshotFormat().

[zmanian]

Really like what I am seeing here. Just starting tor review.

[codecov]

Codecov Report

Merging #10961 (c0bf26f) into master (56af6a7) will decrease coverage by 0.11%.
The diff coverage is 42.12%.

❗ Current head c0bf26f differs from pull request most recent head 81fde71. Consider uploading reports for the commit 81fde71 to get more accurate results

@@            Coverage Diff             @@
##           master   #10961      +/-   ##
==========================================
- Coverage   66.07%   65.95%   -0.12%     
==========================================
  Files         662      661       -1     
  Lines       68217    68327     +110     
==========================================
- Hits        45071    45066       -5     
- Misses      20527    20631     +104     
- Partials     2619     2630      +11     

Impacted Files	Coverage Δ
baseapp/baseapp.go	66.36% <0.00%> (-1.87%)	⬇️
server/mock/store.go	20.21% <0.00%> (ø)
snapshots/chunk.go	85.00% <ø> (ø)
store/v2/multi/store.go	76.23% <0.00%> (-0.09%)	⬇️
snapshots/manager.go	54.58% <30.18%> (-20.42%)	⬇️
snapshots/stream.go	58.82% <58.82%> (ø)
store/rootmulti/store.go	75.34% <59.57%> (+1.42%)	⬆️
baseapp/options.go	62.03% <100.00%> (ø)
x/slashing/types/codec.go	40.00% <0.00%> (-22.50%)	⬇️
x/evidence/types/codec.go	40.00% <0.00%> (-17.15%)	⬇️
... and 42 more

[giansalex]

any example of how it will be configured in app.go?
In a previous PoC for cosmwasm state-sync, i needed a context to access the KVStore like

[yihuang]

any example of how it will be configured in app.go?

It should be something like this:

type WasmSnapshotter struct {
  cms *rootmulti.Store
  storeKey storetypes.StoreKey
}

func NewWasmSnapshotter(cms *rootmulti.Store, storeKey storetypes.StoreKey) *WasmSnapshotter {
  return &{
    cms: cms,
    storeKey: storeKey,
  }
}

func (ws *WasmSnapshotter) Snapshot(height uint64, protoWriter protoio.Writer) error {
  cacheMS, err := app.cms.CacheMultiStoreWithVersion(height)
  if err != nil {
    return err
  }
  store := cacheMS.GetKVStore(cms.storeKey)
  ...
  // write the items
  for {
    snapshottypes.WriteExtensionItem(protoWriter, []byte{...})
  }
}

// app.go
func NewApp() *App {
  ...
  app.snapshotManager.RegisterExtension(
    NewWasmSnapshotter(app.cms, wasmStoreKey),
  )
}

[giansalex]

But cms, snapshotManager are private in BaseApp.

this was an option:

	cms := store.NewCommitMultiStore(db)
	cmsOption := func(b *baseapp.BaseApp) {
		b.SetCMS(cms)
	}
	baseAppOptions = append(baseAppOptions, cmsOption)
	bApp := baseapp.NewBaseApp(Name, logger, db, encodingConfig.TxConfig.TxDecoder(), baseAppOptions...)

for extensions, maybe it can be added to BaseApp/options like the other snapshot methods

baseapp.SetSnapshotStore()
baseapp.SetSnapshotInterval()
baseapp.SetSnapshotKeepRecent()

[yihuang]

But cms, snapshotManager are private in BaseApp.

this was an option:

	cms := store.NewCommitMultiStore(db)

	cmsOption := func(b *baseapp.BaseApp) {

		b.SetCMS(cms)

	}

	baseAppOptions = append(baseAppOptions, cmsOption)

	bApp := baseapp.NewBaseApp(Name, logger, db, encodingConfig.TxConfig.TxDecoder(), baseAppOptions...)

for extensions, maybe it can be added to BaseApp/options like the other snapshot methods

baseapp.SetSnapshotStore()

baseapp.SetSnapshotInterval()

baseapp.SetSnapshotKeepRecent()

Maybe we can simplify add the public methods: "MultiStore" and "SnapshotManager" for this?

[giansalex]

Maybe we can simplify add the public methods: "MultiStore" and "SnapshotManager" for this?

Yes, i was looking for ways without modifying the sdk

[ethanfrey]

What about passing in cms *rootmulti.Store to the Snapshot call rather than in NewWasmSnapshotter.
Not sure if the rootmulti.Store is more accessible elsewhere (when state sync is called)?

[tac0turtle]

@yihuang can you update this pr and we will start reviewing it

[tac0turtle]

@yihuang there seems to be a failing test, can you handle the fix then the bot will merge it

[yihuang]

@yihuang there seems to be a failing test, can you handle the fix then the bot will merge it

that failing test seems to be an accidental one, I clicked rerun and it passed.

[assafmo]

Hi guys, I've used the new snapshot extension feature and it works great for us to snapshot the CosmWasm wasm directory!

For reference here's the code: https://github.com/scrtlabs/SecretNetwork/blob/v1.3.0-beta.10/x/compute/wasm_snapshotter.go
Init code: https://github.com/scrtlabs/SecretNetwork/blob/v1.3.0-beta.10/app/app.go#L626-L628