Problem: memiavl operations on multiple stores are not atomic

[yihuang]

When integrated with sdk, we need to support multiple stores in an atomic way, two potential solutions:

Treat Multiple Stores as First Class Citizen

• The WAL should be handled in a unified way with multiple stores, so we don't end up with different latest versions with different stores
• The snapshot rewrite should be unified with multistore as well, so the snapshot versions are consistent.

Following directory structure should make more sense, treat multistore as first class citizen in memiavl implementation:

memiavl.db
  current -> snapshot-N
  snapshot-N
    bank
      kvs
      nodes
      metadata
    acc
    ... other stores
  wal

A wal entry should include the change sets for all stores, and store upgrades (additions/renaming/deletions)

Advantages

• easier to manage store upgrades.
• single WAL file to manage.

Disadvantages

• harder to integrate with the current store APIs

Store CommitInfo

Similar to current rootmulti store design in sdk, store a common commit info file which records the latest committed information for all stores, the stores must has persisted (fsync) the committed version, on crash, some stores may saved one extra version, on startup the extra version should be ignored and truncated.

Advantages

• more natural to integrate with the current store APIs

Disadvantages

• a dozen WAL files to manage, and have to fsync each of them before writing out the commit info.
• tricky to manage store upgrades

[yihuang]

closed by #959