Add SMT store type #8507

tzdybal · 2021-02-03T23:37:00Z

This PR introduces SMT store. It doesn't support Queryable interface yet, and it's not used anywhere.

This work is related to #8430 and #8297.

SMT (Sparse Merkle Tree) is intended to replace IAVL. New type implements same interfaces as iavl.Store.

Sparse Merkle Tree does not support iteration over keys in order. To provide drop-in replacement for IAVL, Iterator and ReverseIterator has to be implemented. SMT Store implementation use the underlying KV store to: - maintain a list of keys (under a prefix) - iterate over a keys Values are stored only in SMT.

tac0turtle · 2021-02-26T10:58:06Z

@robert-zaremba is it okay to merge this as is for now?

robert-zaremba · 2021-02-26T20:51:41Z

@robert-zaremba is it okay to merge this as is for now?

Yes, all changes must go to ll-smt branch. Let's review the changeset.

robert-zaremba

left few comments.

robert-zaremba · 2021-02-26T22:58:05Z

store/smt/iterator.go

+
+var (
+	indexPrefix = []byte("smt-ordering-idx-")
+	afterIndex  = []byte("smt-ordering-idx.") // '.' is next after '-' in ASCII


would it be more beneficial to have the prefix keys shorter?

I expect this to be compressed away by DB.
Single-byte, numerical value would be much cleaner.

yes, let's use single byte values.

robert-zaremba · 2021-02-26T22:59:13Z

store/smt/iterator.go

+}
+
+func plainKey(key []byte) []byte {
+	return key[len(indexPrefix):]


let's store cache len(indexPrefix) in a module scope.

If prefix is single-byte, we can even hardcode this ;)

robert-zaremba · 2021-02-26T23:03:16Z

store/smt/iterator_test.go

+		assert.Equal(t, p.key, iter.Key())
+		assert.Equal(t, p.val, iter.Value())
+		iter.Next()
+	}


I thought that LL SMT is hashing the keys before inserting, so how possible it preserves a pre-image order?

Here, we're testing cosmos-sdk SMT store, not LL SMT data structure. I'm using ordering from key->value mapping (https://github.com/cosmos/cosmos-sdk/pull/8507/files/6fd860d514daa4c9caf55df383d81b1a1e671ede#diff-0d018f820578ed2dd08d3bd1cb6f854aead4e2b32333b15105743eecb21bba77R41).

robert-zaremba · 2021-02-26T23:05:07Z

store/smt/iterator_test.go

+	for _, p := range pairs {
+		require.True(t, iter.Valid())
+		assert.Equal(t, p.key, iter.Key())
+		assert.Equal(t, p.val, iter.Value())


we should use require here. It will make lot of error messages if something is wrong and it will be messy. Assert doesn't stop the exectuion.
Tip: I'm usually declaring at the top of the function:

require := require.New(t) // works with assert as well

Then in the function I don't need to pass t any more: require.Equal(a, b).

robert-zaremba · 2021-02-26T23:15:32Z

store/smt/store.go

+func (s *Store) Get(key []byte) []byte {
+	defer telemetry.MeasureSince(time.Now(), "store", "smt", "get")
+	s.mtx.RLock()
+	defer s.mtx.RUnlock()


why do we need to lock? Isn't the underlying DB solving this?

robert-zaremba · 2021-02-26T23:19:02Z

store/smt/store.go

+	defer telemetry.MeasureSince(time.Now(), "store", "smt", "get")
+	s.mtx.RLock()
+	defer s.mtx.RUnlock()
+	val, err := s.tree.Get(key)


why do we query data through a tree, rather than directly from the DB?

This is probably leftover from version with only SMT tree (no plain-kv).

DB is used only for iteration in this version. It was implemented before finalization of ADR-040 and deciding that values goes to plain-KV.

robert-zaremba · 2021-02-26T23:20:48Z

store/smt/store.go

+	if err != nil {
+		panic(err.Error())
+	}
+	s.mtx.Unlock()


we should use defer s.mtx.Unlock() right after the Lock.
Unfortunately SDK is using a bad pattern to panic to quickly wind up the stack, rather than passing errors. So, we don't want to leave the store locked in case of panic.

robert-zaremba · 2021-03-23T00:46:12Z

shall we update the PR and merge into the branch?

tac0turtle · 2021-04-09T08:50:34Z

shall we update the PR and merge into the branch?

lets merge and follow up

aaronc · 2021-04-09T16:43:41Z

Not sure we want to add more new features into this release.

tac0turtle · 2021-04-09T16:45:39Z

Not sure we want to add more new features into this release.

This would be merged into a feature branch store/ll-smt. It will not be merged into master until at a later date.

robert-zaremba · 2021-04-12T22:24:10Z

@tzdybal , there are few small changes to do in this PR. Will you like to finish it and merge?

tzdybal · 2021-04-13T05:55:53Z

Yes, I'll fix this today.

robert-zaremba · 2021-04-23T14:55:50Z

Hi @tzdybal - if you need a help let me know - I can clean up this PR for you and merge it.

tac0turtle · 2021-04-23T19:55:29Z

Hi @tzdybal - if you need a help let me know - I can clean up this PR for you and merge it.

can you take over it please. The LL team is busy at the moment, dont want to add more to their plate.

tzdybal · 2021-04-23T20:56:09Z

Sorry for late reply and fix. I addressed all review comments.

SMT stores: * key -> hash(key, value) KV store stores: * key->value in "bucket 1", * hash(key, value) -> key in "bucket 2".

tzdybal · 2021-04-23T21:26:07Z

OK, so I changed the implementation to use ADR-040 data layout for KV and SMT.

Please re-review @robert-zaremba, @marbar3778.

robert-zaremba · 2021-05-14T17:36:33Z

@tzdybal - thanks for handling this, let me merge it so we can move forward.

* Initial SMT store type SMT (Sparse Merkle Tree) is intended to replace IAVL. New type implements same interfaces as iavl.Store. * Add iteration support to SMT Sparse Merkle Tree does not support iteration over keys in order. To provide drop-in replacement for IAVL, Iterator and ReverseIterator has to be implemented. SMT Store implementation use the underlying KV store to: - maintain a list of keys (under a prefix) - iterate over a keys Values are stored only in SMT. * Migrate to smt v0.1.1 * Extra test for SMT iterator * CommitStore implementation for SMT store * Use interface instead of concrete type * Add telemetry to SMT store * SMT: version->root mapping, cleanup * SMT proofs - initial code * Tests for SMT store ProofOp implementation * Fix linter errors * Use simple 1 byte KV-store prefixes * Improve assertions in tests * Use mutex properly * Store data in ADR-040-compatible way SMT stores: * key -> hash(key, value) KV store stores: * key->value in "bucket 1", * hash(key, value) -> key in "bucket 2".

tzdybal added 11 commits February 4, 2021 00:20

Initial SMT store type

6e4934c

SMT (Sparse Merkle Tree) is intended to replace IAVL. New type implements same interfaces as iavl.Store.

Migrate to smt v0.1.1

e208a1e

Extra test for SMT iterator

cd06819

CommitStore implementation for SMT store

65a40a4

Use interface instead of concrete type

d22f575

Add telemetry to SMT store

95b1073

SMT: version->root mapping, cleanup

b2086e7

SMT proofs - initial code

04bb530

Tests for SMT store ProofOp implementation

c1e928b

Fix linter errors

6fd860d

This was referenced Feb 3, 2021

Initial SMT store type celestiaorg/cosmos-sdk#12

Closed

ADR-040: Storage and SMT State Commitments #8430

Merged

tzdybal marked this pull request as ready for review February 4, 2021 09:40

tzdybal requested review from aaronc, alessio and alexanderbez as code owners February 4, 2021 09:40

robert-zaremba self-assigned this Feb 5, 2021

robert-zaremba reviewed Mar 1, 2021

View reviewed changes

Use simple 1 byte KV-store prefixes

c192976

tzdybal added 2 commits April 23, 2021 22:52

Improve assertions in tests

6a43af8

Use mutex properly

549c414

Store data in ADR-040-compatible way

e965573

SMT stores: * key -> hash(key, value) KV store stores: * key->value in "bucket 1", * hash(key, value) -> key in "bucket 2".

roysc mentioned this pull request May 12, 2021

Implement decoupled state commitments and storage vulcanize/cosmos-sdk#11

Merged

9 tasks

robert-zaremba approved these changes May 14, 2021

View reviewed changes

robert-zaremba merged commit 5ef2364 into cosmos:store/ll-smt May 14, 2021

UnitylChaos mentioned this pull request Jun 9, 2021

Proposal: Storage Overhaul f-o-a-m/kepler#262

Open

This was referenced Jun 10, 2021

Epic 1: Hook state store into SDK vulcanize/cosmos-sdk#1

Closed

Epic 3: Replace IAVL with SMT state commitment vulcanize/cosmos-sdk#3

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SMT store type #8507

Add SMT store type #8507

tzdybal commented Feb 3, 2021 •

edited

Loading

tac0turtle commented Feb 26, 2021

robert-zaremba commented Feb 26, 2021

robert-zaremba left a comment

robert-zaremba Feb 26, 2021

tzdybal Mar 1, 2021 •

edited

Loading

robert-zaremba Apr 12, 2021

robert-zaremba Feb 26, 2021

tzdybal Mar 1, 2021

robert-zaremba Feb 26, 2021

tzdybal Mar 1, 2021

robert-zaremba Feb 26, 2021

robert-zaremba Feb 26, 2021

robert-zaremba Feb 26, 2021

tzdybal Mar 1, 2021

tzdybal Apr 23, 2021

robert-zaremba Feb 26, 2021

robert-zaremba commented Mar 23, 2021

tac0turtle commented Apr 9, 2021

aaronc commented Apr 9, 2021

tac0turtle commented Apr 9, 2021

robert-zaremba commented Apr 12, 2021

tzdybal commented Apr 13, 2021

robert-zaremba commented Apr 23, 2021

tac0turtle commented Apr 23, 2021

tzdybal commented Apr 23, 2021

tzdybal commented Apr 23, 2021 •

edited

Loading

robert-zaremba commented May 14, 2021

Add SMT store type #8507

Add SMT store type #8507

Conversation

tzdybal commented Feb 3, 2021 • edited Loading

tac0turtle commented Feb 26, 2021

robert-zaremba commented Feb 26, 2021

robert-zaremba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzdybal Mar 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robert-zaremba commented Mar 23, 2021

tac0turtle commented Apr 9, 2021

aaronc commented Apr 9, 2021

tac0turtle commented Apr 9, 2021

robert-zaremba commented Apr 12, 2021

tzdybal commented Apr 13, 2021

robert-zaremba commented Apr 23, 2021

tac0turtle commented Apr 23, 2021

tzdybal commented Apr 23, 2021

tzdybal commented Apr 23, 2021 • edited Loading

robert-zaremba commented May 14, 2021

tzdybal commented Feb 3, 2021 •

edited

Loading

tzdybal Mar 1, 2021 •

edited

Loading

tzdybal commented Apr 23, 2021 •

edited

Loading