core, trie: new trie #1405

fjl · 2015-07-05T23:28:08Z

This PR contains a rewrite of package trie with the following goals:

reduced API surface: unexported node types, string funcs, other internals
documented core internals
documented API
improved performance
more memory reuse, especially when hashing the trie
hashing is independent of the database

Gustav-Simonsson · 2015-07-06T16:19:14Z

trie/trie.go

 )

-func ParanoiaCheck(t1 *Trie, backend Backend) (bool, *Trie) {
-	t2 := New(nil, backend)
+const lruSize = 200


What unit is this, number of nodes, leaf values?
A comment above with the unit and motivation behind the chosen size would be useful.

Number of DB entries.

fjl · 2015-08-18T23:53:59Z

Preliminary performance report:

benchmark                                       old ns/op     new ns/op     delta
BenchmarkInsertChain_empty_memdb-8              109218        64479         -40.96%
BenchmarkInsertChain_empty_diskdb-8             194874        136614        -29.90%
BenchmarkInsertChain_valueTx_memdb-8            240241        156505        -34.85%
BenchmarkInsertChain_valueTx_diskdb-8           412044        322234        -21.80%
BenchmarkInsertChain_valueTx_100kB_memdb-8      1203981       1010477       -16.07%
BenchmarkInsertChain_valueTx_100kB_diskdb-8     3067184       2969643       -3.18%
BenchmarkInsertChain_uncles_memdb-8             213715        170358        -20.29%
BenchmarkInsertChain_uncles_diskdb-8            301778        251242        -16.75%
BenchmarkInsertChain_ring200_memdb-8            144237467     81842282      -43.26%
BenchmarkInsertChain_ring200_diskdb-8           170420433     93856663      -44.93%
BenchmarkInsertChain_ring1000_memdb-8           194525358     116581937     -40.07%
BenchmarkInsertChain_ring1000_diskdb-8          219006749     127585191     -41.74%
BenchmarkCompactEncode                          135           61.4          -54.52%
BenchmarkGet                                    628           289           -53.98%
BenchmarkGetDB                                  518           1471          +183.98%
BenchmarkUpdateBE                               2935          2002          -31.79%
BenchmarkUpdateLE                               3289          2409          -26.76%
BenchmarkHashBE                                 100965203     40623083      -59.77%
BenchmarkHashLE                                 117250940     43654768      -62.77%

benchmark                                       old allocs     new allocs     delta
BenchmarkInsertChain_empty_memdb-8              743            330            -55.59%
BenchmarkInsertChain_empty_diskdb-8             810            388            -52.10%
BenchmarkInsertChain_valueTx_memdb-8            1500           790            -47.33%
BenchmarkInsertChain_valueTx_diskdb-8           1616           890            -44.93%
BenchmarkInsertChain_valueTx_100kB_memdb-8      1504           794            -47.21%
BenchmarkInsertChain_valueTx_100kB_diskdb-8     1847           1029           -44.29%
BenchmarkInsertChain_uncles_memdb-8             1195           782            -34.56%
BenchmarkInsertChain_uncles_diskdb-8            1260           846            -32.86%
BenchmarkInsertChain_ring200_memdb-8            1182124        339278         -71.30%
BenchmarkInsertChain_ring200_diskdb-8           1220860        345026         -71.74%
BenchmarkInsertChain_ring1000_memdb-8           1600303        429111         -73.19%
BenchmarkInsertChain_ring1000_diskdb-8          1638553        435461         -73.42%
BenchmarkCompactEncode                          2              1              -50.00%
BenchmarkGet                                    2              1              -50.00%
BenchmarkGetDB                                  2              6              +200.00%
BenchmarkUpdateBE                               14             8              -42.86%
BenchmarkUpdateLE                               13             8              -38.46%
BenchmarkHashBE                                 392188         106708         -72.79%
BenchmarkHashLE                                 473505         111063         -76.54%

benchmark                                       old bytes     new bytes     delta
BenchmarkInsertChain_empty_memdb-8              46288         19827         -57.17%
BenchmarkInsertChain_empty_diskdb-8             54900         28002         -48.99%
BenchmarkInsertChain_valueTx_memdb-8            100512        48822         -51.43%
BenchmarkInsertChain_valueTx_diskdb-8           115107        62805         -45.44%
BenchmarkInsertChain_valueTx_100kB_memdb-8      1376513       899592        -34.65%
BenchmarkInsertChain_valueTx_100kB_diskdb-8     3008558       2297909       -23.62%
BenchmarkInsertChain_uncles_memdb-8             68362         41908         -38.70%
BenchmarkInsertChain_uncles_diskdb-8            79106         55894         -29.34%
BenchmarkInsertChain_ring200_memdb-8            94582187      18819301      -80.10%
BenchmarkInsertChain_ring200_diskdb-8           96704176      20032446      -79.28%
BenchmarkInsertChain_ring1000_memdb-8           137721565     23260916      -83.11%
BenchmarkInsertChain_ring1000_diskdb-8          139818770     24621324      -82.39%
BenchmarkCompactEncode                          16            4             -75.00%
BenchmarkGet                                    144           80            -44.44%
BenchmarkGetDB                                  144           320           +122.22%
BenchmarkUpdateBE                               1766          1567          -11.27%
BenchmarkUpdateLE                               1797          1627          -9.46%
BenchmarkHashBE                                 223376483     3735414       -98.33%
BenchmarkHashLE                                 244950514     3927063       -98.40%

zelig · 2015-08-20T01:00:40Z

trie/secure_trie.go

+//
+// If root is the zero hash or the sha3 hash of an empty string, the
+// trie is initially empty. Otherwise, New will panics if db is nil or
+// root does not exist in the database. Accessing the trie loads nodes


will panic [typo]

It will return an error in the final version of this PR.

zelig · 2015-08-20T01:29:01Z

is ethdb.Database put/get meant to be concurrently used? cos AFAIU LDBDatabase implementation is threadsafe due to leveldb and batch calls being safe but MemDatabase is not, nor is its batch.

fjl · 2015-08-20T05:10:35Z

                                                                                  Trie is not thread safe either.

zelig · 2015-08-20T11:55:40Z

Yes the trie is clearly documented not to be. But ethdb.Database is independent and ldbd is actually perfectly functional concurrently right? Either way it's worth a comment

fjl · 2015-09-09T01:53:03Z

I've rebased this and added my initial stab at the merkle proof functions. Some comments on those below. My plan is to move parts of this (rlp changes in #1778, ethdb changes after #1779) into develop and wait for the other pending core changes to be merged before bringing in the trie. This also leaves more time for reviewing.

About the merkle proof code: @ebuchman did an implementation for the old trie in #1624, but I couldn't port these directly because the API wasn't quite right. For the light client, the initial use of these proofs is that we have the state root hash and wish to retrieve the proof and value for a key in one request. The API in #1624 expected the value to be passed as an input to Verify. The PR also defined a custom encoding for proofs. AFAIK the plan is to represent proofs as a list of encoded nodes as defined in the YP.

@zsfelfoldi Please comment on the API. The two functions of interest are

func (*Trie) Prove(key []byte) []rlp.RawValue
func VerifyProof(root common.Hash, key []byte, proof []rlp.RawValue) (value []byte, err error)

We'll probably find some bugs in the proof stuff and Prove isn't exactly fast either. Always room for improvement ;).

zsfelfoldi · 2015-09-09T02:29:06Z

The prove/verify API is perfect for me, I'll probably add support for my skipLevels improvement (omitting the highest levels from the proof if we already have them) later if there's a consensus about adding it to the eth/64 protocol in the first place. I agree about the value being a result of verify, not an input parameter (although it can be read from the proof itself but it's illogical to pass as an input).

zsfelfoldi · 2015-09-09T02:38:41Z

You can also leave the debugging of proofs to me. Do you think there's a chance that the new trie can be merged into develop in the next few days? I need it but I'm already rebased on top of Peter's #1779 and I really wouldn't like to go any further away from develop :)

ebuchman · 2015-09-09T02:42:30Z

Nice work! I agree the Value doesn't belong in the API, I was using it for sanity checking while testing and forgot to remove (it only just checks that the value passed in and the value at the bottom of the proof are the same). Also wasn't sure about encoding so just tried something simple. But looks like you have a shorter implementation anyways so great! I'll take a look at some testing.

fjl · 2015-09-22T20:59:31Z

@zsfelfoldi I've rebased this on develop (including the ethdb move). If you rebase the light client work on it, your PR will no longer be cluttered with them.

core, trie: new trie

* Disable gas price minimum updates * Round robin transfers

fjl added in progress labels Jul 5, 2015

Gustav-Simonsson reviewed Jul 6, 2015
View reviewed changes

obscuren mentioned this pull request Aug 9, 2015

trie: merkle proofs #1624

Closed

fjl force-pushed the lean-trie branch from 5ca920c to 4a6094d Compare August 13, 2015 16:51

obscuren modified the milestone: 1.0.3 Aug 15, 2015

fjl force-pushed the lean-trie branch 11 times, most recently from e0ccd13 to 289130a Compare August 18, 2015 23:34

fjl force-pushed the lean-trie branch from 289130a to e029b90 Compare August 18, 2015 23:57

zelig reviewed Aug 20, 2015
View reviewed changes

obscuren modified the milestones: 1.2.0, Light client stage 1 Aug 21, 2015

obscuren modified the milestones: Light client stage 1, 1.2.0 Aug 21, 2015

fjl force-pushed the lean-trie branch 2 times, most recently from f23bda1 to 43cc17b Compare September 9, 2015 01:41

fjl force-pushed the lean-trie branch from 43cc17b to 439b420 Compare September 10, 2015 19:43

This was referenced Sep 14, 2015

all: move common.Database to ethdb and add NewBatch #1801

Merged

Remove common/rlp*.go, which is implemented in package rlp now. #1804

Closed

Implement patricia trie SPV proofs #307

Closed

fjl added 4 commits September 22, 2015 22:49

trie: improve benchmarks

6b91a4a

core, trie: new trie

565d9f2

core, core/state: batch-based state sync

a2d5a60

trie: add merkle proof functions

c1a352c

fjl force-pushed the lean-trie branch from 439b420 to c1a352c Compare September 22, 2015 20:59

obscuren added a commit that referenced this pull request Oct 1, 2015

Merge pull request #1405 from fjl/lean-trie

49ae538

core, trie: new trie

obscuren merged commit 49ae538 into ethereum:develop Oct 1, 2015

obscuren removed the in progress label Oct 1, 2015

tony-ricciardi pushed a commit to tony-ricciardi/go-ethereum that referenced this pull request Jan 20, 2022

Disable GPM updates during mycelo load tests (ethereum#1405)

d7d84e6

* Disable gas price minimum updates * Round robin transfers

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core, trie: new trie #1405

core, trie: new trie #1405

fjl commented Jul 5, 2015

Gustav-Simonsson Jul 6, 2015

fjl Jul 6, 2015

fjl commented Aug 18, 2015

zelig Aug 20, 2015

fjl Aug 20, 2015

zelig commented Aug 20, 2015

fjl commented Aug 20, 2015

zelig commented Aug 20, 2015

fjl commented Sep 9, 2015

zsfelfoldi commented Sep 9, 2015

zsfelfoldi commented Sep 9, 2015

ebuchman commented Sep 9, 2015

fjl commented Sep 22, 2015

core, trie: new trie #1405

core, trie: new trie #1405

Conversation

fjl commented Jul 5, 2015

Gustav-Simonsson Jul 6, 2015

Choose a reason for hiding this comment

fjl Jul 6, 2015

Choose a reason for hiding this comment

fjl commented Aug 18, 2015

zelig Aug 20, 2015

Choose a reason for hiding this comment

fjl Aug 20, 2015

Choose a reason for hiding this comment

zelig commented Aug 20, 2015

fjl commented Aug 20, 2015

zelig commented Aug 20, 2015

fjl commented Sep 9, 2015

zsfelfoldi commented Sep 9, 2015

zsfelfoldi commented Sep 9, 2015

ebuchman commented Sep 9, 2015

fjl commented Sep 22, 2015