Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge v1.13.6 from geth #1205

Merged
merged 85 commits into from
Apr 9, 2024
Merged
Changes from 1 commit
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
fa8d398
cmd, core, trie: verkle-capable `geth init` (#28270)
gballet Nov 14, 2023
e803ef0
eth/tracers/js: fix isPush for push0 (#28520)
s1na Nov 14, 2023
24d4622
trie: spelling - fix comments in hasher (#28507)
sysvm Nov 14, 2023
2391fbc
tests/fuzzers: move fuzzers into native packages (#28467)
holiman Nov 14, 2023
c5b7cfa
tests: skip tests on windows 32bit CI (#28521)
holiman Nov 14, 2023
984f826
cmd/geth: more special cases logging tests (#28527)
jwasinger Nov 15, 2023
2814ee0
accounts,cmd,console,les,metrics: refactor some errors checked by (S…
BabyHalimao Nov 15, 2023
b9504e4
miner: run tests in parallel (#28506)
estensen Nov 15, 2023
fcc7ae1
internal/jsre/deps: fix typo in jsdoc (#28511)
aliening Nov 15, 2023
db7895d
accounts/abi: improve readability of method-to-string conversion (#2…
Halimao Nov 15, 2023
a75a2d6
all: replace some cases of strings.SplitN with strings.Cut (#28446)
estensen Nov 15, 2023
5bf8769
ethdb/memorydb, trie: reduced allocations (#28473)
holiman Nov 15, 2023
8b78d6a
Dockerfile: update Go to 1.21 (#28538)
estensen Nov 17, 2023
c8a2202
cmd/evm: validate blockchain tests poststate account storage (#28443)
danceratopz Nov 17, 2023
77cb21d
signer: run tests in parallel (#28536)
estensen Nov 20, 2023
a737482
accounts, cmd: fix typos (#28526)
levisyin Nov 20, 2023
14a1e96
core/txpool/legacypool: respect nolocals-setting (#28435)
jp-imx Nov 20, 2023
460cc16
cmd: run tests in parallel (#28546)
estensen Nov 20, 2023
661bd45
core/state/snapshot: print correct error from trie iterator (#28560)
rjl493456442 Nov 21, 2023
ad16f11
cmd/evm: capitalize evm commands (#28569)
tmelhao Nov 21, 2023
525db7b
accounts/abi: context info on unpack-errors (#28529)
levisyin Nov 21, 2023
146e8d9
core, trie, rpc: speed up tests (#28461)
MariusVanDerWijden Nov 21, 2023
6489a0d
ethdb/pebble: don't double-close iterator inside pebbleIterator (#28566)
magicxyyz Nov 21, 2023
e9f59b5
eth/filters: reuse error msg for invalid block range (#28479)
ucwong Nov 21, 2023
347fecd
core/types: make 'v' optional for DynamicFeeTx and BlobTx (#28564)
marioevz Nov 22, 2023
d6cea48
rpc: improve performance of subscription notification encoding (#28328)
jsvisa Nov 22, 2023
104dbf7
cmd/utils: validate pre-existing genesis in --dev mode (#28468)
jwasinger Nov 22, 2023
3cfcd25
cmd/geth: add support for --dev flag in dumpgenesis (#28463)
jwasinger Nov 22, 2023
5ff929c
les/vflux: run tests in parallel (#28524)
estensen Nov 22, 2023
d468c33
cmd/{geth,utils}: add cmd to export preimages in snap enumeration ord…
gballet Nov 22, 2023
63127f5
cmd/geth: fix build error (#28585)
fjl Nov 22, 2023
eec37e3
cmd/devp2p/internal/ethtest: undo debug-hack (#28588)
holiman Nov 23, 2023
d76efbb
params: update discV5 bootnodes (#28562)
cortze Nov 23, 2023
bdf5e38
cmd, les, tests: remove light client code (#28586)
MariusVanDerWijden Nov 23, 2023
2a20130
eth, internal/ethapi: drop some weird indirection (#28597)
karalabe Nov 24, 2023
333dd95
trie: fix random test generator early terminate (#28590)
weiihann Nov 25, 2023
1e28e0b
eth/gasestimator, internal/ethapi: move gas estimator out of rpc (#28…
karalabe Nov 27, 2023
5b57727
go.mod: update uint256 to v1.2.4 (#28612)
holiman Nov 27, 2023
71817f3
eth/catalyst, eth/downloader: expose more sync information (#28584)
rjl493456442 Nov 28, 2023
58297e3
light: remove package light(#28614)
lightclient Nov 28, 2023
63979bc
cmd/evm, core/state: fix post-exec dump of state (statetests, blockch…
holiman Nov 28, 2023
248dc50
ethereum: remove TODO comment about subscription (#28609)
sbs2001 Nov 28, 2023
bbc5db8
eth/tracers/js: fix type inconsistencies (#28488)
s1na Nov 28, 2023
34dcd74
crypto/secp256k1: fix 32-bit tests when CGO_ENABLED=0 (#28602)
gballet Nov 28, 2023
e0c7ad0
consensus: verify the nonexistence of shanghai- and cancun-specific h…
jakub-freebit Nov 28, 2023
61b844f
eth/gasestimator: allow slight estimation error in favor of less iter…
karalabe Nov 28, 2023
28e7371
all: replace log15 with slog (#28187)
jwasinger Nov 29, 2023
ab0eb46
core/state: make stateobject.create selfcontain (#28459)
rjl493456442 Nov 29, 2023
fa0df76
trie/triedb/hashdb: take lock around access to dirties cache (#28542)
magicxyyz Nov 30, 2023
f2b509d
accounts/abi/bind: fix typo (#28630)
BorkBorked Dec 1, 2023
dd0d0a2
slog: faster and less memory-consumption (#28621)
holiman Dec 1, 2023
5fb8ebc
eth/tracers: tx-level state in debug_traceCall (#28460)
jsvisa Dec 1, 2023
6e488c2
cmd/evm: fix Env struct json tag (#28635)
ddl-hust Dec 4, 2023
f04e5bd
accounts/abi/bind: fixed typos (#28634)
BorkBorked Dec 4, 2023
70fd0b6
eth/fetcher: fix invalid tracking of received at time for block (#28637)
manav2401 Dec 4, 2023
2e13b01
accounts: run tests in parallel (#28544)
estensen Dec 4, 2023
3dc071e
eth/tracers/logger: make structlog/json-log stack hex again (#28628)
holiman Dec 5, 2023
b8d44ed
log: remove lazy, remove unused interfaces, unexport methods (#28622)
holiman Dec 5, 2023
69576df
.github: use github actions to run 32-bit linux tests (#28549)
gballet Dec 5, 2023
55b483d
ethdb/pebble: remove a dependency (#28627)
MariusVanDerWijden Dec 6, 2023
a113497
tests/fuzzers/bls12381: deactivate BLS fuzzer when CGO_ENABLED=0 (#28…
gballet Dec 7, 2023
77c4bbc
build: upgrade -dlgo version to Go 1.21.5 (#28648)
sandakersmann Dec 7, 2023
5a45e7a
rpc: fix ns/µs mismatch in metrics (#28649)
holiman Dec 8, 2023
1048e2d
cmd/evm: fix dump after state-test exec (#28650)
holiman Dec 8, 2023
fff843c
beacon/light: add CommitteeChain (#27766)
zsfelfoldi Dec 8, 2023
d98d70f
cmd/utils, eth: disallow invalid snap sync / snapshot flag combos (#2…
karalabe Dec 8, 2023
e206d3f
trie: remove inconsistent trie nodes during sync in path mode (#28595)
rjl493456442 Dec 8, 2023
a3ca1b2
cmd/utils: fix HTTPHost, WSHost flag priority (#28669)
weiihann Dec 12, 2023
17c2b3c
eth/protocols/eth: fix typos in comments (#28652)
guerrierindien Dec 12, 2023
81fd1b3
core/txpool : small cleanup refactors (#28654)
ucwong Dec 12, 2023
b2ced97
eth/fetcher, eth/gasestimator: fix typos in comments (#28675)
Ursulafe Dec 13, 2023
0f74aad
all: fix typos in comments (#28662)
nnsW3 Dec 13, 2023
f1794ba
miner: eliminate the dead loop possibility for `newWorkLoop` and `mai…
FletcherMan Dec 15, 2023
edc864f
all: fix typos in comments (#28682)
bodhi-crypo Dec 18, 2023
5b22a47
p2p/discover: add liveness check in collectTableNodes (#28686)
fjl Dec 18, 2023
02766d3
internal/flags: add missing flag types for auto-env-var generation (#…
sebastianst Dec 18, 2023
05bbc56
cmd/evm: default to mirror mainnet forks enabled (#28691)
jwasinger Dec 18, 2023
553bafc
cmd/evm, cmd/clef, cmd/bootnode: fix / unify logging (#28696)
holiman Dec 18, 2023
c18c5c3
cmd/evm: t8n support custom tracers (#28557)
jsvisa Dec 18, 2023
a18b845
params: release go-ethereum v1.13.6 stable
fjl Dec 18, 2023
5ba3d57
Merge branch 'release/1.13' into release-1.13.6
fjl Dec 18, 2023
da6cdaf
all: release go-ethereum v1.13.6
fjl Dec 18, 2023
5d77c38
Merge tag 'v1.13.6' into upstream_merge
cffls Mar 28, 2024
3fc3de0
Fix build errors
cffls Mar 29, 2024
98a2f47
Fix test-integration
cffls Mar 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
trie/triedb/hashdb: take lock around access to dirties cache (#28542)
Add read locking of db lock around access to dirties cache in hashdb.Database to prevent
data race versus hashdb.Database.dereference which can modify the dirities map by deleting
an item.

Fixes #28541

---------

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
magicxyyz and rjl493456442 authored Nov 30, 2023
commit fa0df76f3cfd186a1f06f2b80aa5dbb89555b009
11 changes: 0 additions & 11 deletions trie/database.go
Original file line number Diff line number Diff line change
@@ -240,17 +240,6 @@ func (db *Database) Dereference(root common.Hash) error {
return nil
}

// Node retrieves the rlp-encoded node blob with provided node hash. It's
// only supported by hash-based database and will return an error for others.
// Note, this function should be deprecated once ETH66 is deprecated.
func (db *Database) Node(hash common.Hash) ([]byte, error) {
hdb, ok := db.backend.(*hashdb.Database)
if !ok {
return nil, errors.New("not supported")
}
return hdb.Node(hash)
}

// Recover rollbacks the database to a specified historical point. The state is
// supported as the rollback destination only if it's canonical state and the
// corresponding trie histories are existent. It's only supported by path-based
69 changes: 23 additions & 46 deletions trie/triedb/hashdb/database.go
Original file line number Diff line number Diff line change
@@ -82,11 +82,6 @@ var Defaults = &Config{
// Database is an intermediate write layer between the trie data structures and
// the disk database. The aim is to accumulate trie writes in-memory and only
// periodically flush a couple tries to disk, garbage collecting the remainder.
//
// Note, the trie Database is **not** thread safe in its mutations, but it **is**
// thread safe in providing individual, independent node access. The rationale
// behind this split design is to provide read access to RPC handlers and sync
// servers even while the trie is executing expensive garbage collection.
type Database struct {
diskdb ethdb.Database // Persistent storage for matured trie nodes
resolver ChildResolver // The handler to resolve children of nodes
@@ -113,7 +108,7 @@ type Database struct {
// cachedNode is all the information we know about a single cached trie node
// in the memory database write layer.
type cachedNode struct {
node []byte // Encoded node blob
node []byte // Encoded node blob, immutable
parents uint32 // Number of live nodes referencing this one
external map[common.Hash]struct{} // The set of external children
flushPrev common.Hash // Previous node in the flush-list
@@ -152,9 +147,9 @@ func New(diskdb ethdb.Database, config *Config, resolver ChildResolver) *Databas
}
}

// insert inserts a simplified trie node into the memory database.
// All nodes inserted by this function will be reference tracked
// and in theory should only used for **trie nodes** insertion.
// insert inserts a trie node into the memory database. All nodes inserted by
// this function will be reference tracked. This function assumes the lock is
// already held.
func (db *Database) insert(hash common.Hash, node []byte) {
// If the node's already cached, skip
if _, ok := db.dirties[hash]; ok {
@@ -183,9 +178,9 @@ func (db *Database) insert(hash common.Hash, node []byte) {
db.dirtiesSize += common.StorageSize(common.HashLength + len(node))
}

// Node retrieves an encoded cached trie node from memory. If it cannot be found
// node retrieves an encoded cached trie node from memory. If it cannot be found
// cached, the method queries the persistent database for the content.
func (db *Database) Node(hash common.Hash) ([]byte, error) {
func (db *Database) node(hash common.Hash) ([]byte, error) {
// It doesn't make sense to retrieve the metaroot
if hash == (common.Hash{}) {
return nil, errors.New("not found")
@@ -198,11 +193,14 @@ func (db *Database) Node(hash common.Hash) ([]byte, error) {
return enc, nil
}
}
// Retrieve the node from the dirty cache if available
// Retrieve the node from the dirty cache if available.
db.lock.RLock()
dirty := db.dirties[hash]
db.lock.RUnlock()

// Return the cached node if it's found in the dirty set.
// The dirty.node field is immutable and safe to read it
// even without lock guard.
if dirty != nil {
memcacheDirtyHitMeter.Mark(1)
memcacheDirtyReadMeter.Mark(int64(len(dirty.node)))
@@ -223,20 +221,6 @@ func (db *Database) Node(hash common.Hash) ([]byte, error) {
return nil, errors.New("not found")
}

// Nodes retrieves the hashes of all the nodes cached within the memory database.
// This method is extremely expensive and should only be used to validate internal
// states in test code.
func (db *Database) Nodes() []common.Hash {
db.lock.RLock()
defer db.lock.RUnlock()

var hashes = make([]common.Hash, 0, len(db.dirties))
for hash := range db.dirties {
hashes = append(hashes, hash)
}
return hashes
}

// Reference adds a new reference from a parent node to a child node.
// This function is used to add reference between internal trie node
// and external node(e.g. storage trie root), all internal trie nodes
@@ -344,16 +328,16 @@ func (db *Database) dereference(hash common.Hash) {

// Cap iteratively flushes old but still referenced trie nodes until the total
// memory usage goes below the given threshold.
//
// Note, this method is a non-synchronized mutator. It is unsafe to call this
// concurrently with other mutators.
func (db *Database) Cap(limit common.StorageSize) error {
db.lock.Lock()
defer db.lock.Unlock()

// Create a database batch to flush persistent data out. It is important that
// outside code doesn't see an inconsistent state (referenced data removed from
// memory cache during commit but not yet in persistent storage). This is ensured
// by only uncaching existing data when the database write finalizes.
nodes, storage, start := len(db.dirties), db.dirtiesSize, time.Now()
batch := db.diskdb.NewBatch()
nodes, storage, start := len(db.dirties), db.dirtiesSize, time.Now()

// db.dirtiesSize only contains the useful data in the cache, but when reporting
// the total memory consumption, the maintenance metadata is also needed to be
@@ -391,9 +375,6 @@ func (db *Database) Cap(limit common.StorageSize) error {
return err
}
// Write successful, clear out the flushed data
db.lock.Lock()
defer db.lock.Unlock()

for db.oldest != oldest {
node := db.dirties[db.oldest]
delete(db.dirties, db.oldest)
@@ -424,10 +405,10 @@ func (db *Database) Cap(limit common.StorageSize) error {
// Commit iterates over all the children of a particular node, writes them out
// to disk, forcefully tearing down all references in both directions. As a side
// effect, all pre-images accumulated up to this point are also written.
//
// Note, this method is a non-synchronized mutator. It is unsafe to call this
// concurrently with other mutators.
func (db *Database) Commit(node common.Hash, report bool) error {
db.lock.Lock()
defer db.lock.Unlock()

// Create a database batch to flush persistent data out. It is important that
// outside code doesn't see an inconsistent state (referenced data removed from
// memory cache during commit but not yet in persistent storage). This is ensured
@@ -449,8 +430,6 @@ func (db *Database) Commit(node common.Hash, report bool) error {
return err
}
// Uncache any leftovers in the last batch
db.lock.Lock()
defer db.lock.Unlock()
if err := batch.Replay(uncacher); err != nil {
return err
}
@@ -499,13 +478,11 @@ func (db *Database) commit(hash common.Hash, batch ethdb.Batch, uncacher *cleane
if err := batch.Write(); err != nil {
return err
}
db.lock.Lock()
err := batch.Replay(uncacher)
batch.Reset()
db.lock.Unlock()
if err != nil {
return err
}
batch.Reset()
}
return nil
}
@@ -574,7 +551,7 @@ func (db *Database) Initialized(genesisRoot common.Hash) bool {
func (db *Database) Update(root common.Hash, parent common.Hash, block uint64, nodes *trienode.MergedNodeSet, states *triestate.Set) error {
// Ensure the parent state is present and signal a warning if not.
if parent != types.EmptyRootHash {
if blob, _ := db.Node(parent); len(blob) == 0 {
if blob, _ := db.node(parent); len(blob) == 0 {
log.Error("parent state is not present")
}
}
@@ -655,7 +632,7 @@ func (db *Database) Scheme() string {
// Reader retrieves a node reader belonging to the given state root.
// An error will be returned if the requested state is not available.
func (db *Database) Reader(root common.Hash) (*reader, error) {
if _, err := db.Node(root); err != nil {
if _, err := db.node(root); err != nil {
return nil, fmt.Errorf("state %#x is not available, %v", root, err)
}
return &reader{db: db}, nil
@@ -666,9 +643,9 @@ type reader struct {
db *Database
}

// Node retrieves the trie node with the given node hash.
// No error will be returned if the node is not found.
// Node retrieves the trie node with the given node hash. No error will be
// returned if the node is not found.
func (reader *reader) Node(owner common.Hash, path []byte, hash common.Hash) ([]byte, error) {
blob, _ := reader.db.Node(hash)
blob, _ := reader.db.node(hash)
return blob, nil
}