v3.0.0-alpha1

Erigon3 user guide

RAM requirement is higher: 64gb. We will reduce it in next releases.

Golang 1.21

Almost all RPC methods are implemented - if something doesn't work - just drop it on our head.

Supported networks: all (except Mumbai).

E3 changes from E2:

• Sync from scratch doesn't require re-exec all history. Latest state and it's history are in snapshots - can download.
• ExecutionStage - now including many E2 stages: stage_hash_state, stage_trie, stage_log_index, stage_history_index,
  stage_trace_index
• E3 can execute 1 historical transaction - without executing it's block - because history/indices have
  transaction-granularity, instead of block-granularity.
• E3 doesn't store Logs (aka Receipts) - it always re-executing historical txn (but it's cheaper then in E2 - see point
  above). Known perf issues: #10747
• --sync.loop.block.limit is enabled by default. (Default: 5_000.
  Set --sync.loop.block.limit=10_000 --batchSize=2g to increase sync speed on good hardware).
• datadir/chaindata is small now - to prevent it's grow: we recommend set --batchSize <= 2G. And it's fine
  to rm -rf chaindata
• can symlink/mount latest state to fast drive and history to cheap drive
• Archive Node is default. Full Node: --prune.mode=full, Minimal Node (EIP-4444): --prune.mode=minimal

Known Problems of E3:

• eth_getLogs fields index always 0: #10324

E3 datadir structure

datadir        
    chaindata   # "Recently-updated Latest State" and "Recent History"
    snapshots   
        domain    # Latest State: link to fast disk
        history   # Historical values 
        idx       # InvertedIndices: can search/filtering/union/intersect them - to find historical data. like eth_getLogs or trace_transaction
        accessors # Additional (generated) indices of history - have "random-touch" read-pattern. They can serve only `Get` requests (no search/filters).
    temp # buffers to sort data >> RAM. sequential-buffered IO - is slow-disk-friendly
   
# There is 4 domains: account, storage, code, commitment 

E3 can store state on fast disk and history on cheap disk

If you can afford store datadir on 1 nvme-raid - great. If can't - it's possible to store history on cheap drive.

# place (or ln -s) `datadir` on slow disk. link some sub-folders to fast disk.
# Example: what need link to fast disk to speedup execution
datadir        
    chaindata   # link to fast disk
    snapshots   
        domain    # link to fast disk
        history   
        idx       
        accessors 
    temp   

# Example: how to speedup history access: 
#   - go step-by-step - first try store `accessors` on fast disk
#   - if speed is not good enough: `idx`
#   - if still not enough: `history` 

E3 datadir size

# eth-mainnet - archive - April 2024

du -hsc /erigon/* 
6G  	/erigon/caplin
50G 	/erigon/chaindata
1.8T	/erigon/snapshots
1.9T	total

du -hsc /erigon/snapshots/* 
100G 	/erigon/snapshots/accessor
240G	/erigon/snapshots/domain
260G	/erigon/snapshots/history
410G	/erigon/snapshots/idx
1.7T	/erigon/snapshots

# bor-mainnet - archive - Jun 2024

du -hsc /erigon/* 

160M	/erigon/bor
50G 	/erigon/chaindata
3.7T	/erigon/snapshots
3.8T	total

du -hsc /erigon/snapshots/* 
260G	/erigon-data/snapshots/accessor
850G	/erigon-data/snapshots/domain
650G	/erigon-data/snapshots/history
1.4T	/erigon-data/snapshots/idx
4.1T	/erigon/snapshots

See also: https://github.com/erigontech/erigon?tab=readme-ov-file#erigon3-users-guide

Upgrade from Erigon2

No. Please re-sync.

v2.60.4

Changes:

• PIP-35 for amoy: enforce 25gwei gas config for amoy for erigon 2 by @manav2401 in #11078

Full Changelog: v2.60.3...v2.60.4

v2.60.3

Improvements:

• eth/tracers: add optional includePrecompiles flag to callTracer - default true is preserved by @taratorio in #10986
• turbo/jsonrpc: add optional includePrecompiles flag to trace_* apis by @taratorio in #10979

Bugfixes:

• Changing Caplin Finality Checkpoint API response to match spec by @angusscott in #10944
• Caplin: Added Blob Sidecar PastFinalization Check by @Giulio2002 in #11006
• Less troublesome way of identifying content-type (#10770) by @Giulio2002 in #11018
• Allow to gracefully exit from CL downloading stage (#10887) by @awskii in #11020
• Add zero check in tx.Sender func by @somnathb1 in #10737
• eth/tracers: fix prestate tracer bug with create with value by @taratorio in #10960
• eth/tracers: always pop precompiles stack in callTracer by @taratorio in #11004
• Diagnostics: loglevel by @dvovk in #11015
• Diagnostics: Optimize db write by @dvovk in #11016

Full Changelog: v2.60.2...v2.60.3

v2.60.2

Bugfixes:

• Revert breaking change to eth_estimateGas introduced in v2.60.1 (#10904) as we added a better fix for gas fee calculation in debug calls by @mininny in #10880
• Fix potential p2p shutdown hangup by @mh0lt in #10626
• Downloader: fix staticpeers flag by @dvovk in #10798
• rpc: Fix incorrect txfeecap by @shohamc1 in #10643

Full Changelog: v2.60.1...v2.60.2

v2.60.1

Breaking change:

• 2nd argument blockNrOrHash was removed from eth_estimateGas. Moreover, eth_estimateGas now correctly defaults MaxFeePerGas to block's base fee by @yperbasis in #10499

Improvements:

• Set tracer-observable value of a delegatecall to match parent value by @AskAlexSharov in #10370

Bugfixes:

• Previously eth_callMany and other RPC endpoints could have been wrong because baseFee was overwritten by maxFeePerGas. Fixed by @marshalys in #10456
• Fixed "method handler crashed" for debug_traceCall (Issue #9090) fixed by @marshalys in #10502
• Fix a regression with json marshalling of checkpoints causing root hash to be 0 and triggering unwinds by @taratorio in #10530
• Fix unintended polygon bor miner gas price and txpool price limit sanitization on ethereum by @taratorio in #10420
• Enable DNS p2p discovery on holesky by @wmitsuda in #10460
• p2p/sentry: sentry doesn't start with ErrNoHead (#10454) by @battlmonstr in #10523
• Caplin: Proper "Normalization" of length of ForkVersions to 8 hex characters by @Giulio2002 in #10578
• bor blocks retire: infinity loop fix by @AskAlexSharov in #10596
• txpool: EIP-3860 should only apply to create transactions by @yperbasis in #10609

Full Changelog: v2.60.0...v2.60.1

v2.60.0 The Last of Erigon 2

WARNING

Erigon 3 has been under R&D for a long time and is getting closer to release. We plan this release, v2.60.0, to be the last significant release based on Erigon 2. There might be patch v2.60.x releases for critical issues, but by and large, we intend all future development to be based on Erigon 3. The code of Erigon 3 now lives in branch main, which is our default branch now. We advise all forks of Erigon to either switch their development to Erigon 3 or use branch release/2.60 if they choose to stay on Erigon 2.
You can read more about Erigon 3 at https://erigon.tech/merging-erigon-3-and-erigon-4-and-possibly-caplin.

Drop support of golang 1.20

Fixes pruning of logs issue introduced in 2.59.x:

For users using prune=hrtc flag or any prune flags with non-zero prune.r.older value following must be performed when upgrading from 2.59.x

• Stop the node, upgrade to v2.60.0
• Then,

integration stage_log_index --reset --datadir=<your_datadir> --chain=<chain_name>

• Then restart and let it sync to the tip.

(Optional) Database compaction: Stop the node at chain-tip (after syncing is finished) Then

integration mdbx_to_mdbx --chaindata=<your_datadir>/chaindata --chaindata.to=<your_datadir>/chaindata2 --datadir=<any_random_location_for_temp_files>
mv chaindata chaindataOld
mv chaindata2 chaindata

Then restart. The new chaindata may be up to 1/3 of the size of the old (Bonus!). If all goes well, you can delete chaindataOld

rm -rf chaindataOld

Tip: To do the integration commands with docker image, you can always do

docker run -it -v <your_datadir>:/data --entrypoint sh thorax/erigon:devel

Then the value of datadir for integration and mdbx commands is /data on the docker shell, where you can run the above.

N.B. Also, just a reminder, 2.58 originally synced node cannot see much benefit on upgrading (it was also in the release notes of 2.59) The above suggestion would work when upgrading from 2.59.x

by @somnathb1 in #10019 #9968 #9932 #9733

Caplin:

• Caplin: beacon committee subscription api by @domiwei in #9721
• Caplin: tweaks to make staking more stable. by @Giulio2002 in #10097
• Caplin: Fixed not calling FCU due to faulty blob handling by @Giulio2002 in #9779
• Caplin: process new attesting indicies before block comes in to avoid occasiona Reorg by @Giulio2002 in #10085

Silkworm:

• Fix arm64 Silkworm build (#9299) by @battlmonstr in #9914
• Add support for customising Silkworm RpcDaemon settings in Erigon++ by @canepat in #10002

Otterscan:

• Fixed rewards and fees calculation in Otterscan APIs: #10038 #10070
• Added support for return values in ots_traceTransaction traces: #10014"

New:

• Grafana: configurable datasource by @AskAlexSharov in #10073
• Enabled diagnostics by default to collect data by @dvovk in #10083
• Added command to verify remote manifests from webseeds by @awskii in #9762
• Add aditional types for handling bor waypoint persistnce and refactor snapshot management code to improve extensibility. #10027 #10051 #9793 #10132 #10147
• RPC: Receipts LRU cache by @AskAlexSharov in #10112

Bugfixes:

• Fix default gas values in debug_ RPCs by @shohamc1 in #9922
• Fixed p2p/sentry: StatusDataProvider ReadCurrentHeader error by @battlmonstr in #9890
• Fixed Engine API: NewPayload fails with a "context canceled" error when calling CurrentHeader/GetHeader (#9786) by @battlmonstr in #9894
• Fix abigen regression which generated non-compilable code with unused imports and duplicate struct definitions by @taratorio in #10091
• Release decompressor mmap on errors by @awskii in #9917
• Web Torrent related fixes to improve downloader reliability. For newly released snapshot files downloading is
  primarily via webseeds. This has become unreliable recently due to issues with web firewall quotas and internal
  torrent library http handling. These changes fix several identified issues in this area which should improve downlaod reliability.#9904 #9962 #10008 #10036 #10183 #10149

Full Changelog: v2.59.3...v2.60.0

v2.59.3 Napoli on Amoy

This release sets the Napoli hard fork on the Amoy test net at block #5423600.

What's Changed

• Napoli block for Amoy network by @anshalshukla in #9840

Full Changelog: v2.59.2...v2.59.3

v2.59.2

This patch release fixes an MDBX size/performance regression introduced in v2.58.0.

Changelog

• a013ec2 Revert "updateForkChoice: support --sync.loop.block.limit (#9315)" (#9833)

v2.59.1

What's Changed

• Caplin: Fixed not calling FCU due to faulty blob handling by @Giulio2002 in #9818
• fixed downloading of unnecessary blocks when --caplin.backfilling=f… by @Giulio2002 in #9820
• Proper Caplin's subscription listen by @Giulio2002 in #9821
• Added basic block production to Caplin by @Giulio2002 in #9822
• downloader: when torrent added to lib and metadata resolved - already… by @AskAlexSharov in #9823

Full Changelog: v2.59.0...v2.59.1

v2.59.0 Post-Dencun Fixes

Attention Caplin Users

1. Upgrade your erigon node, let it run for about a minute, then stop it.
2. Reset the headers & snapshots stages:

rm -rf <datadir>/caplin <datadir>/snapshots <datadir>/downloader
make integration
./build/bin/integration stage_headers --reset --datadir=<datadir>
./build/bin/integration stage_snapshots --reset --datadir=<datadir> 

3. Start your erigon node.

More Aggressive Log/Receipt Pruning

PR #9123 introduces more aggressive pruning of logs & receipts (for nodes with r in their pruning flag, e.g. --prune=hrtc), resulting in a 2x smaller disk footprint.
This feature is only enabled for new nodes. To enable it for an existing node, you have to re-sync it from scratch.

Optional reduce IO for getTxnByHash operation

Transaction lookup by hash did touch too much places in snapshots. It caused high IO and PageCache evictions.
If you have much RPC requests - it will greatly help for this use-case.
To get it: rm datdir/snapshots/*.idx and restart erigon.
By #9506

Release Highlights

• Caplin: blob storage by @bayram98 in #9310
• txpool: Respect updates to block gas limit by @jyellick in #9363
• seg compress: fix LCP initialization by @battlmonstr in #9460
• Filter deposit contract logs before pruning by @somnathb1 in #9123
• metrics: Add memory usage statistics to Prometheus by @shohamc1 in #9465
• "integration warmup": open db in asccede mode - to allow run it without erigon shutdown by @AskAlexSharov in #9558
• debug_traceBlockByHash: reject non-canonical hash by @AskAlexSharov in #9616
• rpc: trace_transaction return error if block is non-canonical by @AskAlexSharov in #9641
• Fix txpool_content for pending blob tx by @somnathb1 in #9668
• Fixed gnosis DA on Caplin by @Giulio2002 in #9682
• support --sync.loop.block.limit in newHeads notification by @AskAlexSharov in #9684
• Allow safe/finalized for eth_getLogs by @adytzu2007 in #9686
• silkworm: proper external/internal txn handling in stage Execution by @canepat in #9594
• Caplin: Added preparation for validator block building by @Giulio2002 in #9709
• evm transaction context not correct when txn is type 3 (EIP-4844) by @indanielo in #9722
• Fix ExcessBlobGas wiring in callMany/debug_traceCallMany by @yperbasis in #9723
• RPC: close tx.Prefix iterator for early release memory by @AskAlexSharov in #9759

New Contributors

• @sjb933 made their first contribution in #9432
• @bloxster made their first contribution in #9459
• @bodhi-crypo made their first contribution in #9492
• @alecalve made their first contribution in #9595
• @adytzu2007 made their first contribution in #9686
• @pavedroad made their first contribution in #9685
• @indanielo made their first contribution in #9722
• @AdventureSeeker987 made their first contribution in #9664
• @domiwei made their first contribution in #9735

Full Changelog: v2.58.2...v2.59.0