After 10 days synchronising mainnet, erigon restarts and all data is lost

[manumonti]

Hi!

I have been trying to start an Erigon node for Ethereum mainnet.

I'm using a LG Gram laptop with Ubuntu (i5 8250U, 16GB RAM, 2TB nvme).

I'm also running an nimbus instance as beacon node.

I run the following command:

./erigon --datadir="/data/erigon/datadir" --authrpc.jwtsecret="/data/erigon/jwtsecret" --http --http.addr="0.0.0.0" --http.port="8545" --http.compression --http.vhosts="*" --http.corsdomain="*" --http.api="eth,admin,debug,net,trace,web3,erigon" --ws --ws.compression --prune=hrtc

After 10 days syncronizing, most of the time in the [7/15 Execution] Executed blocks state, it started again and all the data have been flushed. It was very close to reaching the current mainnet block.

[INFO] [06-15|18:02:11.671] [7/15 Execution] Executed blocks         number=16152021 blk/s=0.1 tx/s=6.9 Mgas/s=1.0 gasState=0.00 batch=0B alloc=3.9GB sys=11.9GB
[INFO] [06-15|18:02:13.880] [p2p] GoodPeers                          eth68=100 eth66=41 eth67=59
[INFO] [06-15|18:02:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=4.0GB sys=11.9GB
[INFO] [06-15|18:02:14.488] [7/15 Execution] Executed blocks         number=16152032 blk/s=3.9 tx/s=474.9 Mgas/s=63.0 gasState=0.00 batch=166.1KB alloc=4.0GB sys=11.9GB
[INFO] [06-15|18:02:34.457] [7/15 Execution] Executed blocks         number=16152180 blk/s=7.4 tx/s=840.8 Mgas/s=112.0 gasState=0.00 batch=2.9MB alloc=4.9GB sys=11.9GB
[INFO] [06-15|18:02:54.480] [7/15 Execution] Executed blocks         number=16152344 blk/s=8.2 tx/s=938.2 Mgas/s=125.0 gasState=0.01 batch=6.1MB alloc=5.8GB sys=11.9GB
[INFO] [06-15|18:03:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=6.9GB sys=11.9GB
[INFO] [06-15|18:03:14.557] [7/15 Execution] Executed blocks         number=16152532 blk/s=9.4 tx/s=1171.8 Mgas/s=145.5 gasState=0.01 batch=9.8MB alloc=7.0GB sys=11.9GB
[INFO] [06-15|18:03:34.575] [7/15 Execution] Executed blocks         number=16152655 blk/s=6.1 tx/s=1721.6 Mgas/s=91.7 gasState=0.02 batch=11.8MB alloc=3.5GB sys=11.9GB
[INFO] [06-15|18:03:54.580] [7/15 Execution] Executed blocks         number=16152799 blk/s=7.2 tx/s=1651.5 Mgas/s=108.1 gasState=0.02 batch=14.6MB alloc=4.6GB sys=11.9GB
[INFO] [06-15|18:04:13.880] [p2p] GoodPeers                          eth68=100 eth67=59 eth66=41
[INFO] [06-15|18:04:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=3.1GB sys=11.9GB
[INFO] [06-15|18:04:14.469] [7/15 Execution] Executed blocks         number=16152968 blk/s=8.5 tx/s=1235.6 Mgas/s=128.8 gasState=0.03 batch=18.1MB alloc=3.1GB sys=11.9GB
[INFO] [06-15|18:04:34.592] [7/15 Execution] Executed blocks         number=16153187 blk/s=10.9 tx/s=1253.0 Mgas/s=166.8 gasState=0.03 batch=23.1MB alloc=4.2GB sys=11.9GB
[INFO] [06-15|18:04:54.583] [7/15 Execution] Executed blocks         number=16153373 blk/s=9.3 tx/s=1025.3 Mgas/s=142.1 gasState=0.04 batch=27.1MB alloc=5.2GB sys=11.9GB
[INFO] [06-15|18:05:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=3.4GB sys=11.9GB
[INFO] [06-15|18:05:14.547] [7/15 Execution] Executed blocks         number=16153574 blk/s=10.1 tx/s=1178.8 Mgas/s=154.1 gasState=0.04 batch=31.8MB alloc=3.5GB sys=11.9GB
[INFO] [06-15|18:05:34.463] [7/15 Execution] Executed blocks         number=16153796 blk/s=11.1 tx/s=1255.0 Mgas/s=169.1 gasState=0.05 batch=37.0MB alloc=4.5GB sys=11.9GB
[INFO] [06-15|18:05:54.543] [7/15 Execution] Executed blocks         number=16154001 blk/s=10.2 tx/s=1156.4 Mgas/s=157.3 gasState=0.05 batch=41.6MB alloc=3.0GB sys=11.9GB
[INFO] [06-15|18:06:13.880] [p2p] GoodPeers                          eth68=100 eth67=59 eth66=41
[INFO] [06-15|18:06:13.928] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=4.0GB sys=11.9GB
[INFO] [06-15|18:06:14.508] [7/15 Execution] Executed blocks         number=16154214 blk/s=10.7 tx/s=1415.8 Mgas/s=162.5 gasState=0.06 batch=46.4MB alloc=4.1GB sys=11.9GB
[INFO] [06-15|18:06:34.551] [7/15 Execution] Executed blocks         number=16154420 blk/s=10.3 tx/s=1788.6 Mgas/s=155.8 gasState=0.07 batch=51.1MB alloc=5.2GB sys=11.9GB
[INFO] [06-15|18:06:54.496] [7/15 Execution] Executed blocks         number=16154612 blk/s=9.6 tx/s=1464.4 Mgas/s=145.6 gasState=0.07 batch=55.6MB alloc=3.6GB sys=11.9GB
[INFO] [06-15|18:07:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=4.7GB sys=11.9GB
[INFO] [06-15|18:07:14.532] [7/15 Execution] Executed blocks         number=16154802 blk/s=9.5 tx/s=1695.1 Mgas/s=145.1 gasState=0.08 batch=60.3MB alloc=4.7GB sys=11.9GB
[INFO] [06-15|18:07:34.556] [7/15 Execution] Executed blocks         number=16154958 blk/s=7.8 tx/s=1970.2 Mgas/s=119.9 gasState=0.08 batch=63.5MB alloc=3.1GB sys=11.9GB
[INFO] [06-15|18:07:54.574] [7/15 Execution] Executed blocks         number=16155125 blk/s=8.3 tx/s=1800.7 Mgas/s=123.9 gasState=0.09 batch=67.5MB alloc=4.2GB sys=11.9GB
[INFO] [06-15|18:08:13.880] [p2p] GoodPeers                          eth66=41 eth67=59 eth68=100
[INFO] [06-15|18:08:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=5.4GB sys=11.9GB
[INFO] [06-15|18:08:14.481] [7/15 Execution] Executed blocks         number=16155301 blk/s=8.8 tx/s=1978.5 Mgas/s=134.2 gasState=0.09 batch=72.1MB alloc=5.5GB sys=11.9GB
[INFO] [06-15|18:08:34.595] [7/15 Execution] Executed blocks         number=16155483 blk/s=9.0 tx/s=1683.7 Mgas/s=135.7 gasState=0.10 batch=77.3MB alloc=3.8GB sys=11.9GB
[INFO] [06-15|18:08:54.500] [7/15 Execution] Executed blocks         number=16155665 blk/s=9.1 tx/s=2489.4 Mgas/s=138.5 gasState=0.10 batch=81.9MB alloc=5.0GB sys=11.9GB
[INFO] [06-15|18:09:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=3.3GB sys=11.9GB
[INFO] [06-15|18:09:14.704] [7/15 Execution] Executed blocks         number=16155855 blk/s=9.4 tx/s=1615.0 Mgas/s=141.5 gasState=0.11 batch=87.5MB alloc=3.4GB sys=11.9GB
[INFO] [06-15|18:09:34.585] [7/15 Execution] Executed blocks         number=16156057 blk/s=10.2 tx/s=1504.6 Mgas/s=153.7 gasState=0.11 batch=93.7MB alloc=4.7GB sys=11.9GB
[INFO] [06-15|18:09:54.457] [7/15 Execution] Executed blocks         number=16156262 blk/s=10.3 tx/s=1347.4 Mgas/s=157.5 gasState=0.12 batch=99.8MB alloc=5.8GB sys=11.9GB
[INFO] [06-15|18:10:13.881] [p2p] GoodPeers                          eth68=100 eth67=59 eth66=41
[INFO] [06-15|18:10:13.928] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=4.2GB sys=11.9GB
[INFO] [06-15|18:10:15.992] [7/15 Execution] Executed blocks         number=16156476 blk/s=9.9 tx/s=1202.4 Mgas/s=150.6 gasState=0.12 batch=105.7MB alloc=4.2GB sys=11.9GB
[INFO] [06-15|18:10:34.540] [7/15 Execution] Executed blocks         number=16156671 blk/s=10.5 tx/s=1615.9 Mgas/s=165.0 gasState=0.13 batch=111.3MB alloc=5.4GB sys=11.9GB
[INFO] [06-15|18:10:54.477] [7/15 Execution] Executed blocks         number=16156873 blk/s=10.1 tx/s=1765.9 Mgas/s=149.5 gasState=0.13 batch=116.3MB alloc=3.5GB sys=11.9GB
[INFO] [06-15|18:11:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=4.6GB sys=11.9GB
[INFO] [06-15|18:11:14.552] [7/15 Execution] Executed blocks         number=16157084 blk/s=10.5 tx/s=1223.5 Mgas/s=159.1 gasState=0.14 batch=122.0MB alloc=4.7GB sys=11.9GB
[INFO] [06-15|18:11:34.598] [7/15 Execution] Executed blocks         number=16157310 blk/s=11.3 tx/s=1400.9 Mgas/s=171.6 gasState=0.15 batch=128.7MB alloc=5.8GB sys=11.9GB
[INFO] [06-15|18:11:54.485] [7/15 Execution] Executed blocks         number=16157522 blk/s=10.7 tx/s=1178.8 Mgas/s=162.4 gasState=0.15 batch=134.5MB alloc=3.9GB sys=11.9GB
[INFO] [06-15|18:12:13.880] [p2p] GoodPeers                          eth68=100 eth67=59 eth66=41
[INFO] [06-15|18:12:13.928] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=5.0GB sys=11.9GB
[INFO] [06-15|18:12:14.557] [7/15 Execution] Executed blocks         number=16157748 blk/s=11.3 tx/s=1234.2 Mgas/s=171.7 gasState=0.16 batch=140.9MB alloc=5.0GB sys=11.9GB
[INFO] [06-15|18:12:34.479] [7/15 Execution] Executed blocks         number=16157971 blk/s=11.2 tx/s=1280.4 Mgas/s=170.4 gasState=0.16 batch=146.9MB alloc=3.1GB sys=11.9GB
[INFO] [06-15|18:12:54.533] [7/15 Execution] Executed blocks         number=16158184 blk/s=10.6 tx/s=1343.3 Mgas/s=161.8 gasState=0.17 batch=152.7MB alloc=4.3GB sys=11.9GB
[INFO] [06-15|18:13:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=5.4GB sys=11.9GB
[INFO] [06-15|18:13:14.542] [7/15 Execution] Executed blocks         number=16158398 blk/s=10.7 tx/s=1260.3 Mgas/s=162.2 gasState=0.18 batch=158.2MB alloc=5.4GB sys=11.9GB
[INFO] [06-15|18:13:34.483] [7/15 Execution] Executed blocks         number=16158611 blk/s=10.7 tx/s=1289.2 Mgas/s=163.0 gasState=0.18 batch=163.7MB alloc=3.5GB sys=11.9GB
[INFO] [06-15|18:13:54.524] [7/15 Execution] Executed blocks         number=16158828 blk/s=10.8 tx/s=1282.2 Mgas/s=165.1 gasState=0.19 batch=169.2MB alloc=4.6GB sys=11.9GB
[INFO] [06-15|18:14:13.880] [p2p] GoodPeers                          eth67=59 eth66=41 eth68=100
[INFO] [06-15|18:14:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=5.7GB sys=11.9GB
[INFO] [06-15|18:14:14.490] [7/15 Execution] Executed blocks         number=16159028 blk/s=10.0 tx/s=1143.2 Mgas/s=152.6 gasState=0.19 batch=174.2MB alloc=5.7GB sys=11.9GB
[INFO] [06-15|18:14:34.618] [7/15 Execution] Executed blocks         number=16159245 blk/s=10.8 tx/s=1168.5 Mgas/s=165.5 gasState=0.20 batch=179.7MB alloc=3.7GB sys=11.9GB
[INFO] [06-15|18:14:54.485] [7/15 Execution] Executed blocks         number=16159471 blk/s=11.4 tx/s=1171.4 Mgas/s=174.1 gasState=0.21 batch=185.2MB alloc=4.8GB sys=11.9GB
[INFO] [06-15|18:15:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=5.9GB sys=11.9GB
[INFO] [06-15|18:15:14.467] [7/15 Execution] Executed blocks         number=16159708 blk/s=11.9 tx/s=1135.2 Mgas/s=182.2 gasState=0.21 batch=190.3MB alloc=5.9GB sys=11.9GB
[INFO] [06-15|18:15:34.524] [7/15 Execution] Executed blocks         number=16159920 blk/s=10.6 tx/s=1633.8 Mgas/s=160.4 gasState=0.22 batch=195.4MB alloc=4.0GB sys=11.9GB
[INFO] [06-15|18:15:54.497] [7/15 Execution] Executed blocks         number=16160111 blk/s=9.6 tx/s=1379.4 Mgas/s=145.2 gasState=0.22 batch=199.5MB alloc=5.0GB sys=11.9GB
[INFO] [06-15|18:16:13.882] [p2p] GoodPeers                          eth68=100 eth66=41 eth67=59
[INFO] [06-15|18:16:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=6.1GB sys=11.9GB
[INFO] [06-15|18:16:14.560] [7/15 Execution] Executed blocks         number=16160291 blk/s=9.0 tx/s=1292.5 Mgas/s=137.5 gasState=0.23 batch=203.5MB alloc=6.1GB sys=11.9GB
[INFO] [06-15|18:16:34.549] [7/15 Execution] Executed blocks         number=16160480 blk/s=9.5 tx/s=1189.9 Mgas/s=144.1 gasState=0.23 batch=207.9MB alloc=4.0GB sys=11.9GB
[INFO] [06-15|18:16:54.458] [7/15 Execution] Executed blocks         number=16160667 blk/s=9.4 tx/s=1023.9 Mgas/s=143.3 gasState=0.24 batch=212.5MB alloc=5.4GB sys=11.9GB
[INFO] [06-15|18:17:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=3.7GB sys=11.9GB
[INFO] [06-15|18:17:14.613] [7/15 Execution] Executed blocks         number=16160845 blk/s=8.8 tx/s=875.2 Mgas/s=136.9 gasState=0.24 batch=217.1MB alloc=3.8GB sys=11.9GB
[INFO] [06-15|18:17:34.492] [7/15 Execution] Executed blocks         number=16161016 blk/s=8.6 tx/s=948.0 Mgas/s=131.1 gasState=0.25 batch=220.7MB alloc=4.7GB sys=11.9GB
[INFO] [06-15|18:17:54.511] [7/15 Execution] Executed blocks         number=16161206 blk/s=9.5 tx/s=1141.6 Mgas/s=145.2 gasState=0.25 batch=225.3MB alloc=5.7GB sys=11.9GB
[INFO] [06-15|18:18:13.881] [p2p] GoodPeers                          eth68=100 eth66=41 eth67=59
[INFO] [06-15|18:18:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=6.8GB sys=11.9GB
[INFO] [06-15|18:18:14.538] [7/15 Execution] Executed blocks         number=16161417 blk/s=10.5 tx/s=1418.4 Mgas/s=160.9 gasState=0.26 batch=230.3MB alloc=6.8GB sys=11.9GB
[INFO] [06-15|18:18:34.570] [7/15 Execution] Executed blocks         number=16161614 blk/s=9.8 tx/s=1541.5 Mgas/s=149.5 gasState=0.27 batch=235.6MB alloc=4.4GB sys=11.9GB
[INFO] [06-15|18:18:54.541] [7/15 Execution] Executed blocks         number=16161788 blk/s=8.7 tx/s=1160.4 Mgas/s=132.7 gasState=0.27 batch=240.3MB alloc=5.4GB sys=11.9GB
[INFO] [06-15|18:19:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=6.5GB sys=11.9GB
[INFO] [06-15|18:19:14.465] [7/15 Execution] Executed blocks         number=16161951 blk/s=8.2 tx/s=1394.5 Mgas/s=129.1 gasState=0.28 batch=244.8MB alloc=6.5GB sys=11.9GB
[INFO] [06-15|18:19:34.485] [7/15 Execution] Executed blocks         number=16162072 blk/s=6.0 tx/s=3141.6 Mgas/s=95.0 gasState=0.28 batch=247.0MB alloc=4.2GB sys=11.9GB
[INFO] [06-15|18:19:54.476] [7/15 Execution] Executed blocks         number=16162216 blk/s=7.2 tx/s=3209.0 Mgas/s=103.8 gasState=0.28 batch=249.9MB alloc=5.3GB sys=11.9GB
[INFO] [06-15|18:20:13.881] [p2p] GoodPeers                          eth68=100 eth66=41 eth67=59
[INFO] [06-15|18:20:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=6.3GB sys=11.9GB
[INFO] [06-15|18:20:14.513] [7/15 Execution] Executed blocks         number=16162380 blk/s=8.2 tx/s=2218.8 Mgas/s=121.2 gasState=0.29 batch=254.2MB alloc=6.4GB sys=11.9GB
[INFO] [06-15|18:20:23.070] Committed State                          gas reached=158650448700 gasTarget=549755813888
[INFO] [06-15|18:20:53.787] [] ETL [2/2] Loading                     into=PlainCodeHash current_prefix=a976f3d9
[INFO] [06-15|18:21:08.807] Write to db                              progress=0.1M/3.8M current table=IncarnationMap
[INFO] [06-15|18:21:13.929] [txpool] stat                            pending=197 baseFee=0 queued=3066 alloc=6.1GB sys=11.9GB
[INFO] [06-15|18:21:18.966] [] Flushed buffer file                   name=/data/erigon/datadir/temp/erigon-sortable-buf-779319483
[INFO] [06-15|18:21:22.026] [] Flushed buffer file                   name=/data/erigon/datadir/temp/erigon-sortable-buf-2598031854
[INFO] [06-15|18:22:38.175] logging to file system                   log dir=/data/erigon/datadir/logs file prefix=erigon log level=info json=false
[INFO] [06-15|18:22:38.176] Build info                               git_branch=HEAD git_tag=v2.44.0 git_commit=f4f10f3b7cad36d6b3e7985cfe974764df53d8f7
[INFO] [06-15|18:22:38.176] Starting Erigon on Ethereum mainnet... 
[INFO] [06-15|18:22:38.178] Maximum peer count                       ETH=100 total=100
[INFO] [06-15|18:22:38.179] starting HTTP APIs                       APIs=eth,admin,debug,net,trace,web3,erigon
[INFO] [06-15|18:22:38.179] torrent verbosity                        level=WRN
[INFO] [06-15|18:22:40.282] Set global gas cap                       cap=50000000
[INFO] [06-15|18:22:40.326] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
[INFO] [06-15|18:22:40.334] Opening Database                         label=chaindata path=/data/erigon/datadir/chaindata
mdbx_setup_dxb:15946 filesize mismatch (expect 2147483648b/524288p, have 463856467968b/113246208p)
[INFO] [06-15|18:22:43.561] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: <nil>, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
[INFO] [06-15|18:22:45.099] Initialising Ethereum protocol           network=1
[INFO] [06-15|18:22:45.099] Disk storage enabled for ethash DAGs     dir=/data/erigon/datadir/ethash-dags count=2
[INFO] [06-15|18:22:45.135] Starting private RPC server              on=127.0.0.1:9090
[INFO] [06-15|18:22:45.135] new subscription to logs established 
[INFO] [06-15|18:22:45.137] rpc filters: subscribing to Erigon events 
[INFO] [06-15|18:22:45.139] new subscription to newHeaders established 
[INFO] [06-15|18:22:45.139] New txs subscriber joined 
[INFO] [06-15|18:22:45.142] Reading JWT secret                       path=/data/erigon/jwtsecret
[INFO] [06-15|18:22:45.143] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true
[INFO] [06-15|18:22:45.144] HTTP endpoint opened                     url=[::]:8545 ws=true ws.compression=true grpc=false
[INFO] [06-15|18:22:45.151] [1/15 Snapshots] Fetching torrent files metadata 
[INFO] [06-15|18:22:45.167] Started P2P networking                   version=68 self=enode://bd7f92ab93f147e0aa59f9c257396f5398639c9063bb441c30559e7b0999a50e2432db283ca2562199f32942d46ee53df760195d692c237fa91ed45c02cbb49a@127.0.0.1:0 name=erigon/v2.44.0-stable-f4f10f3b/linux-amd64/go1.20.4
[INFO] [06-15|18:22:45.168] Started P2P networking                   version=67 self=enode://bd7f92ab93f147e0aa59f9c257396f5398639c9063bb441c30559e7b0999a50e2432db283ca2562199f32942d46ee53df760195d692c237fa91ed45c02cbb49a@127.0.0.1:0 name=erigon/v2.44.0-stable-f4f10f3b/linux-amd64/go1.20.4
[INFO] [06-15|18:22:45.269] [txpool] Started 
[INFO] [06-15|18:22:49.929] [snapshots] Blocks Stat                  blocks=16500k indices=16500k alloc=2.4GB sys=2.5GB
[INFO] [06-15|18:23:09.936] [1/15 Snapshots] Total difficulty index: 1965k/16499k 
[INFO] [06-15|18:23:33.495] [1/15 Snapshots] Total difficulty index: 3728k/16499k 
[INFO] [06-15|18:23:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2346 alloc=2.5GB sys=4.9GB
[INFO] [06-15|18:23:49.930] [1/15 Snapshots] Total difficulty index: 5528k/16499k 
[INFO] [06-15|18:24:13.543] [1/15 Snapshots] Total difficulty index: 7456k/16499k 
[INFO] [06-15|18:24:29.930] [1/15 Snapshots] Total difficulty index: 9257k/16499k 
[INFO] [06-15|18:24:45.100] [p2p] GoodPeers                          eth68=4 eth67=8 eth66=5
[INFO] [06-15|18:24:45.138] [txpool] stat                            pending=201 baseFee=0 queued=2346 alloc=2.6GB sys=5.0GB
[INFO] [06-15|18:24:55.258] [1/15 Snapshots] Total difficulty index: 11184k/16499k 
[INFO] [06-15|18:25:09.930] [1/15 Snapshots] Total difficulty index: 12922k/16499k 
[INFO] [06-15|18:25:29.930] [1/15 Snapshots] Total difficulty index: 14891k/16499k 
[INFO] [06-15|18:25:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2346 alloc=3.0GB sys=5.0GB
[INFO] [06-15|18:25:49.930] [1/15 Snapshots] Total difficulty index: 16148k/16499k 
[INFO] [06-15|18:26:12.289] [1/15 Snapshots] DONE                    in=3m27.138773491s
[INFO] [06-15|18:26:13.057] [7/15 Execution] Blocks execution        from=0 to=16499999
[INFO] [06-15|18:26:33.057] [7/15 Execution] Executed blocks         number=230809 blk/s=11540.4 tx/s=7899.4 Mgas/s=230.7 gasState=0.01 batch=17.4MB alloc=2.6GB sys=5.0GB
[INFO] [06-15|18:26:45.099] [p2p] GoodPeers                          eth68=7 eth67=12 eth66=11
[INFO] [06-15|18:26:45.139] [txpool] stat                            pending=201 baseFee=0 queued=2346 alloc=3.4GB sys=5.0GB
[INFO] [06-15|18:26:53.070] [7/15 Execution] Executed blocks         number=402476 blk/s=8577.8 tx/s=10806.1 Mgas/s=334.3 gasState=0.02 batch=40.6MB alloc=3.1GB sys=5.0GB
[INFO] [06-15|18:27:13.057] [7/15 Execution] Executed blocks         number=530943 blk/s=6427.5 tx/s=9589.1 Mgas/s=354.4 gasState=0.03 batch=60.2MB alloc=2.8GB sys=5.0GB
[INFO] [06-15|18:27:33.057] [7/15 Execution] Executed blocks         number=677042 blk/s=7305.0 tx/s=10599.8 Mgas/s=329.8 gasState=0.05 batch=79.5MB alloc=2.9GB sys=5.0GB
[INFO] [06-15|18:27:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2346 alloc=2.3GB sys=5.0GB
[INFO] [06-15|18:27:53.059] [7/15 Execution] Executed blocks         number=731851 blk/s=2740.2 tx/s=6208.2 Mgas/s=433.7 gasState=0.06 batch=90.5MB alloc=3.3GB sys=5.0GB
[INFO] [06-15|18:28:13.062] [7/15 Execution] Executed blocks         number=769892 blk/s=1901.7 tx/s=5734.2 Mgas/s=334.8 gasState=0.07 batch=102.7MB alloc=3.5GB sys=5.0GB
[INFO] [06-15|18:28:33.058] [7/15 Execution] Executed blocks         number=816700 blk/s=2340.9 tx/s=6356.6 Mgas/s=344.1 gasState=0.09 batch=116.3MB alloc=3.9GB sys=5.0GB
[INFO] [06-15|18:28:45.099] [p2p] GoodPeers                          eth66=10 eth67=11 eth68=14
[INFO] [06-15|18:28:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2346 alloc=3.3GB sys=5.0GB
[INFO] [06-15|18:28:53.057] [7/15 Execution] Executed blocks         number=885912 blk/s=3460.8 tx/s=7861.2 Mgas/s=356.2 gasState=0.10 batch=135.0MB alloc=2.7GB sys=5.0GB
[INFO] [06-15|18:29:13.061] [7/15 Execution] Executed blocks         number=963049 blk/s=3856.1 tx/s=12212.0 Mgas/s=482.6 gasState=0.12 batch=159.9MB alloc=2.4GB sys=5.0GB
[INFO] [06-15|18:29:33.057] [7/15 Execution] Executed blocks         number=1053519 blk/s=4524.3 tx/s=15895.9 Mgas/s=556.2 gasState=0.14 batch=188.1MB alloc=2.8GB sys=5.0GB
[INFO] [06-15|18:29:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2347 alloc=3.6GB sys=5.0GB
[INFO] [06-15|18:29:53.058] [7/15 Execution] Executed blocks         number=1139129 blk/s=4280.4 tx/s=19780.4 Mgas/s=595.2 gasState=0.16 batch=216.6MB alloc=3.4GB sys=5.0GB
[INFO] [06-15|18:30:13.058] [7/15 Execution] Executed blocks         number=1207615 blk/s=3424.2 tx/s=18309.6 Mgas/s=616.1 gasState=0.18 batch=245.1MB alloc=3.8GB sys=5.0GB
[INFO] [06-15|18:30:22.370] Committed State                          gas reached=104361693829 gasTarget=549755813888
[INFO] [06-15|18:30:33.057] [7/15 Execution] Executed blocks         number=1253021 blk/s=2270.4 tx/s=13019.1 Mgas/s=450.0 gasState=0.01 batch=9.1MB alloc=3.4GB sys=5.3GB
[INFO] [06-15|18:30:45.099] [p2p] GoodPeers                          eth68=22 eth67=16 eth66=10
[INFO] [06-15|18:30:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2817 alloc=4.0GB sys=5.3GB
[INFO] [06-15|18:30:53.058] [7/15 Execution] Executed blocks         number=1316010 blk/s=3149.4 tx/s=17085.8 Mgas/s=508.8 gasState=0.03 batch=31.4MB alloc=3.6GB sys=5.3GB
[INFO] [06-15|18:31:13.058] [7/15 Execution] Executed blocks         number=1384217 blk/s=3410.3 tx/s=19528.3 Mgas/s=586.3 gasState=0.05 batch=55.0MB alloc=4.2GB sys=5.3GB
[INFO] [06-15|18:31:33.057] [7/15 Execution] Executed blocks         number=1451703 blk/s=3374.3 tx/s=20468.2 Mgas/s=611.2 gasState=0.07 batch=83.4MB alloc=2.8GB sys=5.3GB
[INFO] [06-15|18:31:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2896 alloc=3.5GB sys=5.3GB
[INFO] [06-15|18:31:53.058] [7/15 Execution] Executed blocks         number=1509523 blk/s=2890.9 tx/s=19059.0 Mgas/s=631.4 gasState=0.09 batch=112.6MB alloc=3.3GB sys=5.3GB
[INFO] [06-15|18:32:13.057] [7/15 Execution] Executed blocks         number=1566762 blk/s=2862.1 tx/s=20631.5 Mgas/s=700.8 gasState=0.12 batch=147.3MB alloc=3.9GB sys=5.3GB
[INFO] [06-15|18:32:33.057] [7/15 Execution] Executed blocks         number=1618937 blk/s=2608.7 tx/s=20990.3 Mgas/s=709.2 gasState=0.14 batch=181.0MB alloc=2.6GB sys=5.3GB
[INFO] [06-15|18:32:45.100] [p2p] GoodPeers                          eth68=29 eth67=19 eth66=11
[INFO] [06-15|18:32:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2896 alloc=3.5GB sys=5.3GB
[INFO] [06-15|18:32:53.057] [7/15 Execution] Executed blocks         number=1681545 blk/s=3130.4 tx/s=21101.3 Mgas/s=719.2 gasState=0.17 batch=212.1MB alloc=3.2GB sys=5.3GB
[INFO] [06-15|18:33:13.080] [7/15 Execution] Executed blocks         number=1719784 blk/s=1909.8 tx/s=14065.0 Mgas/s=589.0 gasState=0.19 batch=237.7MB alloc=2.4GB sys=5.3GB
[INFO] [06-15|18:33:33.092] [7/15 Execution] Executed blocks         number=1721838 blk/s=102.6 tx/s=1315.6 Mgas/s=209.7 gasState=0.20 batch=240.7MB alloc=2.9GB sys=5.3GB
[INFO] [06-15|18:33:45.136] [txpool] stat                            pending=201 baseFee=0 queued=2916 alloc=3.2GB sys=5.3GB
[INFO] [06-15|18:33:53.065] [7/15 Execution] Executed blocks         number=1724125 blk/s=114.5 tx/s=1393.4 Mgas/s=200.4 gasState=0.21 batch=244.4MB alloc=3.4GB sys=5.3GB
[INFO] [06-15|18:34:13.058] [7/15 Execution] Executed blocks         number=1725865 blk/s=87.0 tx/s=885.3 Mgas/s=178.9 gasState=0.21 batch=246.9MB alloc=3.8GB sys=5.3GB
[INFO] [06-15|18:34:33.082] [7/15 Execution] Executed blocks         number=1727212 blk/s=67.3 tx/s=661.1 Mgas/s=165.5 gasState=0.22 batch=248.6MB alloc=4.1GB sys=5.3GB
[INFO] [06-15|18:34:45.099] [p2p] GoodPeers                          eth68=35 eth67=24 eth66=12
[INFO] [06-15|18:34:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2916 alloc=4.3GB sys=5.3GB
[INFO] [06-15|18:34:53.071] [7/15 Execution] Executed blocks         number=1728552 blk/s=67.0 tx/s=857.8 Mgas/s=161.6 gasState=0.22 batch=250.3MB alloc=2.4GB sys=5.3GB
[INFO] [06-15|18:35:09.630] Committed State                          gas reached=127634276628 gasTarget=549755813888
[INFO] [06-15|18:35:13.058] [7/15 Execution] Executed blocks         number=1736369 blk/s=391.1 tx/s=3293.8 Mgas/s=228.4 gasState=0.00 batch=890.6KB alloc=3.8GB sys=5.3GB
[INFO] [06-15|18:35:33.057] [7/15 Execution] Executed blocks         number=1765838 blk/s=1473.5 tx/s=11157.3 Mgas/s=533.0 gasState=0.02 batch=22.8MB alloc=2.5GB sys=5.3GB
[INFO] [06-15|18:35:45.136] [txpool] stat                            pending=201 baseFee=0 queued=2916 alloc=2.9GB sys=5.3GB
[INFO] [06-15|18:35:53.088] [7/15 Execution] Executed blocks         number=1805654 blk/s=1987.8 tx/s=14870.0 Mgas/s=650.1 gasState=0.04 batch=45.2MB alloc=4.2GB sys=5.3GB
[INFO] [06-15|18:36:13.080] [7/15 Execution] Executed blocks         number=1849698 blk/s=2203.1 tx/s=14679.1 Mgas/s=505.2 gasState=0.06 batch=68.0MB alloc=3.4GB sys=5.3GB
[INFO] [06-15|18:36:33.057] [7/15 Execution] Executed blocks         number=1875592 blk/s=1296.2 tx/s=8432.2 Mgas/s=359.3 gasState=0.08 batch=81.9MB alloc=3.5GB sys=5.3GB
[INFO] [06-15|18:36:45.101] [p2p] GoodPeers                          eth67=25 eth68=36 eth66=12
[INFO] [06-15|18:36:45.138] [txpool] stat                            pending=201 baseFee=0 queued=2916 alloc=4.3GB sys=5.3GB
[INFO] [06-15|18:36:53.072] [7/15 Execution] Executed blocks         number=1920339 blk/s=2235.7 tx/s=14791.1 Mgas/s=505.0 gasState=0.09 batch=104.4MB alloc=2.8GB sys=5.3GB
[INFO] [06-15|18:37:13.058] [7/15 Execution] Executed blocks         number=1950774 blk/s=1522.8 tx/s=12776.2 Mgas/s=466.9 gasState=0.11 batch=124.2MB alloc=3.6GB sys=5.3GB
[INFO] [06-15|18:37:33.057] [7/15 Execution] Executed blocks         number=1994490 blk/s=2185.8 tx/s=18225.0 Mgas/s=623.7 gasState=0.13 batch=149.7MB alloc=3.5GB sys=5.3GB
[INFO] [06-15|18:37:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2927 alloc=4.3GB sys=5.3GB
[INFO] [06-15|18:37:53.057] [7/15 Execution] Executed blocks         number=2044836 blk/s=2517.3 tx/s=18806.7 Mgas/s=631.7 gasState=0.16 batch=176.2MB alloc=3.6GB sys=5.3GB
[INFO] [06-15|18:38:13.059] [7/15 Execution] Executed blocks         number=2102443 blk/s=2880.1 tx/s=21350.4 Mgas/s=637.2 gasState=0.18 batch=205.1MB alloc=4.2GB sys=5.3GB
[INFO] [06-15|18:38:33.062] [7/15 Execution] Executed blocks         number=2155908 blk/s=2672.8 tx/s=20162.6 Mgas/s=691.4 gasState=0.20 batch=234.5MB alloc=4.7GB sys=5.3GB
[INFO] [06-15|18:38:45.100] [p2p] GoodPeers                          eth68=43 eth67=31 eth66=13
[INFO] [06-15|18:38:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2927 alloc=3.0GB sys=5.3GB
[INFO] [06-15|18:38:47.636] Committed State                          gas reached=122027033994 gasTarget=549755813888
[INFO] [06-15|18:38:53.059] [7/15 Execution] Executed blocks         number=2196679 blk/s=2038.9 tx/s=16271.6 Mgas/s=498.7 gasState=0.00 batch=626.6KB alloc=4.7GB sys=5.5GB
[INFO] [06-15|18:39:13.059] [7/15 Execution] Executed blocks         number=2250666 blk/s=2699.3 tx/s=21075.4 Mgas/s=709.1 gasState=0.03 batch=29.7MB alloc=2.9GB sys=5.5GB
[INFO] [06-15|18:39:33.234] [7/15 Execution] Executed blocks         number=2287819 blk/s=1841.6 tx/s=14642.7 Mgas/s=489.3 gasState=0.04 batch=50.5MB alloc=3.4GB sys=5.5GB
[INFO] [06-15|18:39:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2927 alloc=4.0GB sys=5.5GB
[INFO] [06-15|18:39:53.078] [7/15 Execution] Executed blocks         number=2306288 blk/s=930.7 tx/s=7858.9 Mgas/s=241.8 gasState=0.05 batch=60.9MB alloc=3.6GB sys=5.5GB
[INFO] [06-15|18:40:13.107] [7/15 Execution] Executed blocks         number=2306715 blk/s=21.3 tx/s=408.2 Mgas/s=36.9 gasState=0.05 batch=61.4MB alloc=3.9GB sys=5.5GB
[INFO] [06-15|18:40:33.080] [7/15 Execution] Executed blocks         number=2312112 blk/s=270.2 tx/s=1167.6 Mgas/s=54.7 gasState=0.06 batch=62.7MB alloc=3.9GB sys=5.5GB
[INFO] [06-15|18:40:45.102] [p2p] GoodPeers                          eth67=31 eth68=51 eth66=13
[INFO] [06-15|18:40:45.138] [txpool] stat                            pending=201 baseFee=0 queued=2929 alloc=4.4GB sys=5.5GB
[INFO] [06-15|18:40:53.060] [7/15 Execution] Executed blocks         number=2317566 blk/s=273.0 tx/s=1844.3 Mgas/s=66.1 gasState=0.06 batch=64.6MB alloc=4.0GB sys=5.5GB
[INFO] [06-15|18:41:13.091] [7/15 Execution] Executed blocks         number=2319520 blk/s=97.5 tx/s=908.3 Mgas/s=45.2 gasState=0.06 batch=65.5MB alloc=4.2GB sys=5.5GB
[INFO] [06-15|18:41:33.093] [7/15 Execution] Executed blocks         number=2320757 blk/s=61.8 tx/s=399.1 Mgas/s=34.2 gasState=0.06 batch=66.0MB alloc=4.3GB sys=5.5GB
[INFO] [06-15|18:41:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2935 alloc=2.6GB sys=5.5GB
[INFO] [06-15|18:41:53.108] [7/15 Execution] Executed blocks         number=2322954 blk/s=109.8 tx/s=660.7 Mgas/s=38.7 gasState=0.06 batch=66.7MB alloc=4.4GB sys=5.5GB
[INFO] [06-15|18:42:13.059] [7/15 Execution] Executed blocks         number=2328295 blk/s=267.7 tx/s=1773.5 Mgas/s=83.5 gasState=0.07 batch=83.0MB alloc=3.0GB sys=5.5GB
[INFO] [06-15|18:42:33.079] [7/15 Execution] Executed blocks         number=2331166 blk/s=143.4 tx/s=1051.7 Mgas/s=64.1 gasState=0.07 batch=154.0MB alloc=3.5GB sys=5.5GB
[INFO] [06-15|18:42:45.100] [p2p] GoodPeers                          eth68=59 eth67=33 eth66=13
[INFO] [06-15|18:42:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2944 alloc=2.6GB sys=5.5GB
[INFO] [06-15|18:42:53.105] [7/15 Execution] Executed blocks         number=2333185 blk/s=100.8 tx/s=725.3 Mgas/s=35.4 gasState=0.07 batch=154.8MB alloc=3.8GB sys=5.5GB
[INFO] [06-15|18:43:13.080] [7/15 Execution] Executed blocks         number=2334633 blk/s=72.5 tx/s=584.1 Mgas/s=37.3 gasState=0.07 batch=155.5MB alloc=3.1GB sys=5.9GB
[INFO] [06-15|18:43:33.102] [7/15 Execution] Executed blocks         number=2336074 blk/s=72.0 tx/s=560.0 Mgas/s=39.3 gasState=0.07 batch=156.3MB alloc=4.5GB sys=5.9GB
[INFO] [06-15|18:43:45.138] [txpool] stat                            pending=201 baseFee=0 queued=2944 alloc=4.5GB sys=5.9GB
[INFO] [06-15|18:43:53.072] [7/15 Execution] Executed blocks         number=2344664 blk/s=430.2 tx/s=3146.1 Mgas/s=149.2 gasState=0.08 batch=161.2MB alloc=4.0GB sys=5.9GB
[INFO] [06-15|18:44:13.060] [7/15 Execution] Executed blocks         number=2359177 blk/s=726.1 tx/s=4760.7 Mgas/s=203.0 gasState=0.09 batch=167.8MB alloc=2.5GB sys=5.9GB
[INFO] [06-15|18:44:33.059] [7/15 Execution] Executed blocks         number=2361760 blk/s=129.2 tx/s=1340.1 Mgas/s=88.2 gasState=0.09 batch=171.7MB alloc=3.7GB sys=5.9GB
[INFO] [06-15|18:44:45.099] [p2p] GoodPeers                          eth68=63 eth67=36 eth66=13
[INFO] [06-15|18:44:45.137] [txpool] stat                            pending=201 baseFee=0 queued=2944 alloc=2.6GB sys=5.9GB
[INFO] [06-15|18:44:53.058] [7/15 Execution] Executed blocks         number=2363443 blk/s=84.2 tx/s=617.0 Mgas/s=64.4 gasState=0.09 batch=174.2MB alloc=2.6GB sys=5.9GB
[INFO] [06-15|18:45:13.073] [7/15 Execution] Executed blocks         number=2366036 blk/s=129.5 tx/s=1033.5 Mgas/s=77.5 gasState=0.09 batch=178.1MB alloc=3.9GB sys=5.9GB
[INFO] [06-15|18:45:33.059] [7/15 Execution] Executed blocks         number=2368637 blk/s=130.1 tx/s=1226.9 Mgas/s=81.1 gasState=0.10 batch=182.3MB alloc=2.6GB sys=5.9GB
[INFO] [06-15|18:45:45.138] [txpool] stat                            pending=201 baseFee=0 queued=2944 alloc=4.5GB sys=5.9GB
[INFO] [06-15|18:45:53.059] [7/15 Execution] Executed blocks         number=2372298 blk/s=183.0 tx/s=1406.8 Mgas/s=106.3 gasState=0.10 batch=185.5MB alloc=3.1GB sys=5.9GB
[INFO] [06-15|18:46:13.064] [7/15 Execution] Executed blocks         number=2381683 blk/s=469.1 tx/s=4055.7 Mgas/s=212.8 gasState=0.11 batch=193.9MB alloc=3.3GB sys=5.9GB
[INFO] [06-15|18:46:33.063] [7/15 Execution] Executed blocks         number=2383606 blk/s=96.2 tx/s=728.2 Mgas/s=45.0 gasState=0.11 batch=195.7MB alloc=4.3GB sys=5.9GB
[INFO] [06-15|18:46:45.100] [p2p] GoodPeers                          eth68=59 eth67=39 eth66=13
[INFO] [06-15|18:46:45.138] [txpool] stat                            pending=201 baseFee=0 queued=2944 alloc=3.1GB sys=5.9GB
[INFO] [06-15|18:46:53.081] [7/15 Execution] Executed blocks         number=2384428 blk/s=41.1 tx/s=348.5 Mgas/s=23.4 gasState=0.11 batch=196.3MB alloc=3.0GB sys=5.9GB
[INFO] [06-15|18:47:13.117] [7/15 Execution] Executed blocks         number=2385474 blk/s=52.2 tx/s=440.2 Mgas/s=26.5 gasState=0.11 batch=197.0MB alloc=4.2GB sys=5.9GB
[INFO] [06-15|18:47:33.095] [7/15 Execution] Executed blocks         number=2386219 blk/s=37.3 tx/s=187.0 Mgas/s=19.0 gasState=0.11 batch=197.3MB alloc=3.6GB sys=5.9GB
[INFO] [06-15|18:47:45.147] [txpool] stat                            pending=201 baseFee=0 queued=2944 alloc=2.9GB sys=5.9GB
[INFO] [06-15|18:47:53.090] [7/15 Execution] Executed blocks         number=2386983 blk/s=38.2 tx/s=257.3 Mgas/s=20.0 gasState=0.11 batch=197.6MB alloc=2.7GB sys=5.9GB
[INFO] [06-15|18:48:13.183] [7/15 Execution] Executed blocks         number=2391958 blk/s=247.6 tx/s=1387.0 Mgas/s=94.1 gasState=0.12 batch=202.7MB alloc=3.2GB sys=5.9GB
[INFO] [06-15|18:48:33.075] [7/15 Execution] Executed blocks         number=2393017 blk/s=53.2 tx/s=488.8 Mgas/s=21.8 gasState=0.12 batch=203.2MB alloc=3.4GB sys=5.9GB

Any idea what could be happening here?

[AskAlexSharov]

v2.45.0 and v2.45.1 had this bug. We fixed it by: https://github.com/ledgerwatch/erigon/releases/tag/v2.45.2

[manumonti]

Thanks for your reply! Actually I'm doing it with v2.44.0, but I'm going to check with v2.45.2

[AskAlexSharov]

then I don't see obvious reason for this

[AskAlexSharov]

oh, i just checked v2.44.0 source - and it has this bug also. I didn't know.

[manumonti]

thanks again. I'll try with v2.45.2 (or even v2.45.3)