Can't sync the lotus node #8001

versionCollector · 2022-01-25T15:41:28Z

versionCollector
Jan 25, 2022

Checklist

This is not a security-related bug/issue. If it is, please follow please follow the security policy.
This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
This is not a new feature request. If it is, please file a feature request instead.
This is not an enhancement request. If it is, please file a improvement suggestion instead.
I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
I did not make any code changes to lotus.

Lotus component

Lotus Version

Daemon:  1.15.0-dev+mainnet+git.da6752ecc.dirty+api1.5.0
Local: lotus version 1.15.0-dev+mainnet+git.da6752ecc.dirty

Describe the Bug

Cant sync a node after importing snapshot. Syncing from genesis seems to have a same problem.

Config file:

[API]
  ListenAddress = "/ip4/0.0.0.0/tcp/1234/http"
  Timeout = "30s"


[Backup]
  #DisableMetadataLog = false

[Libp2p]
  ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0"]
  #AnnounceAddresses = []
  #NoAnnounceAddresses = []
  #DisableNatPortMap = false
  #ConnMgrLow = 150
  #ConnMgrHigh = 180



[Pubsub]
  #Bootstrapper = false
  #RemoteTracer = ""


[Client]
  UseIpfs = false
  #IpfsOnlineMode = false
  #IpfsMAddr = ""
  #IpfsUseForRetrieval = false
  #SimultaneousTransfersForStorage = 20
  #SimultaneousTransfersForRetrieval = 20


[Wallet]
  #RemoteBackend = ""
  #EnableLedger = false
  #DisableLocal = false


[Fees]
  #DefaultMaxFee = "0.07 FIL"

[Chainstore]
  EnableSplitstore = true

  [Chainstore.Splitstore]
    ColdStoreType = "discard"
    #HotStoreType = "badger"
    #MarkSetType = "map"
    #HotStoreMessageRetention = 0
    #HotStoreFullGCFrequency = 20

Logging Information

2022-01-25T15:18:58.478Z	INFO	importmgr	imports/manager.go:99	sanity check completed	{"broken": 0, "total": 0}
2022-01-25T15:18:58.479Z	INFO	splitstore	splitstore/splitstore.go:563	starting splitstore	{"baseEpoch": "0", "warmupEpoch": "0"}
2022-01-25T15:18:58.479Z	INFO	dt-impl	impl/impl.go:152	start data-transfer module
2022-01-25T15:18:58.486Z	INFO	markets	loggers/loggers.go:56	module ready	{"module": "client data transfer"}
2022-01-25T15:18:58.492Z	INFO	markets	loggers/loggers.go:56	module ready	{"module": "discovery"}
2022-01-25T15:18:58.498Z	INFO	markets	loggers/loggers.go:56	module ready	{"module": "retrieval client"}
2022-01-25T15:18:58.504Z	INFO	markets	loggers/loggers.go:56	module ready	{"module": "storage client"}
2022-01-25T15:19:03.475Z	INFO	peermgr	peermgr/peermgr.go:210	connecting to bootstrap peers
2022-01-25T15:19:03.783Z	INFO	chain	chain/sync_manager.go:218	selected initial sync target: [bafy2bzaceagfmpdophqjli5hztbq53kv7sdm4quxolmrceakddjiadw54gvbe bafy2bzacecxtudgsg44w3zglhqnvmdmg2vhndfdigjhelkpj3av7tgzficzd4 bafy2bzacedewezousdsmiwxvnkaxhoekdzlgmealmsjz3htk3si64eykhfpzg bafy2bzaceb5j4kh26aeufqgf7oilgaucg3a6tidqke5oultvjnhsvuu7aeb7i]
2022-01-25T15:19:03.784Z	INFO	chain	chain/sync_manager.go:314	worker 0 syncing in [bafy2bzaceagfmpdophqjli5hztbq53kv7sdm4quxolmrceakddjiadw54gvbe bafy2bzacecxtudgsg44w3zglhqnvmdmg2vhndfdigjhelkpj3av7tgzficzd4 bafy2bzacedewezousdsmiwxvnkaxhoekdzlgmealmsjz3htk3si64eykhfpzg bafy2bzaceb5j4kh26aeufqgf7oilgaucg3a6tidqke5oultvjnhsvuu7aeb7i]
2022-01-25T15:19:03.860Z	INFO	chain	chain/sync.go:789	Got blocks: 1493917 500
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0427688 and block height 1493918: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedewezousdsmiwxvnkaxhoekdzlgmealmsjz3htk3si64eykhfpzg
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01132323 and block height 1493918: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacecxtudgsg44w3zglhqnvmdmg2vhndfdigjhelkpj3av7tgzficzd4
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0442375 and block height 1493918: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceb5j4kh26aeufqgf7oilgaucg3a6tidqke5oultvjnhsvuu7aeb7i
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0142720 and block height 1493918: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceagfmpdophqjli5hztbq53kv7sdm4quxolmrceakddjiadw54gvbe
2022-01-25T15:19:03.936Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:04.275Z	WARN	peermgr	peermgr/peermgr.go:217	failed to connect to bootstrap peer: failed to dial 12D3KooWQZrGH1PxSNZPum99M1zNvjNFM33d1AAu5DcvdHptuU7u:
  * [/ip4/61.147.123.121/tcp/12757] failed to negotiate security protocol: peer id mismatch: expected 12D3KooWQZrGH1PxSNZPum99M1zNvjNFM33d1AAu5DcvdHptuU7u, but remote key matches 12D3KooWDgQrcyZpcMAkbEFSJJYV2qXEMwXX67WTbqpNdbifHaEq
2022-01-25T15:19:06.005Z	WARN	peermgr	peermgr/peermgr.go:217	failed to connect to bootstrap peer: failed to dial 12D3KooWGHpBMeZbestVEWkfdnC9u7p6uFHXL1n7m1ZBqsEmiUzz:
  * [/ip4/61.147.123.111/tcp/12757] failed to negotiate security protocol: peer id mismatch: expected 12D3KooWGHpBMeZbestVEWkfdnC9u7p6uFHXL1n7m1ZBqsEmiUzz, but remote key matches 12D3KooWDqaZkm3oSczUm3dvAJ5aL2rdSeQ5VQbnHRTQNEFShhmc
2022-01-25T15:19:08.786Z	WARN	peermgr	peermgr/peermgr.go:217	failed to connect to bootstrap peer: failed to dial 12D3KooWQnwEGNqcM2nAcPtRR9rAX8Hrg4k9kJLCHoTR5chJfz6d:
  * [/ip4/36.103.232.198/tcp/34721] dial tcp4 0.0.0.0:43201->36.103.232.198:34721: i/o timeout
2022-01-25T15:19:08.910Z	WARN	peermgr	peermgr/peermgr.go:217	failed to connect to bootstrap peer: failed to dial 12D3KooWMKxMkD5DMpSWsW7dBddKxKT7L2GgbNuckz9otxvkvByP:
  * [/ip4/36.103.232.198/tcp/34723] dial tcp4 0.0.0.0:43201->36.103.232.198:34723: i/o timeout
2022-01-25T15:19:11.032Z	WARN	connmgr	go-libp2p-connmgr@v0.3.0/connmgr.go:151	Low on memory, but we only have a few connections	{"num": 50, "low watermark": 150}
2022-01-25T15:19:30.336Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0463143 and block height 1493919: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:30.337Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedb5gpecvn2qhlpox2tozraotlnwaivnxszh2rjuss5isklkpjusm
2022-01-25T15:19:30.337Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:30.376Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f057683 and block height 1493919: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:30.376Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacecbez67nmwjyhnkdadccruqnwo53qcobdbssdmziibqkznwv5pj4q
2022-01-25T15:19:30.376Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:30.382Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01424422 and block height 1493919: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:30.382Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebhbd47zzqdlhv77mucyp2ahjbweao76fddsoslfslazdac4tti6s
2022-01-25T15:19:30.382Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:30.394Z	WARN	connmgr	go-libp2p-connmgr@v0.3.0/connmgr.go:151	Low on memory, but we only have a few connections	{"num": 113, "low watermark": 150}
2022/01/25 15:19:30 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022-01-25T15:19:30.704Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0442370 and block height 1493919: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:30.704Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceahvjtkbwyf62bgjrsco4gmbtrfziekhfy6vvygbpx6al6od6doeg
2022-01-25T15:19:30.704Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:30.704Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0500878 and block height 1493919: (get sset) failed to load miner actor: actor not found
2022-01-25T15:19:30.704Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebzu3y22f32pg2dpw6qbirauhp7nf4bl6y4ul4uij2lxiz6uaxwzc
2022-01-25T15:19:30.704Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:19:54.017Z	WARN	connmgr	go-libp2p-connmgr@v0.3.0/connmgr.go:154	Low on memory. Closing 13 connections.
2022-01-25T15:20:00.303Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f020330 and block height 1493920: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:00.303Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebipxue3mxyxobzpfv4u27stuzr3dplhxtnmtihnuwpddzz4xdi5c
2022-01-25T15:20:00.303Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:00.436Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01501599 and block height 1493920: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:00.436Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceaqejyklqqk4psy2hklaag7365dmzc4bs5hfzv7atwily3zxds4bg
2022-01-25T15:20:00.436Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:00.436Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01030435 and block height 1493920: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:00.436Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebiwywxfhxyvmxlh4njxobkhaoyvv3tvakn7gi47kfthijnpxkz2s
2022-01-25T15:20:00.436Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:00.536Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0154294 and block height 1493920: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:00.536Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceb2ppszcclietw4xhacjd2lmg7nl6cxqy6fkd6fida7nn2fytxdbi
2022-01-25T15:20:00.536Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:00.536Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f065113 and block height 1493920: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:00.536Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedhsuhwre2ccxa6rqlovyxtf6oujudcyti6tyl3oxiwv4pdahzt5k
2022-01-25T15:20:00.536Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:23.047Z	WARN	connmgr	go-libp2p-connmgr@v0.3.0/connmgr.go:154	Low on memory. Closing 53 connections.
2022-01-25T15:20:30.332Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0463143 and block height 1493921: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:30.333Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebpb7hlv3hhy77eesczdkjgqfogzjzqp7szus7q7dsxo7avyku55s
2022-01-25T15:20:30.333Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:30.376Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0469168 and block height 1493921: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:30.376Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacecu22aanajezygjg32ar77r5uibseofacetu2vzigorjjlds6wcdo
2022-01-25T15:20:30.376Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:30.461Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01209185 and block height 1493921: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:30.461Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceb7f3t63j73r3jribnnjrmfpsvcezfbjvcui4h7fovtxvuxcxfghe
2022-01-25T15:20:30.461Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:30.561Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0442378 and block height 1493921: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:30.561Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacea46ezggu3b7vskzdhlxde6aflgozidjscsjw24d7yzacpcnwbjso
2022-01-25T15:20:30.561Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:30.561Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0153930 and block height 1493921: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:30.561Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceanogcaapanjnyvsz5qt5ukgnhj6pxzsb32u7ngwhdmeh5jammka4
2022-01-25T15:20:30.561Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:30.665Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0133509 and block height 1493921: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:30.666Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebbawxbifsko27dgi65vtpyydyca3kvuontniea2xdsxdvx67exbs
2022-01-25T15:20:30.666Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:20:57.192Z	WARN	connmgr	go-libp2p-connmgr@v0.3.0/connmgr.go:151	Low on memory, but we only have a few connections	{"num": 120, "low watermark": 150}
2022-01-25T15:20:59.689Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0161542 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:20:59.689Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedw5mgngdgc5ucipfv75xzkzeepxibqaos5cyptagu62xaaxljs4g
2022-01-25T15:20:59.689Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.303Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0377277 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.303Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedqcp3cn6v52kv32errbuzhda7d4mlforubvmze4ap4x6hg2bn3wc
2022-01-25T15:21:00.303Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.440Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0144618 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.440Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacec3z3nzq76tvac4d6isc7ew4xiuxxmvidnob7x2coobngfcudnor2
2022-01-25T15:21:00.440Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.441Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0724216 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.441Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacec3qrduwsipebezjo3fgkrbttwqxsucwx5fgjarczbvcaqjnbrs3o
2022-01-25T15:21:00.441Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.540Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0680529 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.540Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacea2nrq6kyt63hgtdy26bcd2l2d3jtecoja7wfzdfm72zw765kahru
2022-01-25T15:21:00.540Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.540Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01188117 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.541Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedlh3uy2t7uufgpdvmfpiqucbi6346yhcjoj7m3lilc4okp5gw2bo
2022-01-25T15:21:00.541Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.541Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0127721 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.541Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedposmq2dgehxswyeleclbqfelufqunhyk4zkkh6s43rqne26wkaw
2022-01-25T15:21:00.541Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.541Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0410701 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.541Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceb7ctvykar2bvw3oz34pptpavmul7dfvwkpqspsjhstc55cl6rvn2
2022-01-25T15:21:00.541Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.568Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0867300 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.569Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceajkwzqh2ndjy47pegfwqntrmz6iejakisoju46aknmw7ksfsqqn2
2022-01-25T15:21:00.569Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:00.640Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01208779 and block height 1493922: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:00.640Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacecvbvvzru7welqscg3q2hkmf4wgvdnvtmuorkesmhwmf4miqc26ac
2022-01-25T15:21:00.640Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:30.333Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01079999 and block height 1493923: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:30.333Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceaqg6356wrgywr4x7iph4cgghirff3rbfmvz4bimop3oqakdxfueo
2022-01-25T15:21:30.333Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:30.468Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01108594 and block height 1493923: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:30.468Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceaexysqjxj2b6xplywmtjtiif4kjr2kviqphk6col7nmmltrtzwj4
2022-01-25T15:21:30.468Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:30.499Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f02405 and block height 1493923: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:30.499Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceduf5rsfzw4qflq4jqp4lpghqiolgrojjy2tfh26seskvwx46c6me
2022-01-25T15:21:30.499Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:30.516Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f07830 and block height 1493923: (get sset) failed to load miner actor: actor not found
2022-01-25T15:21:30.516Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacea4ptsgkgejcxreeld6amohttxjrfx6vdcr3kjtbmez7sje7zthqa
2022-01-25T15:21:30.516Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-01-25T15:21:41.356Z	WARN	connmgr	go-libp2p-connmgr@v0.3.0/connmgr.go:151	Low on memory, but we only have a few connections	{"num": 102, "low watermark": 150}



### Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
...

Answered by stuberman

May 6, 2023

Are you running lotus under systemd? Have you tried running by CLI?

View full answer

Reiers · 2022-01-25T18:33:17Z

Reiers
Jan 25, 2022

Hi @versionCollector

Thanks for the report. The issue you are experiencing lies elsewhere.

Moving this issue to Lotus Discussions for help and troubleshooting.

Closing Lotus issue ticket.

0 replies

SgtCoin · 2022-01-26T14:04:17Z

SgtCoin
Jan 26, 2022

Hello @versionCollector

What is the output of lotus sync wait?

1 reply

versionCollector Jan 26, 2022
Author

Hello! The output is

Worker: 0; Base: 0; Target: 1495732 (diff: 1495732)
State: message sync; Current Epoch: 163057; Todo: 1332675
Validated 24392 messages (90 per second)

If i send a api request for Filecoin.ChainHead i get:

{"jsonrpc":"2.0","result":{"Cids":[{"/":"bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2"}],"Blocks":[{"Miner":"f00","Ticket":{"VRFProof":"X4oDOWswmmD7fT0z3RNIPQVGS85f2dBhceeowoDiQhY="},"ElectionProof":{"WinCount":0,"VRFProof":null},"BeaconEntries":[{"Round":0,"Data":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="}],"WinPoStProof":null,"Parents":[{"/":"bafyreiaqpwbbyjo4a42saasj36kkrpv4tsherf2e7bvezkert2a7dhonoi"}],"ParentWeight":"0","Height":0,"ParentStateRoot":{"/":"bafy2bzacech3yb7xlb7c57v2xh7rvmt4skeidk7z2g36llksaz4biflblbt24"},"ParentMessageReceipts":{"/":"bafy2bzacedswlcz5ddgqnyo3sak3jmhmkxashisnlpq6ujgyhe4mlobzpnhs6"},"Messages":{"/":"bafy2bzacecmda75ovposbdateg7eyhwij65zklgyijgcjwynlklmqazpwlhba"},"BLSAggregate":null,"Timestamp":1598306400,"BlockSig":null,"ForkSignaling":0,"ParentBaseFee":"100000000"}],"Height":0},"id":1}

SgtCoin · 2022-01-26T16:29:05Z

SgtCoin
Jan 26, 2022

@versionCollector

It seems to be working, just spamming about nodes it cannot reach. What is the output of lotus sync status and lotus-miner net peers?

0 replies

versionCollector · 2022-01-26T16:37:56Z

versionCollector
Jan 26, 2022
Author

lotus sync status

sync status:
worker 0:
        Base:   [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2]
        Target: [bafy2bzacebs3wa3gy4fmy3fgxm7w4u3kiukt2kjxbq5fbhik3tdibcvzbffri bafy2bzacecefqkdv5if434zibhjdizlmmnacmpbmfuidahavc7kgvcsj4csho bafy2bzacea3fvnezygtxwuf7cgwp3qciocqiapfw76lb3qwy4ouh2r7hopgka] (1495732)
        Height diff:    1495732
        Stage: message sync
        Height: 164236
        Elapsed: 10h9m11.688011406s

lotus-miner net peers gives an error

ERROR: could not get API info for Markets: repo directory does not exist. Make sure your configuration is correct

1 reply

SgtCoin Jan 26, 2022

~~Do you have your LOTUS_MINER_PATH set?~~

I think we just have to wait.

versionCollector · 2022-01-31T15:40:38Z

versionCollector
Jan 31, 2022
Author

@SgtCoin
After a few days, node was OOM killed and restarted. The responses was like

Worker: 0; Base: 0; Target: 1495732 (diff: 1495732)
State: message sync; Current Epoch: 0 Todo: 1495732
Validated 0 messages

I have droped node db, set up LOTUS_MINER_PATH and now I got this outputs for cli commands
lotus sync wait

Worker: 0; Base: 0; Target: 1511200 (diff: 1511200)
State: header sync; Current Epoch: 1429085; Todo: 82115
Validated 0 messages (0 per second)

lotus sync status

sync status:
worker 0:
        Base:   [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2]
        Target: [bafy2bzacebaq5abc25unxomedxh2xj24nbcu5acmneyn5ozyjt4hx7m3j4g7s bafy2bzaced7b3zare7b2vykq67dmjumf3ckyu7z7mtthiyc2zcggb6tgtjdbe bafy2bzacec6dlwh4jvu5j7d5gtc74lrwpyubol3umzsaflzfu6ptj42sb32iq] (1511200)
        Height diff:    1511200
        Stage: header sync
        Height: 1425057
        Elapsed: 8m30.676411066s

The Filecoin.ChainHead request returns this

{"jsonrpc":"2.0","result":{"Cids":[{"/":"bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2"}],"Blocks":[{"Miner":"f00","Ticket":{"VRFProof":"X4oDOWswmmD7fT0z3RNIPQVGS85f2dBhceeowoDiQhY="},"ElectionProof":{"WinCount":0,"VRFProof":null},"BeaconEntries":[{"Round":0,"Data":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="}],"WinPoStProof":null,"Parents":[{"/":"bafyreiaqpwbbyjo4a42saasj36kkrpv4tsherf2e7bvezkert2a7dhonoi"}],"ParentWeight":"0","Height":0,"ParentStateRoot":{"/":"bafy2bzacech3yb7xlb7c57v2xh7rvmt4skeidk7z2g36llksaz4biflblbt24"},"ParentMessageReceipts":{"/":"bafy2bzacedswlcz5ddgqnyo3sak3jmhmkxashisnlpq6ujgyhe4mlobzpnhs6"},"Messages":{"/":"bafy2bzacecmda75ovposbdateg7eyhwij65zklgyijgcjwynlklmqazpwlhba"},"BLSAggregate":null,"Timestamp":1598306400,"BlockSig":null,"ForkSignaling":0,"ParentBaseFee":"100000000"}],"Height":0},"id":1}

So my question is node syncing properly rn? And what are node requirements for api node? https://lotus.filecoin.io/docs/set-up/install/ says node only needs 32Gb ram, but it's frequently restarts with given 64Gb of RAM

0 replies

versionCollector · 2022-02-01T10:21:22Z

versionCollector
Feb 1, 2022
Author

@SgtCoin
Small update. Now looks like node is almost synced, but its allways lags behind ~500 blocks. Right now lotus sync wait prints this

lotus sync wait
Worker: 6; Base: 0; Target: 1513424 (diff: 1513424)
State: persisting headers; Current Epoch: 1512922; Todo: 502
Validated 0 messages (0 per second)

The state changes from header sync to persisting headers and then to error. After current epoch starts from 0 to ~5k and then worker nuber increases by 1 and curent epoch value updates to Target value. e.g.

Worker: 7; Base: 0; Target: 1513429 (diff: 1513429)
State: persisting headers; Current Epoch: 1513429; Todo: 0
Validated 0 messages (0 per second)
...
Worker: 7; Base: 0; Target: 1513429 (diff: 1513429)
State: fetching messages; Current Epoch: 5770; Todo: 1507659
Validated 0 messages (0 per second)
...
Worker: 8; Base: 0; Target: 1513464 (diff: 1513464)
State: persisting headers; Current Epoch: 1512962; Todo: 502
Validated 0 messages (0 per second)

This process repeats few more times until the node process freezes.

lotus sync status output:

worker 6:
        Base:   [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2]
        Target: [bafy2bzacecgcql47ebws45jpmkc2ulfg6f77zcnbd6gl325jw3mod2lyw3way bafy2bzaced656x4cxvypu4kwalimnyohwotwjd24fxzh5j5rowoyc2e5eqkhw bafy2bzacedqkhtyterw75ridqptg3ursacl5dykjix254ysbt47c5r23ckhgk bafy2bzacecuudynm56zzydeqml63ty2uw5ywunqnmg7gk2u5ml4awqr4hsegw bafy2bzacedlzgae7n52vr5rwd52h7y6wju7v3ru5mmvpx3pi2iovs47ohkmzg bafy2bzacecnpqfilmer6eam7tui7qq5udl5omzd6k4podt7klywql2hda3bo2 bafy2bzaced2zzfpx5vdb5ki6ubfioxqhdlh57m6dnhxs7jszxu4pwbgodqlri bafy2bzacedmrymrnvkenqatphaczfhuasg2czvkxoutkhskkfmkkc2l7rb6ls bafy2bzacebfpmxrjagr5apj72mmzcb3sse7mpxyn52pzifrfr43r4t2chj4z6 bafy2bzaceazzzgu64ghodvrzkhbnfrb432y6c6cc7ie5c5yksminnv4ahyu5c] (1513424)
        Height diff:    1513424
        Stage: error
        Height: 5770
        Elapsed: 8m42.626767961s
        Error: collectChain syncMessages: failed to fetch messages: 80 errors occurred:
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers


worker 7:
        Base:   [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2]
        Target: [bafy2bzaced7zrapmhutflcxh37x72cf4uk7kjoyigpqef5zbwk7t4jp6swdms bafy2bzacead3yehsfvvjbywnjrjh23y5k5qvzptrfvb5iy57eaqtitzr5hgk6 bafy2bzacedzdaguqco6rfreph5p4aou4b5daw5umaqsn3bxryqx6e2kc7wto2 bafy2bzacea5teil6qyijnvgjakunt7nirir6buld47xpooz62olvddf76kzmk bafy2bzaceaalcf4jqj6drgvtempck4sw36nopqygleiwpbdiuilyrrojps5oo] (1513429)
        Height diff:    1513429
        Stage: error
        Height: 5770
        Elapsed: 8m41.859455477s
        Error: collectChain syncMessages: failed to fetch messages: 80 errors occurred:
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers
        * doRequest failed for all peers


worker 8:
        Base:   [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2]
        Target: [bafy2bzaceakavaxo6h3su3blmj46d44ko3ewrs4ycnr53rhf277ivuyvx7ohs bafy2bzacecafkcf242hcjaexv7hkummr2yhbnti5cakpy2kem5h5b7bzlgzok] (1513464)
        Height diff:    1513464
        Stage: persisting headers
        Height: 1512962
        Elapsed: 6m46.307894373s

0 replies

versionCollector · 2022-02-02T11:47:42Z

versionCollector
Feb 2, 2022
Author

A few hours later after I gave an update, the node stopped again.
Logs:

2022-02-01T13:58:00.375Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceb52d6jndv5vxwsuhc2n3tpwtwtx3htmx4ebxmdyjwprupgltvrqw
2022-02-01T13:58:00.375Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:58:00.390Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01028902 and block height 1513916: (get sset) failed to load miner actor: actor not found
2022-02-01T13:58:00.390Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedvw2odhtkar5jj72lgfaqmimutgtzuzuotp6ugjld44zamadbeiy
2022-02-01T13:58:00.390Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:58:00.434Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0146393 and block height 1513916: (get sset) failed to load miner actor: actor not found
2022-02-01T13:58:00.434Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceab2qhd4tvnhiac7nmq6nnz5qearmjgqvn2tp63fcymja53i34epi
2022-02-01T13:58:00.434Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:58:16.106Z	INFO	badgerbs	v2@v2.2007.3/db.go:1031	Storing value log head: {Fid:515 Len:35 Offset:578076189}

2022-02-01T13:58:30.281Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f083625 and block height 1513917: (get sset) failed to load miner actor: actor not found
2022-02-01T13:58:30.281Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebkvuixp7dxuiarww3vrw5amrrsco7eelhbv7x6vhzeoi4eu7dllq
2022-02-01T13:58:30.281Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:58:30.451Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0118167 and block height 1513917: (get sset) failed to load miner actor: actor not found
2022-02-01T13:58:30.451Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedarl3elrktm57pzjrjrzglroxaej3osbdbtp36guqdx4dl2p7xpy
2022-02-01T13:58:30.451Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:58:34.818Z	INFO	badgerbs	v2@v2.2007.3/db.go:1031	Storing value log head: {Fid:516 Len:35 Offset:161697401}

2022-02-01T13:58:35.126Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 0] Running compaction: {level:0 score:1 dropPrefixes:[]} for level: 0

2022-02-01T13:58:37.577Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 0->1, del 9 tables, add 7 tables, took 2.450336179s

2022-02-01T13:58:37.577Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 0] Compaction for level: 0 DONE
2022-02-01T13:58:37.577Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 0] Running compaction: {level:1 score:1.7704651542007923 dropPrefixes:[]} for level: 1

2022-02-01T13:58:39.864Z	WARN	watchdog	go-watchdog@v1.2.0/watchdog.go:349	system-driven watchdog triggering GC; 50264297472/47152875929 bytes (used/threshold)
2022-02-01T13:58:39.864Z	INFO	watchdog	go-watchdog@v1.2.0/watchdog.go:368	watchdog is forcing GC
2022-02-01T13:58:40.365Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.0078190345317126 dropPrefixes:[]} for level: 2

2022-02-01T13:58:40.397Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 1->2, del 9 tables, add 9 tables, took 2.820002232s

2022-02-01T13:58:40.397Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 0] Compaction for level: 1 DONE
2022-02-01T13:58:40.397Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 0] Running compaction: {level:1 score:1.5102410838007927 dropPrefixes:[]} for level: 1

2022-02-01T13:58:41.113Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 1 tables, add 1 tables, took 748.244741ms

2022-02-01T13:58:41.114Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T13:58:43.268Z	INFO	watchdog	go-watchdog@v1.2.0/watchdog.go:382	watchdog-triggered GC finished; notifications took: 480ns, took: 3.404087656s; current heap allocated: 27999452632 bytes
2022-02-01T13:58:43.268Z	WARN	connmgr	go-libp2p-connmgr@v0.3.0/connmgr.go:151	Low on memory, but we only have a few connections	{"num": 39, "low watermark": 150}
2022-02-01T13:58:44.565Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.0066439263522624 dropPrefixes:[]} for level: 2

2022-02-01T13:58:44.647Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 1->2, del 12 tables, add 12 tables, took 4.249544296s

2022-02-01T13:58:44.647Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 0] Compaction for level: 1 DONE
2022-02-01T13:58:44.647Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 0] Running compaction: {level:1 score:1.2500138022005558 dropPrefixes:[]} for level: 1

2022-02-01T13:58:44.686Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 2 tables, add 1 tables, took 120.356018ms

2022-02-01T13:58:44.686Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T13:58:44.686Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.006631089374423 dropPrefixes:[]} for level: 2

2022-02-01T13:58:45.086Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 1 tables, add 1 tables, took 399.837566ms

2022-02-01T13:58:45.086Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T13:58:45.465Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.0061194807291032 dropPrefixes:[]} for level: 2

2022-02-01T13:58:45.485Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 1->2, del 3 tables, add 3 tables, took 838.083024ms

2022-02-01T13:58:45.485Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 0] Compaction for level: 1 DONE
2022-02-01T13:58:45.648Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 2 tables, add 1 tables, took 183.035852ms

2022-02-01T13:58:45.648Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T13:58:45.648Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.0001366034150123 dropPrefixes:[]} for level: 2

2022-02-01T13:58:45.770Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 2 tables, add 1 tables, took 121.518911ms

2022-02-01T13:58:45.770Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T13:58:45.770Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.0001279700547456 dropPrefixes:[]} for level: 2

2022-02-01T13:58:45.892Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 2 tables, add 1 tables, took 121.987571ms

2022-02-01T13:58:45.892Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T13:58:45.892Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.0000633221119641 dropPrefixes:[]} for level: 2

2022-02-01T13:58:46.014Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 2 tables, add 1 tables, took 121.971372ms

2022-02-01T13:58:46.014Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T13:58:46.014Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.0000537786632777 dropPrefixes:[]} for level: 2

2022-02-01T13:58:46.137Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 2 tables, add 1 tables, took 122.858793ms

2022-02-01T13:58:46.137Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T13:58:53.746Z	INFO	badgerbs	v2@v2.2007.3/db.go:1031	Storing value log head: {Fid:516 Len:35 Offset:817903929}

2022-02-01T13:59:00.310Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0473677 and block height 1513918: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:00.310Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebu3bwuqygmdsearm6vckrlnpbyq6bpwsojh4fih6mvdozo4qocda
2022-02-01T13:59:00.310Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:00.363Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f010616 and block height 1513918: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:00.363Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceaeaxeeijz2zgpzh7eksl34ny4p7qwuid24jmh3gbiochmxvvrkoc
2022-02-01T13:59:00.363Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:00.371Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01272 and block height 1513918: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:00.371Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebscrmxbhscwolr2stbbklt36spgnk6i23z7lbxlcduxmi5aars2a
2022-02-01T13:59:00.371Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:00.432Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01472185 and block height 1513918: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:00.432Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceb3mq2ba2izvprp5bvtb5tjkqznkegjugoapupszbs4adpu6gd7fg
2022-02-01T13:59:00.432Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:00.643Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0135467 and block height 1513918: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:00.643Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceanvdoautgssuyxhjbv7smrzet6lnw4pkx7zop7egmybrswjymefg
2022-02-01T13:59:00.643Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:00.760Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0123261 and block height 1513918: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:00.760Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacea74zsay2ror3vj2l7jifp4nipy6odmrbft2vu7sb2bbcnmsznil2
2022-02-01T13:59:00.760Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:10.410Z	INFO	badgerbs	v2@v2.2007.3/db.go:1031	Storing value log head: {Fid:517 Len:35 Offset:399958514}

2022-02-01T13:59:27.453Z	INFO	badgerbs	v2@v2.2007.3/db.go:1031	Storing value log head: {Fid:517 Len:35 Offset:1054739600}

2022-02-01T13:59:30.323Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0707777 and block height 1513919: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:30.323Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaced3gm62x3xzda7qosva5f63u3xad2vrov7r3wqxz3u2s3t7n3yqjy
2022-02-01T13:59:30.323Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:30.362Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01674133 and block height 1513919: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:30.362Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceau32tjuxuluhegp77ru2aibtf67ruptzrr2p2442l2lmdxxn2evg
2022-02-01T13:59:30.362Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:30.378Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01260725 and block height 1513919: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:30.378Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceailetiwcjeoeaxpqriyxn2f4pdd6zxv54mhuqnokxh7bjz6acejs
2022-02-01T13:59:30.378Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:30.409Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f089200 and block height 1513919: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:30.409Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacea3wwlabnc2sudxqkwq3ysw3sqlzxdjvzjsjcd4leoa7tt5njwzcg
2022-02-01T13:59:30.409Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:30.431Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f022373 and block height 1513919: (get sset) failed to load miner actor: actor not found
2022-02-01T13:59:30.431Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebwhe54g23ejlyvscqnw2gbnfg6qwxwvmtjceeuguxhudhvdzpt7s
2022-02-01T13:59:30.431Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T13:59:44.898Z	INFO	badgerbs	v2@v2.2007.3/db.go:1031	Storing value log head: {Fid:518 Len:35 Offset:635901689}

2022-02-01T14:00:00.257Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0240035 and block height 1513920: (get sset) failed to load miner actor: actor not found
2022-02-01T14:00:00.257Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceakbrx5ob7b25zuiqd2nug3hk55763dhnpjp2uft6x2kz3muyzy6a
2022-02-01T14:00:00.257Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T14:00:00.405Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0883083 and block height 1513920: (get sset) failed to load miner actor: actor not found
2022-02-01T14:00:00.405Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebg2izid3l4ge3nff7e5xm4kpxjtwzu3x3gc5s64w2zsdkdx76eso
2022-02-01T14:00:00.405Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T14:00:00.445Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0441372 and block height 1513920: (get sset) failed to load miner actor: actor not found
2022-02-01T14:00:00.445Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacecugt3kw3nbl7fe6l2arvit6smp3cmihsccmp237tk2m75auyttvi
2022-02-01T14:00:00.445Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T14:00:00.477Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0858986 and block height 1513920: (get sset) failed to load miner actor: actor not found
2022-02-01T14:00:00.477Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceaurlgi43ptjrajrl5jzfwzxoqnfni44zok5o6jbo46a27cqbf2ee
2022-02-01T14:00:00.477Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T14:00:00.500Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0884781 and block height 1513920: (get sset) failed to load miner actor: actor not found
2022-02-01T14:00:00.501Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceaojh3re4hnqsgd4udg6f6btrpw7cfeca7pf6um7zmlhf3d6wb6qu
2022-02-01T14:00:00.501Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T14:00:00.791Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f024483 and block height 1513920: (get sset) failed to load miner actor: actor not found
2022-02-01T14:00:00.791Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedg7vud7pmye7qgevstsj3nogofkyt7vkmwkxtzwb2gzc375bxm64
2022-02-01T14:00:00.791Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-01T14:00:02.228Z	INFO	badgerbs	v2@v2.2007.3/db.go:1031	Storing value log head: {Fid:519 Len:35 Offset:216084520}

2022-02-01T14:00:02.627Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 0] Running compaction: {level:0 score:1 dropPrefixes:[]} for level: 0

2022-02-01T14:00:05.384Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 0->1, del 9 tables, add 8 tables, took 2.75724124s

2022-02-01T14:00:05.384Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 0] Compaction for level: 0 DONE
2022-02-01T14:00:05.384Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 0] Running compaction: {level:1 score:1.8720015697181225 dropPrefixes:[]} for level: 1

2022-02-01T14:00:07.265Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.024599726498127 dropPrefixes:[]} for level: 2

2022-02-01T14:00:07.274Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 1->2, del 6 tables, add 6 tables, took 1.890085871s

2022-02-01T14:00:07.274Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 0] Compaction for level: 1 DONE
2022-02-01T14:00:07.274Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 0] Running compaction: {level:1 score:1.6117664761841297 dropPrefixes:[]} for level: 1

2022-02-01T14:00:07.451Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 2 tables, add 1 tables, took 185.167752ms

2022-02-01T14:00:07.451Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T14:00:07.451Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.024585671722889 dropPrefixes:[]} for level: 2

2022-02-01T14:00:07.608Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 2 tables, add 1 tables, took 157.04782ms

2022-02-01T14:00:07.608Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T14:00:07.608Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.0245349232107401 dropPrefixes:[]} for level: 2

2022-02-01T14:00:07.990Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 1->2, del 3 tables, add 2 tables, took 715.424563ms

2022-02-01T14:00:07.990Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 0] Compaction for level: 1 DONE
2022-02-01T14:00:07.990Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 0] Running compaction: {level:1 score:1.5613758563995361 dropPrefixes:[]} for level: 1

2022-02-01T14:00:08.040Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 1 tables, add 1 tables, took 431.926657ms

2022-02-01T14:00:08.040Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE
2022-02-01T14:00:08.040Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1000	[Compactor: 1] Running compaction: {level:2 score:1.003161996603012 dropPrefixes:[]} for level: 2

2022-02-01T14:00:08.374Z	INFO	watchdog	go-watchdog@v1.2.0/watchdog.go:503	heap profile captured; path: /lotus-data/heapprof/2022-02-01T14:00:08.280199654Z.heap
2022-02-01T14:00:08.374Z	WARN	watchdog	go-watchdog@v1.2.0/watchdog.go:349	system-driven watchdog triggering GC; 60817858560/57257063628 bytes (used/threshold)
2022-02-01T14:00:08.374Z	INFO	watchdog	go-watchdog@v1.2.0/watchdog.go:368	watchdog is forcing GC
2022-02-01T14:00:08.645Z	INFO	badgerbs	v2@v2.2007.3/levels.go:962	LOG Compact 2->3, del 1 tables, add 1 tables, took 605.294254ms

2022-02-01T14:00:08.645Z	INFO	badgerbs	v2@v2.2007.3/levels.go:1010	[Compactor: 1] Compaction for level: 2 DONE

0 replies

versionCollector · 2022-02-03T13:05:00Z

versionCollector
Feb 3, 2022
Author

@SgtCoin are you still here? Can you help me please?:)

0 replies

SgtCoin · 2022-02-03T14:35:54Z

SgtCoin
Feb 3, 2022

Hi @versionCollector

Are you going through a firewall or anything else that would cause connection issues?

0 replies

versionCollector · 2022-02-03T23:51:21Z

versionCollector
Feb 3, 2022
Author

Hi! Looks like it was the issue. Now node has new logs, still got errors, but now node is validating blocks

2022-02-03T23:41:59.699Z	WARN	vm	vm/vm.go:539	Send actor error	{"from": "f3wcfa5vw7t54rilkeee63x4w6fhytjznk4ywaxjrbr5g5cxogvvrgllwcjtgd4hd5uask7ulzpq4qhk45pu7q", "to": "f02770", "nonce": 224701, "method": "7", "height": "22189", "error": "failed to submit proof for bulk verification (RetCode=7):\n    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess\n        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26"}
2022-02-03T23:42:00.240Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0103710 and block height 1520844: (get sset) failed to load miner actor: actor not found
2022-02-03T23:42:00.240Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceb5inxhaldaa3qaue3unee6qdf3dmfv22m5qr2cxi42ykt6ykiyew
2022-02-03T23:42:00.241Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-03T23:42:00.283Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0700033 and block height 1520844: (get sset) failed to load miner actor: actor not found
2022-02-03T23:42:00.283Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacedd5xrabszfbyhcrwefuie32ezzxnnvxgfigqdhge7gyn3iwi2xn4
2022-02-03T23:42:00.283Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-03T23:42:00.323Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0757610 and block height 1520844: (get sset) failed to load miner actor: actor not found
2022-02-03T23:42:00.324Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebpweh4l7ykqfvioemwoizd3kcecbai5imxcipenipqwtfskkrgki
2022-02-03T23:42:00.324Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-03T23:42:00.373Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f01111131 and block height 1520844: (get sset) failed to load miner actor: actor not found
2022-02-03T23:42:00.373Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacecv3a2zgehnqpm5evtwgjjxxhtzjs7hpjw7bdf7e7knumcj5wmvtc
2022-02-03T23:42:00.373Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-03T23:42:00.519Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f092165 and block height 1520844: (get sset) failed to load miner actor: actor not found
2022-02-03T23:42:00.519Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzacebu6ezm6qycsspd2libymgfezyooqhllwojainrvi4qg5tvvofc7a
2022-02-03T23:42:00.519Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-03T23:42:00.781Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f070932 and block height 1520844: (get sset) failed to load miner actor: actor not found
2022-02-03T23:42:00.781Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceb4uk6adt24n6x2wbhqq2tq2p7yhknemhjczbma36nbkqojieqehq
2022-02-03T23:42:00.781Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-03T23:42:01.015Z	WARN	fil-consensus	filcns/filecoin.go:835	failed to resolve worker key for miner f0123346 and block height 1520844: (get sset) failed to load miner actor: actor not found
2022-02-03T23:42:01.015Z	WARN	fil-consensus	filcns/filecoin.go:805	cannot validate block message; unknown miner or miner that doesn't meet min power in unsynced chain: bafy2bzaceah6bfjfsjjul5gsnvfn7nkbddvx2wetidh6chb72dney2cvahcvc
2022-02-03T23:42:01.015Z	WARN	fil-consensus	filcns/filecoin.go:692	ignoring block msg: soft validation failure
2022-02-03T23:42:01.340Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.880505591, "height": "22190", "age": 44959621.34050086}
2022-02-03T23:42:01.363Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.903415007, "height": "22190", "age": 44959621.363408424}
2022-02-03T23:42:02.241Z	INFO	chain	chain/sync.go:624	block validation	{"took": 0.868366039, "height": "22191", "age": 44959592.24160316}
2022-02-03T23:42:02.272Z	INFO	chain	chain/sync.go:624	block validation	{"took": 0.899108305, "height": "22191", "age": 44959592.27233378}
2022-02-03T23:42:02.517Z	WARN	vm	vm/runtime.go:161	VM.Call failure in call from: f02614 to f04: not enough gas: used=18994138, available=53435711, use=34721049 (RetCode=7):
    github.com/filecoin-project/lotus/chain/vm.(*Runtime).chargeGasInternal
        /opt/filecoin/chain/vm/runtime.go:593
2022-02-03T23:42:02.517Z	WARN	vm	vm/runtime.go:394	vmctx send failed: from: f02614 to: f04, method: 8: err: not enough gas: used=18994138, available=53435711, use=34721049 (RetCode=7)
2022-02-03T23:42:02.517Z	WARN	vm	vm/runtime.go:350	Abortf: failed to submit proof for bulk verification
2022-02-03T23:42:02.517Z	WARN	vm	vm/runtime.go:161	VM.Call failure in call from: f01095 to f02614: failed to submit proof for bulk verification (RetCode=7):
    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess
        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26
2022-02-03T23:42:02.517Z	WARN	vm	vm/vm.go:539	Send actor error	{"from": "f3ro3i54tule2vtdcsjdkzjkf6djx3wwe5znv5h4kwxprxjyojvgyqtyvxus5bj73dy64x4paiqriwupmei3va", "to": "f02614", "nonce": 237507, "method": "7", "height": "22191", "error": "failed to submit proof for bulk verification (RetCode=7):\n    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess\n        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26"}
2022-02-03T23:42:03.354Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.067915019, "height": "22192", "age": 44959563.354589395}
2022-02-03T23:42:03.380Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.093765562, "height": "22192", "age": 44959563.3803945}
2022-02-03T23:42:03.905Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537186383", "err": "invalid proof"}
2022-02-03T23:42:03.958Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537175121", "err": "invalid proof"}
2022-02-03T23:42:04.053Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "430268", "err": "invalid proof"}
2022-02-03T23:42:04.693Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.300395528, "height": "22193", "age": 44959534.693060085}
2022-02-03T23:42:04.713Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.320518957, "height": "22193", "age": 44959534.71322776}
2022-02-03T23:42:04.732Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.340111766, "height": "22193", "age": 44959534.73282321}
2022-02-03T23:42:04.752Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.359663523, "height": "22193", "age": 44959534.752328016}
2022-02-03T23:42:04.771Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.378929411, "height": "22193", "age": 44959534.771640845}
2022-02-03T23:42:05.865Z	INFO	pubsub	go-libp2p-pubsub@v0.6.0/gossipsub.go:1593	peer QmZNjKc5GSNhHu1Uj1AcSp6P5a7XnDov2HoZbbNkT92YnS didn't follow up in 1 IWANT requests; adding penalty
2022-02-03T23:42:06.865Z	INFO	pubsub	go-libp2p-pubsub@v0.6.0/gossipsub.go:1593	peer QmZNjKc5GSNhHu1Uj1AcSp6P5a7XnDov2HoZbbNkT92YnS didn't follow up in 1 IWANT requests; adding penalty
2022-02-03T23:42:07.068Z	INFO	chain	chain/sync.go:624	block validation	{"took": 2.278438287, "height": "22194", "age": 44959507.06813894}
2022-02-03T23:42:07.080Z	INFO	chain	chain/sync.go:624	block validation	{"took": 2.291152338, "height": "22194", "age": 44959507.080853045}
2022-02-03T23:42:07.100Z	INFO	chain	chain/sync.go:624	block validation	{"took": 2.310433576, "height": "22194", "age": 44959507.10012791}
2022-02-03T23:42:07.112Z	INFO	chain	chain/sync.go:624	block validation	{"took": 2.322631715, "height": "22194", "age": 44959507.1122993}
2022-02-03T23:42:07.769Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f01782", "sectorNumber": "95051", "err": "invalid proof"}
2022-02-03T23:42:07.865Z	INFO	pubsub	go-libp2p-pubsub@v0.6.0/gossipsub.go:1593	peer QmZNjKc5GSNhHu1Uj1AcSp6P5a7XnDov2HoZbbNkT92YnS didn't follow up in 1 IWANT requests; adding penalty
2022-02-03T23:42:09.015Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.884918082, "height": "22195", "age": 44959479.015164405}
2022-02-03T23:42:09.042Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.911857788, "height": "22195", "age": 44959479.0420823}
2022-02-03T23:42:09.054Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.924184748, "height": "22195", "age": 44959479.05441002}
2022-02-03T23:42:09.066Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.936728798, "height": "22195", "age": 44959479.06693259}
2022-02-03T23:42:09.086Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.956152226, "height": "22195", "age": 44959479.086348094}
2022-02-03T23:42:09.098Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.968640497, "height": "22195", "age": 44959479.098845944}
2022-02-03T23:42:09.566Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537220174", "err": "invalid proof"}
2022-02-03T23:42:09.609Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537097404", "err": "invalid proof"}
2022-02-03T23:42:09.854Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "502973", "err": "invalid proof"}
2022-02-03T23:42:09.871Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "468156", "err": "invalid proof"}
2022-02-03T23:42:09.885Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537361530", "err": "invalid proof"}
2022-02-03T23:42:09.909Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "285887", "err": "invalid proof"}
2022-02-03T23:42:10.029Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537038011", "err": "invalid proof"}
2022-02-03T23:42:10.052Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537152591", "err": "invalid proof"}
2022-02-03T23:42:10.063Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "445551", "err": "invalid proof"}
2022-02-03T23:42:10.066Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537205833", "err": "invalid proof"}
2022-02-03T23:42:10.933Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.820720054, "height": "22196", "age": 44959450.933719315}
2022-02-03T23:42:11.798Z	INFO	chain	chain/sync.go:624	block validation	{"took": 0.854381296, "height": "22197", "age": 44959421.79814052}
2022-02-03T23:42:11.810Z	INFO	chain	chain/sync.go:624	block validation	{"took": 0.866835836, "height": "22197", "age": 44959421.810550176}
2022-02-03T23:42:11.833Z	INFO	chain	chain/sync.go:624	block validation	{"took": 0.889809863, "height": "22197", "age": 44959421.83352735}
2022-02-03T23:42:12.936Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.092097737, "height": "22198", "age": 44959392.936372384}
2022-02-03T23:42:12.969Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.12563483, "height": "22198", "age": 44959392.96994057}
2022-02-03T23:42:12.982Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.138024201, "height": "22198", "age": 44959392.98232016}
2022-02-03T23:42:13.734Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537131088", "err": "invalid proof"}
2022-02-03T23:42:13.798Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "536986732", "err": "invalid proof"}
2022-02-03T23:42:13.899Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "537129040", "err": "invalid proof"}
2022-02-03T23:42:13.904Z	WARN	vm	vm/syscalls.go:342	seal verify in batch failed	{"miner": "f03350", "sectorNumber": "259262", "err": "invalid proof"}
2022-02-03T23:42:14.664Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.669827845, "height": "22199", "age": 44959364.66438404}
2022-02-03T23:42:14.676Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.682073875, "height": "22199", "age": 44959364.676602885}
2022-02-03T23:42:14.688Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.694198315, "height": "22199", "age": 44959364.688696966}
2022-02-03T23:42:14.719Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.72462543, "height": "22199", "age": 44959364.71917843}
2022-02-03T23:42:14.731Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.73716841, "height": "22199", "age": 44959364.73169632}
2022-02-03T23:42:16.192Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.449456264, "height": "22200", "age": 44959336.19218279}
2022-02-03T23:42:16.204Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.461453494, "height": "22200", "age": 44959336.20417837}
2022-02-03T23:42:16.210Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.467564889, "height": "22200", "age": 44959336.21029029}
2022-02-03T23:42:16.231Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.4883764, "height": "22200", "age": 44959336.23108112}
2022-02-03T23:42:16.340Z	WARN	vm	vm/runtime.go:161	VM.Call failure in call from: f01012 to f04: not enough gas: used=50757187, available=50780408, use=363070 (RetCode=7):
    github.com/filecoin-project/lotus/chain/vm.(*Runtime).chargeGasInternal
        /opt/filecoin/chain/vm/runtime.go:593
2022-02-03T23:42:16.340Z	WARN	vm	vm/runtime.go:394	vmctx send failed: from: f01012 to: f04, method: 8: err: not enough gas: used=50757187, available=50780408, use=363070 (RetCode=7)
2022-02-03T23:42:16.340Z	WARN	vm	vm/runtime.go:350	Abortf: failed to submit proof for bulk verification
2022-02-03T23:42:16.340Z	WARN	vm	vm/runtime.go:161	VM.Call failure in call from: f01010 to f01012: failed to submit proof for bulk verification (RetCode=7):
    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess
        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26
2022-02-03T23:42:16.340Z	WARN	vm	vm/vm.go:539	Send actor error	{"from": "f3q2kzhc2zzwdclfj6imgiffn2xvtyfu42tz7h2gfstj7j7xjlxgefnh35dxnc3frwvpr35hgayldy6kssrt2a", "to": "f01012", "nonce": 76398, "method": "7", "height": "22200", "error": "failed to submit proof for bulk verification (RetCode=7):\n    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess\n        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26"}
2022-02-03T23:42:16.348Z	WARN	vm	vm/runtime.go:161	VM.Call failure in call from: f02775 to f04: not enough gas: used=18232138, available=50837908, use=34721049 (RetCode=7):
    github.com/filecoin-project/lotus/chain/vm.(*Runtime).chargeGasInternal
        /opt/filecoin/chain/vm/runtime.go:593
2022-02-03T23:42:16.348Z	WARN	vm	vm/runtime.go:394	vmctx send failed: from: f02775 to: f04, method: 8: err: not enough gas: used=18232138, available=50837908, use=34721049 (RetCode=7)
2022-02-03T23:42:16.348Z	WARN	vm	vm/runtime.go:350	Abortf: failed to submit proof for bulk verification
2022-02-03T23:42:16.348Z	WARN	vm	vm/runtime.go:161	VM.Call failure in call from: f02643 to f02775: failed to submit proof for bulk verification (RetCode=7):
    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess
        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26
2022-02-03T23:42:16.348Z	WARN	vm	vm/vm.go:539	Send actor error	{"from": "f3w6qx35gmtrvi2rmmhqkudocewarjbt2msoxlzigej3cwzd24skvhud57m6725iq6dl45fk56pplurvojakaa", "to": "f02775", "nonce": 134792, "method": "7", "height": "22200", "error": "failed to submit proof for bulk verification (RetCode=7):\n    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess\n        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26"}
2022-02-03T23:42:16.358Z	WARN	vm	vm/runtime.go:161	VM.Call failure in call from: f02775 to f04: not enough gas: used=16390450, available=50892908, use=34721049 (RetCode=7):
    github.com/filecoin-project/lotus/chain/vm.(*Runtime).chargeGasInternal
        /opt/filecoin/chain/vm/runtime.go:593
2022-02-03T23:42:16.358Z	WARN	vm	vm/runtime.go:394	vmctx send failed: from: f02775 to: f04, method: 8: err: not enough gas: used=16390450, available=50892908, use=34721049 (RetCode=7)
2022-02-03T23:42:16.358Z	WARN	vm	vm/runtime.go:350	Abortf: failed to submit proof for bulk verification
2022-02-03T23:42:16.358Z	WARN	vm	vm/runtime.go:161	VM.Call failure in call from: f02643 to f02775: failed to submit proof for bulk verification (RetCode=7):
    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess
        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26
2022-02-03T23:42:16.358Z	WARN	vm	vm/vm.go:539	Send actor error	{"from": "f3w6qx35gmtrvi2rmmhqkudocewarjbt2msoxlzigej3cwzd24skvhud57m6725iq6dl45fk56pplurvojakaa", "to": "f02775", "nonce": 134808, "method": "7", "height": "22200", "error": "failed to submit proof for bulk verification (RetCode=7):\n    github.com/filecoin-project/specs-actors/actors/builtin.RequireSuccess\n        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/builtin/shared.go:26"}
2022-02-03T23:42:17.393Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.148820677, "height": "22201", "age": 44959307.39349397}
2022-02-03T23:42:17.406Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.161310238, "height": "22201", "age": 44959307.40601017}
2022-02-03T23:42:17.418Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.173646438, "height": "22201", "age": 44959307.4183223}
2022-02-03T23:42:19.135Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.705503232, "height": "22202", "age": 44959279.13584183}
2022-02-03T23:42:19.162Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.731922157, "height": "22202", "age": 44959279.16226077}
2022-02-03T23:42:20.362Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.187495771, "height": "22203", "age": 44959250.36248548}
2022-02-03T23:42:20.374Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.199886452, "height": "22203", "age": 44959250.374884}
2022-02-03T23:42:20.387Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.212413433, "height": "22203", "age": 44959250.38740459}
2022-02-03T23:42:22.324Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.924994639, "height": "22204", "age": 44959222.32424174}
2022-02-03T23:42:22.350Z	INFO	chain	chain/sync.go:624	block validation	{"took": 1.950932503, "height": "22204", "age": 44959222.35017112}
2022-02-03T23:42:23.317Z	INFO	chain	chain/sync.go:624	block validation	{"took": 0.95473485, "height": "22205", "age": 44959193.3177366}
2022-02-03T23:42:23.330Z	INFO	chain	chain/sync.go:624	block validation	{"took": 0.967117701, "height": "22205", "age": 44959193.33011938}
2022-02-03T23:42:23.342Z	INFO	chain	chain/sync.go:624	block validation	{"took": 0.979511632, "height": "22205", "age": 44959193.34253691}

lotus sync wait
Worker: 0; Base: 0; Target: 1520081 (diff: 1520081)
State: message sync; Current Epoch: 22558; Todo: 1497523
Validated 4730 messages (465 per second)

0 replies

SgtCoin · 2022-02-04T03:26:16Z

SgtCoin
Feb 4, 2022

@versionCollector

Awesome! Lets monitor for a bit and see how things go. Everything else working ok?

0 replies

versionCollector · 2022-02-04T09:21:12Z

versionCollector
Feb 4, 2022
Author

@SgtCoin
Filecoin.ChainHead request still returns first block.

{"jsonrpc":"2.0","result":{"Cids":[{"/":"bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2"}],"Blocks":[{"Miner":"f00","Ticket":{"VRFProof":"X4oDOWswmmD7fT0z3RNIPQVGS85f2dBhceeowoDiQhY="},"ElectionProof":{"WinCount":0,"VRFProof":null},"BeaconEntries":[{"Round":0,"Data":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="}],"WinPoStProof":null,"Parents":[{"/":"bafyreiaqpwbbyjo4a42saasj36kkrpv4tsherf2e7bvezkert2a7dhonoi"}],"ParentWeight":"0","Height":0,"ParentStateRoot":{"/":"bafy2bzacech3yb7xlb7c57v2xh7rvmt4skeidk7z2g36llksaz4biflblbt24"},"ParentMessageReceipts":{"/":"bafy2bzacedswlcz5ddgqnyo3sak3jmhmkxashisnlpq6ujgyhe4mlobzpnhs6"},"Messages":{"/":"bafy2bzacecmda75ovposbdateg7eyhwij65zklgyijgcjwynlklmqazpwlhba"},"BLSAggregate":null,"Timestamp":1598306400,"BlockSig":null,"ForkSignaling":0,"ParentBaseFee":"100000000"}],"Height":0},"id":1}

And validating speed is kinda slow, does importing chain would help with it?

1 reply

SgtCoin Mar 14, 2022

@versionCollector

You should not have to import the the entire chain. You originally imported via snapshot correct?

raojianli · 2022-03-14T02:46:34Z

raojianli
Mar 14, 2022

i have a same error, have any way to solved it?

1 reply

SgtCoin Mar 14, 2022

@raojianli

The original issue was somewhat resolved with firewall updates. Have you already checked your firewalls?

betayes66 · 2022-04-12T09:13:37Z

betayes66
Apr 12, 2022

i have a same error, have any way to solved it? @versionCollector @raojianli @SgtCoin
do you deal it

0 replies

yorickdowne · 2023-03-19T17:43:36Z

yorickdowne
Mar 19, 2023

In my case lotus got OOM-killed on a 64 GiB machine that was also running a few other services to the tune of ~10 GiB, during "persisting headers" stage. I solved this by installing zram-tools, editing /etc/default/zramswap to zstd algorithm and 25 percent, and doing a sudo systemctl restart zramswap. This adds 16 GiB of swap space, which will take maybe 1/2 to 1/3 of that in actual RAM, and allows lotus to complete.

You may also want to set vm.swappiness=100 in /etc/sysctl.conf when using zramswap.

0 replies

NoFeeding · 2023-03-20T06:48:08Z

NoFeeding
Mar 20, 2023

I have same issue with both my nodes.
Firewall already checked no physical one and ufw disabled.
when I cast lotus sync wait it will be like "0 per second" at validated message section like this

lotus sync wait
Worker: 0; Base: 2696563; Target: 2699146 (diff: 2583)
State: message sync; Current Epoch: 2697745; Todo: 1401
Validated 119536 messages (0 per second)

it's making my node out of sync for quite some time and for the sync status it's like this

lotus sync status
sync status:
worker 0:
Base: [bafy2bzaceajyjoathb7foq53xea4lxxjtn5tqzpcbjw7ftnsjmrgeipp7ovdw]
Target: [bafy2bzacedjnmjsnfxpfby66ebbpkcnzq7cdivo65vnh3ksz4txig6lurj7yu bafy2bzacea43cyweymukjbppkqesagk5ge66cilzcyvimzffd7k3midi5mbgy bafy2bzaceby6qcgw3tkglxt4s7evmv3nnyelvlljyclwn6drflupae2eesgd2 bafy2bzacedwg5sb3medbixiykgjhgpbhldio4mkr5dhbrc22dzuiqht6nttxm bafy2bzacebuxzx4jhz4wtqh6swzxskskkozrhwwmybpcv5d6pydrd6ls672oc bafy2bzacecezd2eol3slreylbjb4i26nts6dj2aqsc5staj6szoqyesxlkrrm bafy2bzacebqevghya4yktwygjv4ixr5jmv5b24ianq22kb662wjfq6244djt4 bafy2bzacedbvq7xjr7ntizsacaexfmmrnsq73nwaukirngn5jpa5j6faj3zy4] (2699146)
Height diff: 2583
Stage: message sync
Height: 2697746
Elapsed: 3h49m2.904668731s

Wondering if anything I can edit to make my node catch up newest height as before?

I am currently on lotus version 1.20.3+mainnet+git.b332db0a3 while the other node is facing the same problem on lotus version 1.20.4+mainnet+git.5f91072d4.dirty but same issue. Please help!

2 replies

mithmith May 6, 2023

Has the same issue on lotus version 1.23.0+mainnet+git.d1d4b35ad
After restarting, the lotus daemon is successfully synchronized and runs for 2-3 hours, then freezes and synchronization disappears.

lotus log is:

2023-05-06T14:44:13.526Z        WARN    hello   hello/hello.go:108      other peer hasnt completed libp2p identify, waiting a bit
2023-05-06T14:44:20.695Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWFgpeYMKHBRgC4Qc3HRVRWFfbJX7moqEKKqT1vMFCySU4 addr=/ip4/1.83.125.173/tcp/26826 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T14:45:26.287Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWSyE7CspqGinUWdDqvZdVRJS3dnnJ4SXFnbu9MnW7DTNd addr=/ip4/39.109.125.191/tcp/13003 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T14:53:55.440Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWL4fa7WQTn51A31hf3muMofHZnSKixTTYnXkDjdcZYnFW addr=/ip4/139.215.226.71/tcp/33459 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T15:04:47.965Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWECcYQ4Mwvq5eSD1VvLtBU5HpunmHR9wHGz9AAKid1mWh addr=/ip4/122.190.56.248/tcp/55888 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T15:08:20.540Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWFPzNZHn6nq6C9eePghgxuBti8NpAgPrfbhW38vycxtpH", "err": "rate limit exceeded"}
2023-05-06T15:08:31.019Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWFPzNZHn6nq6C9eePghgxuBti8NpAgPrfbhW38vycxtpH", "err": "rate limit exceeded"}
2023-05-06T15:08:43.098Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWFPzNZHn6nq6C9eePghgxuBti8NpAgPrfbhW38vycxtpH", "err": "rate limit exceeded"}
2023-05-06T15:13:56.387Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWBvnurf9L8yESf4W3GaZfAzQ6n9c5w5PCzHFwGJeDU2qG addr=/ip4/154.210.12.64/tcp/10101 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T15:14:04.489Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWB5k1a8E6TaJ8Ru5A3VhixHe2Njdm7toHx7MehPiSXAx9 addr=/ip4/154.42.3.136/tcp/24001 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T15:24:39.974Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWAv3kxvUAmJXmQns4GEVqeXhRSRpEmWHANWxbM9p9Uj2d addr=/ip4/116.169.54.254/tcp/43864 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T15:26:51.660Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T15:27:32.933Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T15:28:11.266Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T15:28:49.399Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T15:29:44.119Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T15:29:54.246Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T15:35:36.629Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWBwajPfNpGr3hTbiMTP1KoLA7BBWefkpXk69zhPB1VcZC", "err": "rate limit exceeded"}
2023-05-06T15:35:46.650Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWBwajPfNpGr3hTbiMTP1KoLA7BBWefkpXk69zhPB1VcZC", "err": "rate limit exceeded"}
2023-05-06T15:35:46.795Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWBwajPfNpGr3hTbiMTP1KoLA7BBWefkpXk69zhPB1VcZC", "err": "rate limit exceeded"}
2023-05-06T15:36:06.569Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWBwajPfNpGr3hTbiMTP1KoLA7BBWefkpXk69zhPB1VcZC", "err": "rate limit exceeded"}
2023-05-06T15:36:18.672Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWBwajPfNpGr3hTbiMTP1KoLA7BBWefkpXk69zhPB1VcZC", "err": "rate limit exceeded"}
2023-05-06T15:44:00.718Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWQyX7XAX6ngjgdZTDNfbstvC58nzQFfbEaxBFHPWVQfdP addr=/ip4/63.217.225.70/tcp/43319 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T15:44:57.273Z        INFO    net/identify    identify/id.go:396      failed negotiate identify protocol with peer    {"peer": "12D3KooWHC3ZVATqH8ryUz7WnjdcmtuyTQ1SrFKAu69aUuujyD6q", "error": "stream reset"}
2023-05-06T15:44:57.273Z        WARN    net/identify    identify/id.go:371      failed to identify 12D3KooWHC3ZVATqH8ryUz7WnjdcmtuyTQ1SrFKAu69aUuujyD6q: stream reset
2023-05-06T15:45:39.445Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWNf7AN5JGcvtfiVtEbw84UXKRZmbLKQvQWH7HxGJtXnU3 addr=/ip4/221.206.40.14/tcp/61214 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T15:53:55.560Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWSqaeFgtgx1idtZfqzMebKLQyrxVADz1z4miD4wYcourM addr=/ip4/65.181.92.94/tcp/24001 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T16:02:39.172Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWANtvNpKWnzATuruX8AS4BSCwj52KLcqgQdaw9bZ1oTv5", "err": "rate limit exceeded"}
2023-05-06T16:05:13.045Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWQyX7XAX6ngjgdZTDNfbstvC58nzQFfbEaxBFHPWVQfdP addr=/ip4/63.217.225.70/tcp/43319 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T16:24:06.703Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWBxsG4ChM3ioufdp4sZ4oKh3EDKjcWUf863GvhfgZFwgc addr=/ip4/121.66.119.169/tcp/30170 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T16:24:11.581Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWFgpeYMKHBRgC4Qc3HRVRWFfbJX7moqEKKqT1vMFCySU4 addr=/ip4/1.83.125.173/tcp/26826 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T16:24:13.852Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWGziXA8iNKdoaVyYAGVuPbsCD2HT3Snv3UuExvbFS9Xbu addr=/ip4/203.208.141.111/tcp/16168 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T16:24:40.517Z        INFO    canonical-log   swarm/swarm_dial.go:500 CANONICAL_PEER_STATUS: peer=12D3KooWAdmv6VV141JD9HwXCaPhzKDquGskQuEwMN7Aa2HmAUeT addr=/ip4/144.48.65.140/tcp/31149 sample_rate=100 connection_status="established" dir="outbound"
2023-05-06T16:27:15.291Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T16:27:40.914Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T16:29:37.823Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNxvbu1bijMHQfaXsGLRUwLToegzbgGVGD8hooECjMb6F", "err": "rate limit exceeded"}
2023-05-06T16:34:06.402Z        WARN    hello   hello/hello.go:108      other peer hasnt completed libp2p identify, waiting a bit
2023-05-06T16:37:44.736Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNmbBtLWZDrWJgg8FpvDvMYJ51WDhkxWunqeSfJ3da9LU", "err": "rate limit exceeded"}
2023-05-06T16:38:30.287Z        WARN    sub     sub/incoming.go:588     ignoring indexer message        {"sender": "12D3KooWNmbBtLWZDrWJgg8FpvDvMYJ51WDhkxWunqeSfJ3da9LU", "err": "rate limit exceeded"}

stuberman May 6, 2023

Are you running lotus under systemd? Have you tried running by CLI?

Answer selected by TippyFlitsUK

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't sync the lotus node #8001

{{title}}

Replies: 16 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Can't sync the lotus node #8001

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Replies: 16 comments · 6 replies

versionCollector Jan 26, 2022 Author

versionCollector Jan 26, 2022 Author

versionCollector Jan 31, 2022 Author

versionCollector Feb 1, 2022 Author

versionCollector Feb 2, 2022 Author

versionCollector Feb 3, 2022 Author

versionCollector Feb 3, 2022 Author

versionCollector Feb 4, 2022 Author

Replies: 16 comments 6 replies

versionCollector Jan 26, 2022
Author

versionCollector
Jan 26, 2022
Author

versionCollector
Jan 31, 2022
Author

versionCollector
Feb 1, 2022
Author

versionCollector
Feb 2, 2022
Author

versionCollector
Feb 3, 2022
Author

versionCollector
Feb 3, 2022
Author

versionCollector
Feb 4, 2022
Author