Sync stuck

[simlecode]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

v1.24.1

Repro Steps

1. Run '...'
2. Do '...'
3. See error '...'
   ...

Describe the Bug

Stuck when synchronizing block bafy2bzacec4vdpopvdefza454tfcvsp7aqc45afjdzmf3oxk6hd3tnh45ov6w, cannot get the block when forking. Block bafy2bzacec4vdpopvdefza454tfcvsp7aqc45afjdzmf3oxk6hd3tnh45ov6w has two parents bafy2bzaceb4en7hmequdcyfocmwplkzgtsgrwmnbjav7zbunjhh6c5g45x554 and bafy2bzaceb7nrihhr26zyfy2f2ydnu2rbpednsu7obdy7vr33zzkbfbimy5f6

Logging Information

./lotus sync status 

worker 90252:
        Base:   [bafy2bzaceb7nrihhr26zyfy2f2ydnu2rbpednsu7obdy7vr33zzkbfbimy5f6 bafy2bzacectfqu6fl5u7tteswtumn2tv53nthnemuoucbwl7uhxnpdyn3qd4w bafy2bzaceb4en7hmequdcyfocmwplkzgtsgrwmnbjav7zbunjhh6c5g45x554]
        Target: [bafy2bzacec4vdpopvdefza454tfcvsp7aqc45afjdzmf3oxk6hd3tnh45ov6w] (3613317)
        Height diff:    1
        Stage: header sync
        Height: 3613317
        Elapsed: 13h53m27.850425061s

2024-01-31T20:58:12.287+0800	INFO	chainstore	store/store.go:649	New heaviest tipset! [bafy2bzaceb7nrihhr26zyfy2f2ydnu2rbpednsu7obdy7vr33zzkbfbimy5f6 bafy2bzacectfqu6fl5u7tteswtumn2tv53nthnemuoucbwl7uhxnpdyn3qd4w bafy2bzaceb4en7hmequdcyfocmwplkzgtsgrwmnbjav7zbunjhh6c5g45x554] (height=3613316)
2024-01-31T20:58:12.288+0800	INFO	chain	chain/sync_manager.go:323	worker 90250 done; took 37.688551ms
2024-01-31T20:58:15.064+0800	INFO	chain	chain/sync_manager.go:233	selected sync target: [bafy2bzaceb7nrihhr26zyfy2f2ydnu2rbpednsu7obdy7vr33zzkbfbimy5f6 bafy2bzacectfqu6fl5u7tteswtumn2tv53nthnemuoucbwl7uhxnpdyn3qd4w bafy2bzaceb4en7hmequdcyfocmwplkzgtsgrwmnbjav7zbunjhh6c5g45x554]
2024-01-31T20:58:15.064+0800	INFO	chain	chain/sync_manager.go:315	worker 90251 syncing in [bafy2bzaceb7nrihhr26zyfy2f2ydnu2rbpednsu7obdy7vr33zzkbfbimy5f6 bafy2bzacectfqu6fl5u7tteswtumn2tv53nthnemuoucbwl7uhxnpdyn3qd4w bafy2bzaceb4en7hmequdcyfocmwplkzgtsgrwmnbjav7zbunjhh6c5g45x554]
2024-01-31T20:58:15.064+0800	INFO	chain	chain/sync_manager.go:323	worker 90251 done; took 5.208µs
2024-01-31T20:58:21.779+0800	INFO	consensus-common	consensus/compute_state.go:306	ApplyBlocks stats	{"early": 0.000003, "earlyCronGas": 0, "vmMsg": 0.592255871, "msgGas": 13274982402, "vmCron": 1.744678497, "cronGas": 44085414241, "vmFlush": 0.076273434, "epoch": "3613316", "tsk": "{bafy2bzaceb7nrihhr26zyfy2f2ydnu2rbpednsu7obdy7vr33zzkbfbimy5f6,bafy2bzacectfqu6fl5u7tteswtumn2tv53nthnemuoucbwl7uhxnpdyn3qd4w,bafy2bzaceb4en7hmequdcyfocmwplkzgtsgrwmnbjav7zbunjhh6c5g45x554}"}
2024-01-31T20:58:22.815+0800	WARN	rpc	go-jsonrpc@v0.1.9/handler.go:456	error in RPC call to 'Filecoin.StateMarketStorageDeal': deal 5321260 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed:
    github.com/filecoin-project/lotus/chain/stmgr.GetStorageDeal
        /home/runner/work/lotus/lotus/chain/stmgr/actors.go:264
2024-01-31T20:58:30.075+0800	INFO	chain	chain/sync_manager.go:233	selected sync target: [bafy2bzacec4vdpopvdefza454tfcvsp7aqc45afjdzmf3oxk6hd3tnh45ov6w]
2024-01-31T20:58:30.075+0800	INFO	chain	chain/sync_manager.go:315	worker 90252 syncing in [bafy2bzacec4vdpopvdefza454tfcvsp7aqc45afjdzmf3oxk6hd3tnh45ov6w]
2024-01-31T20:58:30.076+0800	WARN	chain	chain/sync.go:852	(fork detected) synced header chain ([bafy2bzacec4vdpopvdefza454tfcvsp7aqc45afjdzmf3oxk6hd3tnh45ov6w] - 3613317) does not link to our best block ([bafy2bzaceb7nrihhr26zyfy2f2ydnu2rbpednsu7obdy7vr33zzkbfbimy5f6 bafy2bzacectfqu6fl5u7tteswtumn2tv53nthnemuoucbwl7uhxnpdyn3qd4w bafy2bzaceb4en7hmequdcyfocmwplkzgtsgrwmnbjav7zbunjhh6c5g45x554] - 3613316)
2024-01-31T20:58:30.170+0800	INFO	canonical-log	swarm/swarm_dial.go:506	CANONICAL_PEER_STATUS: peer=12D3KooWReTnGWK7PTpJx3t8GF7QwdAdgrTgkYUPwgS5219R3mm1 addr=/ip4/124.239.180.3/tcp/43949 sample_rate=100 connection_status="established" dir="outbound"
2024-01-31T20:58:30.205+0800	WARN	net/identify	identify/id.go:505	error reading identify message: stream reset
2024-01-31T20:58:30.205+0800	WARN	net/identify	identify/id.go:399	failed to identify 12D3KooWSd8R5Mbc9u5KwU7kbkUwmD7czWnvtV1tL4u4upf78nGJ: stream reset
2024-01-31T20:58:31.317+0800	INFO	chain	chain/sync.go:212	received block outside of consensus range at height 3552934
2024-01-31T20:58:32.082+0800	INFO	chain	chain/sync.go:212	received block outside of consensus range at height 3608023
2024-01-31T20:58:32.300+0800	INFO	chain	chain/sync.go:212	received block outside of consensus range at height 3485450
2024-01-31T20:58:32.406+0800	WARN	hello	hello/hello.go:108	other peer hasnt completed libp2p identify, waiting a bit
2024-01-31T20:58:42.053+0800	INFO	net/identify	identify/id.go:427	failed negotiate identify protocol with peer	{"peer": "12D3KooWCHmHJQPJBGaHX46iAKKggZbHPDajdfVFpKjKL1TjonAg", "error": "stream reset"}
2024-01-31T20:58:42.053+0800	WARN	net/identify	identify/id.go:399	failed to identify 12D3KooWCHmHJQPJBGaHX46iAKKggZbHPDajdfVFpKjKL1TjonAg: stream reset
2024-01-31T20:58:49.505+0800	INFO	chain	chain/sync.go:212	received block outside of consensus range at height 3552934
2024-01-31T20:58:51.694+0800	INFO	chain	chain/sync.go:212	received block outside of consensus range at height 3535277
2024-01-31T20:58:53.632+0800	INFO	net/identify	identify/id.go:427	failed negotiate identify protocol with peer	{"peer": "12D3KooWHyoJ46Zvw9YGzrBU1tWcpp2ZQiYpUYRBG3wjb85Lw1im", "error": "stream reset"}
2024-01-31T20:58:53.632+0800	WARN	net/identify	identify/id.go:399	failed to identify 12D3KooWHyoJ46Zvw9YGzrBU1tWcpp2ZQiYpUYRBG3wjb85Lw1im: stream reset
2024-01-31T20:58:54.058+0800	WARN	hello	hello/hello.go:108	other peer hasnt completed libp2p identify, waiting a bit

[rjan90]

Hey @simlecode! This syncing bug seems to be related to issues that was fixed in:

• feat: syncer: optimize syncFork for one-epoch forks #11533
• fix: sync: do not include incoming in return of syncFork #11541

Both of which has landed in the Lotus v1.25.2 version.

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 24 hours.

[github-actions]

This issue was closed because it is missing author input.